Transcript UNIX-Linux-Investigation-by-Yuli-Chen-Rajesh

Procedures in Intrusion
Investigation of a UNIX/Linux
Host
CS-585F:
Computer-Related Law and Computer
Forensics
Fall 2002
By
Yu-Li Chen
Rajesh Menon
Joe Meslovich
1
Introduction
• UNIX does not have a good reputation for
reliability or security (Gollmann, 1999).
• UNIX systems directly connected to the Internet
are often subject to hacking attempts (Kruse &
Heiser, 2002).
• The skill and knowledge to investigate a
compromised UNIX system becomes essential for
both UNIX users and forensic investigators.
2
Approach
• Looking for signs of a compromised UNIX system
• Preserving and gathering evidence
• Collecting information and seizing the data
• Analyzing the file system
• Examining signs of intrusion
3
Signs of a compromised UNIX system
1. Examine log files
2. Look for setuid and setgid files
3. Check the system binaries
4. Look for signs of a network sniffer
5. Examine files run by cron and at
6. Check for unauthorized services
7. Examine /etc/passwd file
8. Check system and network configuration
9. Look every where for unusual or hidden files
10. Examine all machines on the local network
4
Examine log files
Refer to /etc/syslog.conf to see how logging is configured for each
system service. The default location for these log file is the /var/log
directory.
• messages log - look for anomalies
• xferlog will contain log files for all of the ftp transfers
• utmp contains binary information for every user currently logged inuse who or w to access this file
• wtmp is modified every time a user successfully logs in, logs out, or
your machine reboots-use last to get information
• secure is used by some versions of UNIX (RedHat Linux for
example) to log tcp wrapper messages -look for anomalies
5
[root@localhost amanda]# more /etc/syslog.conf
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*
/dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none
/var/log/messages
# The authpriv file has restricted access.
authpriv.*
/var/log/secure
# Log all the mail messages in one place.
mail.*
/var/log/maillog
# Everybody gets emergency messages, plus log them on another
# machine.
*.emerg
*
# Save mail and news errors of level err and higher in a
# special file.
6
[root@localhost amanda]# w /var/log/utmp
1:16pm up 4:04, 2 users, load average: 0.09, 0.04, 0.01
USER TTY FROM
LOGIN@ IDLE JCPU PCPU WHAT
[root@localhost amanda]# last /var/log/wtmp
wtmp begins Fri Dec 6 22:55:48 2002
[root@localhost amanda]# more /var/log/secure
Dec 8 13:17:29 localhost login: ROOT LOGIN ON tty2
Dec 8 13:25:14 localhost login: LOGIN ON tty1 BY amanda
Dec 10 10:20:52 localhost login: ROOT LOGIN ON tty2
7
Look for setuid and setgid files
• Intruders often leave setuid copies of /bin/sh or
/bin/time around to allow them root access at a later time
• The UNIX find program can be used to hunt for setuid
and/or setgid files
•
find / -user root –perm –4000 –print
find / -group kmem –perm –2000 –print
8
[root@localhost amanda]# find / -user root -perm -4000 -print
find: /mnt/cdrom: Input/output error
find: /proc/6/fd: Permission denied
find: /proc/1376/fd/4: No such file or directory
/bin/mount
/bin/umount
/bin/su
/bin/ping
/sbin/pwdb_chkpwd
/usr/X11R6/bin/Xwrapper
/usr/X11R6/bin/imwheel-solo
/usr/X11R6/bin/xlock
/usr/bin/chage
/usr/bin/gpasswd
/usr/bin/chfn
/usr/bin/chsh
/usr/bin/newgrp
/usr/bin/crontab
/usr/bin/passwd
9
Check the system binaries
• Intruders may change program on UNIX systems such as login,
su, telnet, netstat, ifconfig, ls, find du, df, libc, sync
• Compare the versions on your systems with known good copies
• Trojan horse programs may produce the same standard checksum
and timestamp as the legitimate version
• Standard UNIX sum command and the timestamps associate
with the programs are not sufficient
• The use of cmp, MD5, Tripwire, and other cryptographic
checksum tools is sufficient
10
Look for signs of a network sniffer
• Intruders may install a network monitoring program,
commonly called a sniffer (or packet sniffer)
• A sniffer could be installed on your system if any process
currently has any of your network interfaces in
promiscuous mode
• Some legitimate network monitors and protocol
analyzers will set a network interface in promiscuous
mode
11
Examine files run by cron and at
• The cron and at commands are used to schedule
commands and processes that repeat at specified intervals
• Intruders may leave back doors in files run from cron or
submitted to at
• These techniques can let an intruder back on the system
• Verify that all files/programs referenced (directly or
indirectly) by the cron and at jobs
at -l
crontab -l
12
Check for unauthorized services
• Inspect /etc/inetd.conf or /etc/xinetd.conf for
unauthorized additions or changes
• Search for entries that execute a shell program(for
example, /bin/sh or /bin/csh
• Check for legitimate services that you have
commented out in your /etc/inetd.conf
• Determine if intruders have turned on a service or
replaced the inetd program with a Trojan horse
program.
13
Examine the /etc/passwd file
• Check the /etc/passwd file on the system for
modifications
• Look for the unauthorized creation of new accounts,
accounts with no passwords, or UID changes (especially
UID 0) to existing accounts
• Use stat /etc/passwd to look at access and
modification information
• Use cat /etc/passwd to show the contents of the file
14
[root@localhost amanda]# more /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:
daemon:x:2:2:daemon:/sbin:
adm:x:3:4:adm:/var/adm:
lp:x:4:7:lp:/var/spool/lpd:
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:
news:x:9:13:news:/var/spool/news:
uucp:x:10:14:uucp:/var/spool/uucp:
operator:x:11:0:operator:/root:
games:x:12:100:games:/usr/games:
15
Check system and network configuration
• Find any unauthorized entries, especially ‘+’(plus sign)
entries on the system
• Find inappropriate non-local host names in /etc/hosts
• Confirm that these files existed prior to any intrusion and
were not created by the intruder.
16
Look for unusual or hidden files
• Files that start with a period . and are normally not
shown by ls can be used to hide tools and information
• A common technique on UNIX systems is to put a
hidden directory in a user’s account with an unusual name
• The find program can be used to look for hidden files
For example:
find / -name “..’’ –print –xdev
find / -name “.’’ –print –xdev | cat –v
17
Examine all machines on the local network
• If one host has been compromised, others on the network
have been, too.
18
Before collecting evidence….
• Keep a regular old notebook handy and take careful notes of what
you do during your investigation.
• Start with the script command to ensure that everything you do is
captured in a file.
script investigation.txt
date
uname –a
• use your own clean tools CD (if you have one) to read all the files
that you are going to investigate
mount -t iso9660 /dev/cdrom /mnt/cdrom
PATH = /mnt/cdrom
echo $PATH
/mnt/cdrom
19
Ways to preserve evidence
• CPU storage
• Video
• System memory
• Network connections
• Running processes
20
CPU storage
a. Registers: Infeasible to capture and of minimal
utility.
b. Caches: Infeasible to capture as a discrete entity
but should be captured as part of system memory
image
When UNIX systems are shutdown with the
shutdown command, all services are clearly
shutdown and cached file system buffers are
flushed, data is written to disc and can be captured
as part of system memory image.
21
Video
To save a copy of the screen:
1. Press Ctrl-Alt-F2 ( on a RedHat Linux system) to enter another
virtual terminals on the console and log on as root(if you know the
password).
2. Use xwd command to dump an individual window or use –root
option to dump entire screen
xwd –display localhost:0 –root > screenshot.xwd
3. Return to the X display by pressing Ctrl-Alt-F7
4. Examine it to make sure that you didn’t capture the screen saver
with xwud command
xwud –in screenshot.xwd
22
System memory
• Easy to capture, but the act of capturing it
changes it
• The physical memory on a UNIX system is
/dev/mem; the virtual memory is /dev/kmem
• use the dd command to copy the memory from
the suspect machine
dd if =/dev/mem of =suspect.mem.image
dd if =/dev/kmem of = suspect.kmem.image
• It is impossible to verify accuracy when
copying such highly volatile
23
Network connections
•Network state provides important information on both
current network connections and listening processes
• Know when attacker has left a network process
running or when unauthorized connections are taking
place either inbound or outbound
• netstat –p show the processes associated with
specific network connections
• netstat –rn display the routing tables
• arp –v capture the address resolution tables
• (date; netstat –p; netstat –rn; arp –v) >
suspect.netstatus.txt
24
Running processes
utilities :
•ps Display list of all running processes
•last, w, who Get listings of logged in users, prior logins, etc
•uptime Show current processing load and 2 previous load values
•top Password cracking tools will show up clearly with top.
•lsof Provide a list of all currently open files
•fuser Identifies processes using a specific file or network socket.
•strace List all system calls being made by a running process.
•truss, ktrace Earlier versions of system call trace.
•ltrace Library routine trace.
25
[root@localhost amanda]# ps
PID TTY
TIME CMD
1339 pts/2 00:00:00 bash
1404 pts/2 00:00:00 ps
[root@localhost amanda]# uptime
1:42pm up 4:30, 2 users, load average: 0.24, 0.13, 0.09
26
/proc
• Pseudo-filesystem that provides a structured interface
to /dev/kmem
• No attacks that hide /proc entries have been reported
• A more reliable guide to currently running processes
than the commonly hacked ps utility
• ls –d /proc/[0-9]* >suspect_proc.txt
27
Signs of hostile processes
• Any discrepancy between ps, top, and /proc
• Unrecognized commands, especially ones that start
with a . or ./
• Daemons running more than once that should be
running only once, such as inetd
• High uptimes (high system utilization) and
especially a process using an unusual level of system
resources
• High network utilization
28
The scenario
• The production server of the ABC company
is suspected to be compromised
• The precondition imposed on the forensics
team --- the server cannot be brought down
• Forensics is to be conducted on the live
server
• MD5 hash of the original disks was not
taken
29
Steps of the investigation
• 1) Collection of the evidence for the filesystem analysis
• 2) Investigation into the suspect file-system
30
Steps in the collection of evidence
• Using the netcat command
• Command on collection system
# nc –l –p 10000 > /tmp/suspect.passwd_file
• Command on the suspect system
# cat /etc/passwd /etc/shadow | nc 192.168.0.2 10000 –w3
31
Steps in the collection of evidence
(contd)
1) Log the investigation steps carried out on the
suspect system
• Command:
# script investigation.txt
2) Maintain a journal to enumerate the results of the
investigation
32
Steps in the collection of evidence
(contd)
3) Verify the date and time of the suspect system and
determine the DNS name and the OS version of
the system
• Commands:
# date
# uname -a
33
Steps in the collection of evidence
(contd)
4) Obtain the screen shots of the suspect system’s
desktop
• Command:
# xwd –display localhost:0 –root > screen.xwd
5) Save the copy of the suspect’s password and
shadow files and sent them to collection system
• Command:
# cat /etc/passwd/etc/shadow | nc 192.168.0.2 10000 –
w3
34
Steps in the collection of evidence
(contd)
6) Send the copies of the regular and kernel memory
to collection system
• Commands:
# dd bs=1024 < /dev/mem | nc 147.138.69.59 10002 –
w3
# dd bs=1024 < /dev/kmem | nc 147.138.69.59 10002 –
w3
35
Steps in the collection of evidence
(contd)
• 7) Send the date, details of processes with network
connections, the kernel routing table and the
address resolution protocol to collection system
• Command:
# (date; netstat –p; netstat –rn; arp –v) | nc
147.138.69.59 10002 –w3
36
Steps in the collection of evidence
(contd)
8) Commands to verify the state of suspect machine
1. Command to find the time for which the system
was running since its last reboot
• Command: # uptime
2. Command to find the users remotely connected
to the system
• Command: # who
37
Steps in the collection of evidence
(contd)
3) Check for the processes that take up system
resources
• Command:
# top
38
Steps in the collection of evidence
(contd)
9) Information about the running processes sent to
the collection system
• Command:
# (ps –aux; ps –auxeww; lsof) | nc 147.138.69.59 10002
–w3
39
Steps in the collection of evidence
(contd)
10) Find the list of running processes on the suspect
system and make a tar file to be sent to the
collection system
• Commands:
# ls –d /proc/[0-9]* | nc 147.138.69.59 10002 –w3
# tar –cvpf proc.tar /proc/[0-9]*
# nc 147.138.69.59 10003 –w3 < proc.tar
40
Steps in the collection of evidence
(contd)
11) Determine the physical devices and partitions on
the suspect system
• Commands:
# mount
# fdisk –l /dev/hda
41
Steps in the collection of evidence
(contd)
12) Make a data dump of the partitions of the suspect
system hard disk and send it to collection system
• Commands:
# dd if=/dev/hda1 bs = 1024 | nc 147.138.69.59 10002 –
w3
42
Steps in the collection of evidence
(contd)
13) Making the MD5 hash of the files received and
the data dump of the partitions of the suspect hard
disk
• Command:
# md5sum –b suspect.hda1.image
43
The Analysis
44
The Suspect System
•
•
•
•
Hardware: Dell Optiplex GX1
Operating System: RedHat Linux 8.0
Kernel Version: 2.4.18-14
Primary Use: Web Server (Apache/2.0.40)
45
The Collection System
• Hardware: Dell Optiplex GX110
• Operating System: RedHat Linux 8.0
• Kernel Version: 2.4.18-14
46
The first problem during collection
• We were unable to collect the /dev/kmem file.
• The dd command terminated with the an error.
# dd bs=1024 < /dev/kmem | nc 147.138.69.69 10003 w3
dd: reading `standard input': Bad address
0+0 records in
0+0 records out
47
The second problem during collection
• The images of the hard drive created using dd
were corrupt.
• Each dd displayed an error message after each
dump completed.
dd: reading `/dev/hda1': Input/output error
48
Image corruption
• The image files created were corrupt in two ways:
– Superblocks within the image were damaged
– Files being written to by the OS were empty
49
Compensating for the Corruption
• The images that were used had to be mounted
once in read-write mode to correct the superblocks
if they were to be used.
• The live file-systems were investigated instead of
the image.
• Tar balls of the missing log files were created.
50
Analyzing the Stored Data
•
•
•
•
passwd files – no anomalies
Network status file – no anomalies
Process status file – no anomalies
/proc directory – no anomalies
51
Analyzing the Log Files
• Web server access_log and error_log
– No anomalies found
• messages log
– Anomalies found!
52
The messages log
• Two attackers attempted to gain access between
12:17 PM EST and 2:31 PM EST on Friday
December 6, 2002
• Attacker number one attempted to gain access to
the system by way of the ftp server.
• Attacker number two attempted at least one port
scan and tried to gain access through the ftp and
telnet services
53
Attacker Number One
• Made twelve distinct attempts to open an ftp
session with the suspect server.
• Tried to access the service as the root user, and an
unprivileged user. In both cases the attacker failed
to guess the passwords.
Dec 6 12:53:10 dorm69-52 ftp(pam_unix)[1613]: authentication failure;
logname=uid=0
euid=0
tty=/dev/ftpd1613
ruser=
rhost=Venger.bridgewater.edu user=joe
Dec 6 12:53:12 dorm69-52 ftpd: Venger.bridgewater.edu: connected:
IDLE[1613]: failed login from Venger.bridgewater.edu [147.138.20.15]
54
Attacker Number Two
• Initially attempted an anonymous ftp to the
system.
• Waited one hour then made several attempts to
telnet into the system as root, lp, and as a blank
username.
• Ended attack with a port scan of the suspect
server.
Dec 6 14:31:07 dorm69-52 fingerd[2273]: Client hung up - probable portscan
55
Looking at the Filesystem
•
•
•
•
•
xferlog – no anomalous entries
/etc/hosts – no trust relationships
/dev – no hidden regular files
SUID/SGID – no anomalous files
core files – no files found
56
Binary Package Consistency
• rpm –Va – RedHat utility that checks the
consistency of the binaries package installed
using rpm.
– The only changes were to log files and default
configuration files. The utility did not find any
modified executables.
57
Conclusion
• Always double check syntax
• Always have an alternate way of collecting
a piece of data
58
Any Questions?
59