Transcript 2003-07-23-HTCIA-InvestigatingProductionServers

Analysts International
Analyzing Production
Windows Servers
Introductions
• Mark Lachniet from Analysts International,
Sequoia Services Group (Lansing & Detroit)
• Member of the HTCIA
• Not in law enforcement or a lawyer
• Senior Security Engineer and Security
Services technical lead
• Frequent presenter and trainer
• Certified Information Systems Security
Professional (CISSP)
• Microsoft MCSE, Novell Master CNE, Linux
LPI Certified LPIC-1, Check Point Certified
CCSE, TruSecure TICSA, etc.
2
Agenda
• Where a technical security engineer fits
into an investigation
• Frequent types of incidents
• Types of investigation
• Capturing volatile information on a
Windows box (tools and techniques)
• Expanding the scope of investigation
• Real life examples
3
Where the Security Engineer Fits
•
Dedicated security consultants can help in both
prevention and response
In prevention – designing and maintaining secure
technological and organizational systems (not just
technology!) Too bad people don’t do this!
In response – the topic at hand
•
•
–
–
–
–
For specific tools and technical expertise for a variety
of systems (servers, workstations, network devices)
To investigate an incident before deciding whether or
not to prosecute
To help weigh costs and benefits of various courses of
action – how to investigate, how to secure
To assist in prosecution by thoroughly researching
and documenting findings without the constrictions
that law enforcement would have (?)
4
Frequent Security Incidents
• The vast majority of calls I get are in regard
to a “hacking incident”
• Almost all of these incidents are on Internetconnected machines, most are Windows
• Most incidents are precipitated by:
– An external complaint (your mail server is sending me
a lot of spam e-mail)
– A change in the system (the hard drive is full, strange
new programs are running, tape backups are taking a
lot longer)
– The Internet is “slow” or we see strange activity
– A threat from an insider – usually a network
administrator making casual statements about how
they could “take them out” if they ever got fired
5
Types of Investigation
• Once a call comes in requesting help in investigation,
the engineer is dispatched on-site
• The first (and perhaps most important) step is discuss
the situation with the victim before doing any work
• There are basically three ways we approach an
investigation:
– “Pull the Plug” – don’t touch the machine
– “Limited Investigation” – tread lightly
– “Extensive Investigation” – heavy footprint, honeypot?
• Each of these approaches have advantages and
disadvantages, depending on your goals
• The most important question to ask is how strongly
the customer feels about trying to prosecute
• The second most important question to ask is how
much $$ they have to spend
6
Analyzing Volatile Information
• WARNING: This may not be the best way to analyze
a production system. I am not advocating that you
adopt this methodology, or that you should even
attempt to analyze a production server. I am not a
lawyer! This hasn’t been defended in court!
• Obviously, simply shutting down a computer that is
running so you can image the hard drive can result in
the loss of lots of interesting information
• We might also trip a “bomb” by not unplugging, but
we take our changes – hackers aren’t usually smart
• Interesting volatile information:
–
–
–
–
Running processes (and the specific executable)
System users (logged on and not logged on)
Network connections (incoming and outgoing)
Open files (which processes are using what files)
7
Getting Started
• At this point, we have already had discussions
with the victim and have the “go ahead”
• If it is an Internet-connected computer, perform a
vulnerability assessment against the machine
(most hackers won’t be particularly bothered by
another hacker scanning the box from an
unknown IP, so its not necessarily a sign that
someone is on to them)
• Identify any obvious holes in the box that could
be the means of entry (sometimes hackers will fix
the holes once they are in)
• This may lead you to focus on a particular service
(such as a Trojan) or a particular log file (such as
IIS)
8
Getting Started
• Examine the server as a hacker would –if it is an
FTP server, is there any warez or pr0n there?
• Do a google on the DNS name and IP address to
see if it is a known open proxy server (this
sometimes works!) Also e-mail blacklist services
• Create an updated version of the forensics CD
(updated virus signatures and anti-trojan
software)
• Create and label floppy disks
• Create a worklog to record on
• Start up a “known good” command interpreter
from the CD and start working, get date & time
• Results are normally saved to floppy
9
Using ‘pslist’
• d:\pstools\pslist -t > a:\pslistt.txt
• Gives you a “tree” view of processes
10
Using ‘pslist’
• d:\pstools\pslist -x > a:\pslistx.txt
• Gives you a “full” view of processes
11
Using ‘psloggedon’
• d:\pstools\psloggedon > a:\psloggedon.txt
• Gives you a listing of logged on users, both
locally and remotely via ‘net use’ mapping
• Useful for finding unauthorized users, people
accessing the system through terminal server
12
Using ‘netstatp’
• d:\netstatp\release\netstatp -a -n > a:\netstatp.txt
• Gives you a listing of listening network
servers *and* current TCP and UDP
connections
• A basic netstat comes with windows, but this
may be replaced by a “fixed” or Trojaned
version that will specifically ignore a traffic
pattern they want to hide so don’t use it
• Useful for finding illegal users of the systems
and their IP addresses to track down later
• Does *not* map specific programs to the
ports that are listening or established
13
Using ‘netstatp’
SMTP e-mail is listening
Web Server is listening
Windows NetBIOS is
listening
An established session
between our computer and
another computer
14
Enough Evidence?
• Based on the last few slides, you already
have enough information to determine
at least one of the attacks on the server
• Can anyone say what it is?
15
Using ‘fport-NG’
• d:\fportng\fport > a:\fport.txt
• Allows you to map listening services to specific
executables – a *very* useful
16
Using ‘handle’
• d:\handle\handle > a:\handle.txt
• Allows you to view current file “handles” (or
references) to files and objects in use
17
Grab MAC Times
• d:\perl\perl.exe \sfile\sfile.pl -d c:\ | \unix\gzip > a:\sfile.gz
• Refers to file system Modify, Access, Create
• This is old news to a forensic examiner, but
what if you can’t image the machine?
• Its too much data to fit well on a floppy, so
gzip it up using a UNIX command line tool
• Exports in a handy comma-delimited format
for import into databases, spreadsheets and
other analysis tools
• Note: Date conversion may take some work
• Now you can recreate events by date, or figure
out when a specific file was created
18
View MAC Times in Excel
19
Other Miscellaneous Items
• There are various other things to look at
including:
–
–
–
–
–
Scheduled jobs (using the ‘at’ command)
Netbios info (using the ‘nbtstat –c’ command)
Net info (using ‘net accounts’, ‘net users’, etc.)
Create MD5 hashes of files (esp. system files)
Back up large files to a forensic laptop over the
network:
• IIS Logs (WWW, FTP, SMTP) from
c:\winnt\system32\LogFiles\*.*
• Windows Event Logs (if possible – not on open files)
• Log files from applications, temp files, browser caches
• Copies of suspect binaries for inspection
• Any warez or pr0n that you need to analyze later
• Etc.
20
Do Your Research!
• At this point, you should more or less know all of the
running programs and network connections on the
box
• Dig in and identify each of them to make sure they
are legitimate – this is very time consuming!
• Use the Internet, google, etc. to identify unknown
components and figure out what they do
• Verify “known” components actually match the MD5
signatures of their counterparts on a known-good
system (if available) or compare file sizes and dates
• Verify that each “known” component is actually using
a port that it is expected to use
• Some trojans name themselves the same as other
legitimate executables (e.g. services.exe), but you’ll
find them in a different directory with fport-NG
21
Our Test Case
• Pretend we exhaustively verified each running
process, using google searches such as server.exe “port
139” until we get to patch.exe
22
We have identified a trojan!
• In this case, it’s the “Net Bus” Trojan
• Admittedly, we did this the hard way
• We can now research the Trojan, learn what it does,
and how it works
• In this case we can tell the MAC times of the
executable
• We can correlate these times with other server logs
(such as firewall, web and FTP, and Windows Event
Logs) to try to pin the activity to a user
• We can also look for the traffic pattern to see if we
can find out who was using it. In this case, we did
collect this volatile state data back on the ‘netstatp’
slide and know the IP address of the trojan user
(192.168.109.1)
• Maybe we can find out more about the attacker from
that IP address
23
Additional Analysis: f-prot
• Run a virus scan on the server (we use f-prot)
• Either map a drive to the server and run it across the
network, or run it directly from the forensic CD
• There may be more than one naughty file to find
Here is our
culprit
24
Additional Analysis: Anti-Trojan
• Run an anti-trojan scan on the server
• Either map a drive to the server and run it across the
network, or install it and leave a big footprint
• It can also check via portscan and registry scan
25
Additional Analysis: DumpReg
• Leaves a foot print on the server
• Shows registry keys *AND* the time of last
modification (a little-known ability)
• Useful for tracking software installations and
modifications over time
• Allows for export to comma delimited for Excel
26
Additional Analysis: Misc
• There are plenty of other tools you can use, mainly to
monitor ongoing system activity
• This may be useful to find out the depth of the attack,
gather more evidence, or trick them into revealing
something useful
• Some realtime tools:
–
–
–
–
Filemon: Monitors file access
Regmon: Monitors registry access
Tdimod: Monitors TCP/IP activity
Snort: Intrusion Detection System to identify attack
signatures in use on the network (especially useful if you are
being used as a stooge to launch attacks)
– Ethereal: Free protocol analyzer to capture and analyze
network traffic – may have a useful “payload” like IRC or
IM messages, usernames, email addresses, etc.
27
Analyze Other Log Sources
• In the networked world, no machine is an island
• If systems have been appropriately designed and
implemented, which isn’t that often, there will be
useful information in a variety of places
• The investigator must expand the scope from the
“victim system” and look elsewhere
• Additional evidence can be found in many places:
–
–
–
–
Network and security devices on location
Internet Service Providers (AOL, DSL providers, etc)
Other servers on the network
Client workstations (especially if an insider is
suspected)
– Authentication systems
– The attacker’s workstation
28
Expanding the Scope of Investigation
29
Analyzing Router/Firewall logs
• Some of the best information for figuring out how an
attack occurred and subsequent activity is by
examining the logs of network devices such as routers
and firewalls
• Unfortunately, many people don’t collect this data
and store it, or even know that its possible
• Network device logs can provide a detail of what type
of information traveled between network systems:
– Determine how the system was profiled (reconnaissance)
– Determine how the system was attacked (vulnerability)
– Determine what happened after the attack – did the hacker
use your system to store files? Attack other systems?
– Determine if multiple parties were involved (hackers tend to
run in packs in different parts of the world)
– Note: you need to compare time to victim system and “real”
time to that of all the other log sources
30
Create a Deliverable Document
• Once you have as much information as possible, you
need to document all of the data you have collected
and provide an analysis of the raw data
• This document should attempt to summarize:
–
–
–
–
What happened (chronological sequence of events)
How it happened (what vulnerability was used)
Problem areas (what couldn’t be done / analyzed)
Next steps (both short term recovery and long term security
steps that should be taken)
– Full appendix of collected data
• All of this information needs to be thoroughly
explained so that non-technical people can
understand the scope and impact of the incident and
make decisions
• This document can be given to law enforcement to
save time –hopefully we didn’t ruin any evidence!
31
Next Steps
• The decision to prosecute is not an easy one to make
because there are many implications:
– What will be the cost of prosecuting, in terms of legal
expenses, time spent, interruption to operations, etc.
– What is the likelihood of success?
– What is to be gained by prosecuting?
– What are the implications to public image? Nobody
wants to be in the newspaper, nobody wants to be
exposed as having poor security
– There is no guarantee that you will even be able to
prosecute if you want to. What if the perpetrator lives
in a developing country with no computer laws?
• Unless it was an insider job, or a specifically targeted
attack, most people consider it a “learning
experience” and hopefully secure their systems
32
Examples: The Warez Server
• For this presentation, I did a little experiment, and
set up a “honeypot” server on the Internet
• This server was a standard Windows 2000 server,
and was fully up to date (no vulnerabilities)
• The only change made from the default
configuration was a single (confusing) checkbox
that said to allow write access on the File Transfer
Protocol (FTP) server – an easy mistake to make
• I put the machine on the Internet to see how long it
would take for hackers to find it and abuse it
• The answer is: 3 days. Within 3 days, hackers had
found the server, and discovered that it was
possible to store files there anonymously
33
Examples: The Warez Server
• Within a week, a “tag” had been placed
(hacker lingo for claiming the server – there is
honor among thieves)
• A few days later, a huge number of “hidden”
directories were created on the server, and
software was uploaded to it.
• A few days after that, people from the
Internet were downloading the illicit content,
and I pulled the plug
• I’m still not sure what they uploaded, but
most of the time its porn
• The lesson here is that they WILL find you,
and quickly at that
34
Examples: Manufacturing
• A manufacturing company was getting
complaints from people claiming that spam was
coming from their mail server
• Their ISP shut them down due to abuse calls
• They had investigated internally and couldn’t
figure out what was happening
• Analysis of the server found that they were
directly connected to the Internet without a
firewall or other protection
• Further analysis found several problems:
– An open mail relay (allows spam)
– An open proxy server (allows anonymous web access)
– An open socks server (allows full Internet access)
35
Examples: Manufacturing
• Analysis of log files showed that people from all
over the world had been relaying connections
through their server
• Abuse included people looking at pornographic
web sites, sending spam
• A search of the Internet found that the company
server had been listed on multiple hacker sites as
being an “open” relay
• Thus, not only are the hackers who find you
going to abuse you, but they are going to share
their good fortune with others
• What are the legal liabilities of being a third
party to this type of activity?
36
Examples: Marketing
• A marketing firm calls with concerns because
the network administrator found a remotecontrol program on the server (very bad)
• The server was connected to the Internet
without a firewall
• Additional user ID’s had been created and
granted administrative access
• Client suspected internal involvement
• Logging on the server was turned off, so no
good data was collected
• Logging on the network devices was also
turned off, so there was no data there either
37
Examples: Marketing
• Examination of the server turned up some
evidence, such as the time and date that the
remote control software was installed, and
evidence that there was a hack but not much!
• However, because there was no logging, there
was no sure way to know if the attack was
internal or external
• Also because there was no logging, there was
no way to track to an offending workstation
by IP address
• The only real option was to clean up the
damage, and start recommending some
security services to stop it from happening
again
38
Examples: K12 District
• School district in Michigan with a fast
connection to the Internet
• No problems were known
• The district contracted with us to have a
managed firewall installed
• As soon as we turned it on and started
analyzing traffic, it was obvious that they
were currently being abused
• Investigation showed that they were
unknowingly hosting child pornography – not
a good thing for a school
• Many other people have found existing
problems just by logging
39
Prevention and Response
• None of the previous incidents made it to the legal
system, it just wasn’t worth it for them
• None the less, it was an expensive, emotional and
painful experience for them
• Much of that pain could have been minimized
through prevention instead of response
• Unfortunately, computer security is a somewhat
like the wild west – its somewhat lawless,
although serious crimes can be pursued its
usually not worth it
• We use the metaphor of the neighborhood when
describing computer security – the best approach
is to make your own home hard enough to break
into that they go to your neighbor instead
40
Discussion
Mark Lachniet, Sr. Security Engineer
CISSP, MCNE, MCSE, CCSE, LPIC-1, TICSA
Analysts International - Sequoia Services
3101 Technology Blvd. Suite A
Lansing, MI 48910
phone: 517.336.1004
fax: 517.336.1004
41