Transcript 2008-04-21-MACUL

Analysts International
Basic Security
Incident Response
Mark Lachniet
Agenda
•
•
•
•
•
•
•
•
•
•
Frequent types of incidents
Anonymous hacks vs. targeted
How hacking happens
Types of investigation
Expanding the scope of investigation
Volatile data
Hard drives and Metadata
Documentation and procedures
Real life examples
Detection and prevention
2
Frequent Security Incidents
• The vast majority of incident response calls I
get are in regard to a “hacking incident” or
malware
• Almost all of these incidents are on Internetconnected machines
• Most incidents are precipitated by:
– An external complaint (your mail server is sending me
a lot of spam e-mail)
– A change in the system (the hard drive is full, strange
new programs are running, tape backups are taking a
lot longer)
– The Internet is “slow” or we see strange activity
– A threat from an insider – usually a network
administrator making casual statements about how
they could “take them out” if they ever got fired
3
Frequent Security Incidents
• Many complaints focus on inappropriate use
of company technology:
– Employees looking at pornography at work
– A user is suspected of having “hacking” tools
– Suspected theft of trade secrets / proprietary info
• Another frequent event is an “employee
termination” scenario:
–
–
–
–
Employee is usually a computer administrator
Employee has extensive access to many systems
Employee is a “troublemaker”
Employer wishes help in terminating the employee, and
wants to remove their access FIRST before firing him
– Typically involves a lot of brainstorming to identify all
possible points of ingress to the computing
environment
4
An Impersonal World
• There are really two different types of computer
hacking incidents – personal and impersonal
• In my experience, they are almost always impersonal
hacking attacks, not someone who intentionally
targeted the victim (though might be happening more
often lately)
• Most hackers could care less who you are, or what
sensitive information you have, they simply want to
control an Internet-connected server
• Usually this the hacked computer is intended to:
– Use you as part of a bot network, probably spam and
denial of service attacks on others
– Share questionable material, using your Internet
connection and server space (the “warez”server)
– Access questionable material, using you as a relay to hide
their origin (an open proxy server)
– Steal information such as passwords to systems
5
The Lucrative World
of Bot Herding
• See http://www.mellorsecurity.com/Botnets.pdf
• People are making money! Millions of dollars
• There are economies based on:
– High-tech attack and defense malware
– Distributed “bot” networks that can be remote
controlled by a command and control server
– Used for Denial of Service extortion, spamming,
stealing network or system information, click
advertisement abuse, etc.
• For this reason, bot networks are becoming a very
real problem, and will only get worse
• Knowing how to identify and remove malware may
be a useful skill
6
Attack Vectors
• Hacking is generally possible due to a
vulnerability or a mis-configuration in some
device or software
• Vulnerabilities exist in old versions, and new
vulnerabilities are constantly discovered
• Researchers, hackers, software developers all
work to find them and there are mailing lists
such as bugtraq to stay informed
• In some case, patches are reverse engineered
to see what was fixed and exploit it before
everyone gets their patches on
7
Attack Vectors
• Sometimes user activity (e.g. Internet browsing or
running attachments) is the source of malware,
• In other cases unsolicited attacks come across
local networks and the Internet
• These attacks are usually scripted and don’t
require much (or any) manual interaction by the
attacker
• Anti-Virus (e.g. SAV 10.x or later) or Host
Intrusion Prevention Systems (e.g. Cisco CSA)
are helpful to block them
• Good egress filters on your firewalls will also help
to block them from communicating
• Provide user education on the risks!
8
A Typical Network
9
Understanding Networks
• The example given previously is an example of “best
practices” in network design, and provides some defense
against Internet attacks
• Many (most?) organizations do not have an adequate
network design, and have significant risk from the Net
• Even the BEST network design can’t protect a machine
that is insecure!
• Each machine that can talk to the Internet has a unique
identifier called an “IP Address”
• IP addresses are sometimes static, and sometimes change
frequently (especially for dial-up users)
• Regardless, tracking IP addresses is frequently our only
recourse to track network attacks
• For example, if the IP address of a hacker can be tracked
to AOL, it is then possible to obtain further info from AOL
through legal action
10
Types of Investigation
• The first (and perhaps most important) step is discuss
the situation before doing any work
• You will use different approaches for different types
of attacks – a possible disclosure of a large database
of credit card numbers is much different from a
simple workstation virus
• There are several ways to approach an investigation:
– “Pull the Plug” – don’t touch the machine
– “Limited Investigation” – tread lightly
– “Extensive Investigation” – tread heavily
• Each of these approaches have advantages and
disadvantages, depending on your goals
• The “footprint” on the evidence from your work, the
importance of volatile information, the probability
that the machine will be used as evidence in a court of
law all need to be considered
11
“Pull the Plug”
• Not recommended!
• Used when a company is VERY intent on prosecution,
if there is contraband on the computer or you are
concerned about accidentally tampering with
evidence
• Is the typical “nintendo forensics” approach - unplug
it, “image” it and analyze it for content
• This is highly disruptive and expensive, as the server
is no longer available but you’ll probably want to do
this anyway so you can image the hard drives
• There is also no opportunity to examine the “state” of
the machine that will be lost when turned off:
– Which programs are running
– Memory contents
– Current network connections, etc
12
“Limited Investigation”
• Used to try to validate that there is a problem, figure
out whether or not to image the machine, determine
the scope and nature of the incident, etc.
• Useful to see what the malware was doing – it might
be nice to know if you just leaked something that will
require an “oops letter” mass mailing
• May be less disruptive and less expensive – the server
doesn’t need to be taken down to do a preliminary
examination
• Must analyze the system with tools that leave a very
light “footprint” and will not modify much system
information
• Incident Response CD’s such as F.I.R.E. are designed
for this purpose see: http://fire.dmzs.com/
• Interpreting results of the tools is the hard part…
13
“Extensive Investigation”
• Most extensive data-gathering, thus slightly more
expensive due to labor
• Still somewhat non-disruptive, the system is up and
running, although it may need to be restarted
occasionally and may be slow
• More invasive tools can be used – these will leave a
trail on the filesystem, but will provide the maximum
of information
• For example, it may be possible to do things such as:
–
–
–
–
–
Monitor all file and registry accesses
Monitor and record network traffic
Turn on more logging
Attempt to shadow suspected users and sessions
Plant honeypots (password.xls, etc.)
• May set up a “sandbox” to test and analyze it in a
controlled environment
14
Analyze Other Log Sources
• In the networked world, no machine is an island
• If systems have been appropriately designed and
implemented, which isn’t that often, there will be
useful information in a variety of places
• The investigator should expand the scope from the
“victim system” and look for evidence elsewhere
• Beware of timesync issues
• Additional evidence can be found in many places:
–
–
–
–
Network and security devices on location
Internet Service Providers (AOL, DSL providers, etc)
Other servers on the network
Client workstations (especially if an insider is
suspected)
– Authentication systems
– The attacker’s workstation
15
Expanding the Scope of Investigation
16
Analyzing Firewall / IDS logs
• Some of the best information for figuring out how an
attack occurred and subsequent activity is by
examining the logs of network devices like firewalls
and Intrusion Detection/Prevention Systems
• Unfortunately, many people don’t collect this data
and store it, or even know that its possible
• Network device logs can provide a detail of what type
of information traveled between network systems:
– Determine how the system was profiled (reconnaissance)
– Determine how the system was attacked (vulnerability)
– Determine what happened after the attack – did the hacker
use your system to store files? Attack other systems?
– Determine if multiple parties were involved
17
Small IR toolkit
• http://lachniet.com/smallir.zip has a small Windows
Incident Response Kit, or use a boot CD such as Helix
• See http://www.e-fense.com/helix/
• Before unplugging it, we want to gather volatile data
from the target system
• You might want to use a script such as the Windows
Forensic Toolchest
• See http://www.foolmoon.net/security/wft/
• We will look at the output of a few commands that
can be used to figure out what a potentially
compromised system is doing
–
–
–
–
Pslist – list processes
Netstat – list network connections
Handles – list open “handles” (resources)
Listdlls – list DLLs in use by the application
18
pslist
• Lists processes in a nested view
• Can see which programs spawned others
• Investigate every running process for possible
malware (google search the filename if you don’t
know what it is)
19
netstat
•
•
•
•
Shows listening and established network connections
Use ‘netstat –a –n –b’ on recent windows machines
Identify listening services to see if they are malware
Look for outgoing connections to the Internet that
don’t seem appropriate
20
handles
• Handy for finding where the malware is residing on
the system
• Shows open files and system resources
21
listdlls
• Can sometimes indicate the location of malware or
other trickiness
22
Forensic Imaging
• Analyzing a hard drive for forensic information
is a discipline all its own
• There is a lot of information that is left behind
• One must assume that any case taken will
eventually end up in the legal system
• As such, one must be very careful with potential
evidence
• In general one should use a read-only means of
copying the original hard drive (such as
hardware write blocker or a Helix boot CD) to an
image file and work ONLY from the image file
• Never work on the original system!
• Keep chain of custody documentation
23
Data and Metadata
• One thing to be aware of is that hard drives
(and other storage devices) use different types
of filesystems
• Filesystems behave in different ways, but
generally they are intended to make disk
access organized and efficient (not secure)
• To do this, they use “metadata” to keep track
of where data on a disk is – for example the
“fat file” that you may have heard about
• Metadata is like a card catalog – it is data
about where other data resides on the system
24
Filesystem Metadata
• Because filesystems aim to be efficient, they also leave
behind remnants of data
• For example, if you delete a file, the computer
generally does not go back and wipe out the old data
with 0’s or 1’s
• Instead, it simply marks the space the data used to
use as unallocated
• This means that you might be able to find the
metadata and simply “undelete” the file
• Or, you might be able to “carve” the data from the
filesystem directly (bypassing the metadata)
• May also have data in the ends of disk allocation units
(slack space)
25
Data Carving
• Data carving, for example the open source
“foremost” program is used to recover files
from the hard drive, even if they have been
deleted and their metadata has been removed
• It does this by identifying file headers (the
first few bytes that are associated with a file
type) and footers
• When it sees a header that matches, for
example, a JPG file, it cuts it out and saves it
to a new directory
• In this way, I can take an image of a hard
drive and carve out every identifiable JPG
and GIF image on it for review
26
Thumbnails
• Another little tidbit to know about is the Windows
thumbnails feature
• When you have your windows explorer window in
thumbnail view, it creates a small index file called
thumbs.db (slightly different for Vista)
• Inside of this is a very small thumbnail of every image
that is in the directory
• However, when you delete the image, you do NOT
delete the thumbnail inside of thumbs.db
• Hence you have a running record of all the files that
once existed, no matter how long ago they were
deleted, even if they were over-written)
• This can be a surprise to people who thought they
were being sneaky by deleting
27
E-Mail Programs
• Another area where a lot of time is spent is on email programs
• Many client programs such as Thunderbird and
Outlook leave a cached copy of all of your email
on the local workstation
• Sometimes webmail fragments can be recovered
from memory, the swap file, or hard drive
• This can sometimes be recovered if it is deleted
• It is possible to convert these into different
formats (for example from an outlook PST file to
a MBOX format UNIX filesystem) and then do
all manner of analysis on
28
Record Keeping and
Static Procedures
• Analyst should take detailed written notes
• Actions taken should be detailed along with the
time it was done if possible
• It is good if more than one person is involved,
with the second person signing off on it
• Standardized procedures should be used to
eliminate the risk of error and to have a
standardized methodology
• Electronic record keeping must also be secured to
minimize the risk of modification – one way is
through digital signatures (cryptographic hashes
that prove the integrity of data)
29
Create a Deliverable Document
• Once you have as much information as possible, you
need to document all of the data you have collected
and provide an analysis of the raw data
• This document should attempt to summarize:
–
–
–
–
What happened (chronological sequence of events)
How it happened (what vulnerability was used)
Problem areas (what couldn’t be done / analyzed)
Next steps (both short term recovery and long term security
steps that should be taken)
– Full appendix of collected data
• All of this information needs to be thoroughly
explained so that non-technical people can
understand the scope and impact of the incident and
make decisions, and technical people can validate and
recreate your findings
30
Should you prosecute?
• The decision to prosecute is not an easy one to make
because there are many implications:
– What will be the cost of prosecuting, in terms of legal
expenses, time spent, interruption to operations, etc.
– What is the likelihood of success?
– What is to be gained by prosecuting?
– What are the implications to public image? Nobody
wants to be in the newspaper, nobody wants to be
exposed as having poor security
– There is no guarantee that you will even be able to
prosecute if you want to. What if the perpetrator lives
in a developing country with now computer laws?
• Unless it was an insider job, or a specifically targeted
attack, most people consider it a “learning
experience” and hopefully secure their systems
31
Examples: The Warez Server
• I once did a little experiment, and set up a
“honeypot” server on the Internet
• This server was a standard Windows 2000 server,
and was fully up to date (no known vulnerabilities
at that time)
• The only change made from the default
configuration was a single (confusing) checkbox
that said to allow write access on the File Transfer
Protocol (FTP) server – an easy mistake to make
• I put the machine on the Internet to see how long it
would take for hackers to find it and abuse it
• The answer is: 3 days. Within 3 days, hackers had
found the server, and discovered that it was
possible to store files there anonymously
32
Examples: The Warez Server
• Within a week, a “tag” had been placed
(hacker lingo for claiming the server – there is
honor among thieves)
• A few days later, a huge number of “hidden”
directories were created on the server, and
software was uploaded to it.
• A few days after that, people from the
Internet were downloading the illicit content,
and I pulled the plug
• I’m still not sure what they uploaded, but
most of the time its porn
• The lesson here is that they WILL find you,
and quickly at that
33
Examples: Manufacturing
• A manufacturing company was getting
complaints from people claiming that spam was
coming from their mail server
• Their ISP shut them down due to abuse calls
• They had investigated internally and couldn’t
figure out what was happening
• Analysis of the server found that they were
directly connected to the Internet without a
firewall or other protection
• Further analysis found several problems:
– An open mail relay (allows spam)
– An open proxy server (allows anonymous web access)
– An open socks server (allows full Internet access)
34
Examples: Manufacturing
• Analysis of log files showed that people from all
over the world had been relaying connections
through their server
• Abuse included people looking at pornographic
web sites, sending spam
• A search of the Internet found that the company
server had been listed on multiple hacker sites as
being an “open” relay
• Thus, not only are the hackers who find you
going to abuse you, but they are going to share
their good fortune with others
• What are the legal liabilities of being a third
party to this type of activity?
35
Examples: Marketing
• A marketing firm calls with concerns because
the network administrator found a remotecontrol program on the server (very bad)
• The server was connected to the Internet
without a firewall
• Additional user ID’s had been created and
granted administrative access
• Client suspected internal involvement
• Logging on the server was turned off, so no
good data was collected
• Logging on the network devices was also
turned off, so there was no data there either
36
Examples: Marketing
• Examination of the server turned up some
evidence, such as the time and date that the
remote control software was installed, and
evidence that there was a hack but not much!
• However, because there was no logging, there
was no sure way to know if the attack was
internal or external
• Also because there was no logging, there was
no way to track to an offending workstation
by IP address
• The only real option was to clean up the
damage, and start recommending some
security services to stop it from happening
again
37
Examples: K12 District
• School district in Michigan with a fast
connection to the Internet
• No problems were known
• The district contracted with us to have a
managed firewall installed
• As soon as we turned it on and started
analyzing traffic, it was obvious that they
were currently being abused
• Investigation by a district employee related
that they were unknowingly hosting child
pornography – not a good thing for a school
• Many other people have found existing
problems just by logging
38
Examples: Disciplinary Action
• Often I will be brought in to confirm or deny an
allegation of misuse of computer resources
• In a few cases I have had to prove that an employee
did something they shouldn’t have with company
material
• In one instance I recovered a bunch of dirty pictures
that were circulating in e-mail. The individual had
sued my client for wrongful termination (until they
saw my report)
• In another instance I had to prove and an employee
was working two jobs simultaneously. I was able to
find e-mails from their new employer’s HR people
about orientation procedures
• This particular individual was also writing a pseudo
pornographic humorous screenplay at the same time
39
Prevention and Response
• None of the previous incidents made it to the legal
system, it just wasn’t worth it for them
• None the less, it was an expensive, emotional and
painful experience for them
• That pain could have been minimized through
prevention instead of response
• Unfortunately, computer security is a somewhat
like the wild west – its somewhat lawless,
although serious crimes can be pursued its
usually not worth it
• We use the metaphor of the neighborhood when
describing computer security – the best approach
is to make your own home hard enough to break
into that they go to your neighbor instead
40
Security Stuff to Do
• The following list doesn’t do justice to the wide scope
of information security, but there are a few things
that every company needs to do:
– Design secure solutions - networks, systems and
software with security in mind.
– Have vulnerability assessments performed
(Internet, internal network, and organizational)
– Ensure that all servers that are Internet connected
or store important data are properly “hardened”
– Use some kind of auditing and logging system to
maintain an audit trail
– Maintain appropriate computer use policies
– Retain security staff to regularly evaluate log data,
perform analysis, etc.
– Consider maintaining an Incident Response plan
41
Thank You!
Mark Lachniet, Solutions Architect
[email protected]
http://lachniet.com/powerpoint
for this document
42