Transcript Taking on the Giant (anatomy of an attack)

Limb from Limb: Taking on the Giant
(or for those who like boring titles –
Anatomy of an Attack)
Harvard Townsend
IT Security Officer
[email protected]
October 31, 2007
1
Thursday, August 23

8:40am - Josh Ballard warns SIRT about global
increase in scanning for vulnerable Trend Micro
ServerProtect (TMSP) instances





Recommends making sure TMSP is patched.
Reports some inbound traffic on TCP port 5168
Refers to SANS Internet Storm Center report from
that morning
SANS reports says “It does indeed look like
machines are getting owned with this vulnerability”
(actually in Aug. 22 diary)
10:16am - Seth Galitzer reports odd behavior of 3
Windows servers that started the afternoon of
Aug. 22, including the “blue screen of death”
(BSOD) and continual rebooting
2
Thursday, August 23



Seth tried backing out Aug. 14 Windows
patches. Didn’t help
11:39am – Seth associates reboots with
a crash of spntsvc.exe – Trend Micro
ServerProtect service; Disabled the
service and systems stabilized. Other
servers with ServerProtect not affected.
1:42pm – Harvard speculates that Seth’s
problem and Josh’s announcement are
related; Points out that Trend Micro has a
patch to fix the vulnerability in TMSP
3
Friday, August 24






8:44am – Marin Dowlin reports same problem
with a server
8:53am – another one reported in Biochemistry
9:41am – Seth reports TMSP patches won’t
install
10:18am – Shea McGrew recommends removing
TMSP and upgrading to OfficeScan 8.0 if TMSP
patches fail.
3:28pm – Brandon Utech joins the chorus;
reported two servers rebooted with the same
error immediately after a ServerProtect-initiated
scan started; provided detailed analysis that
pointed to TMSP as the culprit
4:55pm – sporadic network outages start,
continued until nearly midnight
4
Saturday, August 25



12:50pm – more network problems
reported
3:06pm – Redundant campus Internet
connection disabled to stabilize the
network
7:21pm – A server is compromised a
second time from a different source;
uploads malware to the server
5
Monday, August 27




8:44am – Dereatha Cross reports no
problems with Windows servers running
Trend Micro OfficeScan, which does not have
the same vulnerability
Josh identifies four compromised servers that
launched a Distributed Denial of Service
Attack; network access for these servers is
blocked
The DDoS packets failed to leave the campus
since the source IP addresses were spoofed
Josh begins analyzing network flow data to
look for other compromised servers
6
Diversion for Definitions

Distributed Denial of Service Attack
(DDoS)


Multiple compromised systems flood the
bandwidth or resources of a targeted
system, usually one or more web servers
(i.e., they deny the ability of the target to
provide the intended service)
When the source is hundreds, maybe
thousands of computers all over the
world, is very difficult to stop
7
Spoofed Source IP Address





Source IP address identifies the computer where
the network packet originated
All K-State IP addresses start with 129.130 (for
example, 129.130.12.18)
A “spoofed” source IP address re-writes the data
packet header with a false address to hide its
origin
Any packet with a source IP address NOT starting
with 129.130 is discarded at the campus border
Thus the DDoS attack flooded our campus
network with traffic, but not the intended target
(trendmicro.com web server)
8
Network Flow Data


Cisco routers collect information about
IP network traffic
“Flow” = a unidirectional sequence of
packets all sharing these 5 values:



Source + destination IP address
Source + destination port
IP protocol (IP, TCP, UDP, ICMP, etc.)
9
Cisco Netflow Record Contents



Timestamps for the flow start and finish time
Number of bytes and packets observed in
the flow
Layer 3 headers:





Source & destination IP addresses
Source and destination port numbers
IP protocol
Type of Service (ToS) value
Nothing about packet payload (content)
recorded (i.e., we’re not snooping)
10
Monday, August 27 continued



3:08pm – Harvard discusses strategy with
SIRT; still some uncertainty and not much
information in security community about this
exploit; have not been able to confirm how our
four servers were compromised
3:33pm – Bryan Boutz reports that one of the
compromised servers had the ServerProtect
patch applied on Thursday, Aug. 23, but he
had to reinstall TMSP for patch to take
4:06pm – Harvard indicates that activity on
port 5168 started on Wednesday, Aug. 22, so
his server was likely compromised the day
before he patched it.
12
Monday, August 27 continued



4:23pm – From flow data, Josh determines
that the DDoS targeted trendmicro.com,
further evidence that the servers were
compromised by the TMSP vulnerability
4:26pm – Harvard sends urgent warning to
campus about four compromised servers,
impending block of port TCP/5168 at the
campus border, urges sysadmins to remove
ServerProtect and upgrade to OfficeScan 8.0
4:48pm – Bryan Boutz updated TM pattern
file, scanned the compromised server, and the
backdoor was identified and removed.
13
Tuesday, August 28

Peace and quiet returns to Oz
Wednesday, August 29
 TCP port 5168 blocked at the border
14
Timeline in Retrospect







June 2007 – Trend Micro privately informed of the
vulnerability
July 27 – Trend Micro quietly releases patch, which is
not part of automatic pattern file updates; has to be
manually applied
Aug. 22 – Trend Micro announces the vulnerability and
the patch since exploits are now occurring in the wild
Aug. 22 – SANS Internet Storm Center reports
increased activity on port 5168 on the Internet
Aug. 22, 4:08pm – first server compromised
Aug. 22-23 – 10 servers compromised at K-State
Aug. 23 – first speculation that the rebooting servers
related to TMSP vulnerability; sysadmins start patching
ServerProtect, which prevents further spread
15
Timeline in Retrospect







Aug. 23 – Trend Micro pattern file able to detect/clean
malware from this exploit
Aug. 25 – Dormant malware on four compromised
server awakened by remote control via IRC to launch
DDoS attack on trendmicro.com, causing network
problems; DDoS unsuccessful because of K-State’s
border protection
Aug. 25 – One server compromised a second time
Aug. 27 – Four compromised servers blocked; netflow
analysis confirms source of compromise is exploited
vulnerability in ServerProtect.
Aug. 27 – urgent call to patch/upgrade issued
Aug. 28 – Netflow analysis completed, revealing a total
of 10 compromised servers that are blocked
Aug. 29 – blocked port TCP/5168 at the border
16
Zero-Day Exploit or Attack



Defn: Attack launched either before or
on the same day that a vulnerability is
announced or discovered
In our case, the vulnerability and patch
were publicized on Aug. 22, the same
day K-State servers were compromised
However, the patch was actually
released on July 27
17
Lessons Learned




Josh is a netflow wizard and master analyst!
Critically important for the campus community to work
together in response to threats (collective wisdom wins
again!)
Communication critical to coordinated, appropriate
response
Should have blocked 5168 at the border sooner, but it
would not have prevented the infections



Except one case where a server was compromised twice
via port 5168 from two different sources; the second
Zero-day exploits are very, very difficult to defend
against
An aggressive “default deny” host-based firewall config
would have prevented infection (i.e., only allow specific
hosts to connect to port 5168)
18
Lessons Learned


Our normal IRC botnet detection missed this because the
botnet controller was a new one (it’s now in the list of
known controllers!)
Was a stealth botnet, so very difficult to detect until it
launched the DDoS






No extra processes running
Injected its code in svchost.exe process so it wasn’t noticed
Log of IRC commands in memory, not on disk
It is sometimes difficult to correlate anomalous behavior
with an exploit/attack
Even though the Trend Micro pattern file detected the
malware on Aug. 23, it was installed on Aug. 22 so it might
not get detected until the next scheduled scan on Wed.
Aug. 29.
Is it time to recommend daily scheduled scans? Beware of
the performance impact on a server
19
Lessons Learned




This could and has happened to other anti-virus
software products, like Symantec. Trend Micro
products are still an effective and important part of
our security arsenal.
Ironic that security software (Trend Micro
ServerProtect) was the source of the compromise,
but not surprising since security software operates
deep within the OS. It is therefore an attractive
target for hackers.
Interesting connection between exploit of Trend
Micro software that was then used to attack
trendmicro.com
Hackers are now targeting applications as much as
operating systems, compounding the challenge of
keeping systems secure
20
Lessons Learned

The challenge of patching as a preventative
measure






Are hundreds of vulnerabilities announced
each week and hundreds of patches in dozens
of different applications and OSes
Zero-day exploits are relatively rare
Some patches break things
Best practice is to test patches thoroughly
before deploying in production, but that’s not
practical now
Yet, a vulnerable system is a substantial risk
So what’s a nerd to do?!?!
21
Recommendations





Aggressive firewalls
Regularly scan for open ports/services, and
evaluate their necessity
Automated patching of as much as possible
Consider a daily TMOS scheduled scan
(again, beware of performance impact)
More IT security staff to:





Monitor and evaluate alerts
Implement IDS/IPS technology
Manage firewalls
Perform host and network forensics analysis
Handle incidents
22
Lessons Learned


The threat is real
We are living dangerously
23
What’s on your mind?
24