Transcript Slide 1
SERVICE PROVIDER RESPONSES
TO DENIAL OF SERVICE ATTACKS
SESSION SEC-2008
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
1
Denial of Service Can Happen to Anyone!
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
2
What Is Denial of Service?
…Gets Access
…Gets No Access
Authorized User…
Unauthorized User…
DoS
Intrusion
Location of Attacks: Layer 1–7
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
3
Denial of Service Attacks
• We understand intrusions: Do things right,
and you’re okay
• What about DoS? Do things right, and still get
drowned…
• DoS is often driven by financial motivation
DoS for hire :-(
• DoS cannot be ignored, your business depends
on effective handling of attacks
• Worms and DoS are closely related
Secondary worm effects can lead to denial of service
Worms enable DoS
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
4
News: January 22, 2002
Cloud-Nine Officially Closes ISP!
By: mark.j @ 10:44:AM—Comments (35)—SendNews [HERE]/PrintNews [HERE]
• Today looks set to be a sad and frustrating one for anybody who
was ever a customer of the once popular unmetered dialup and
broadband ISP Cloud-Nine
• At precisely 10:16 am a few minutes ago Emeric Miszti (CEO) and
John Parr (Operations Director) of the C9 ISP posted what’s likely
to be their final announcement on our forums; C9 is now the latest
ISP to close, although it's the first we've ever seen to go from a
hack attack!:
Cloud Nine regret to announce that at 7:45 this morning the decision was taken to
shut down our Internet connections with immediate effect
We tried overnight to bring our web servers back online but were seeing denial of
service attacks against all our key servers, including email and DNS; these were of
an extremely widespread nature
http://www.ispreview.co.uk/cgi-bin/ispnews/printnews.cgi?newsid1011696274,91619
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
5
DoS: The Procedure
1. Cracking
2. Signalling
3. Flooding
ISP
Hacker
CPE
Target
mbehring
Innocent
User PCs
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
“Zombies”
or “Bots”
6
An SP View: Denial of Service
Zombies on
Innocent
Computers
Peering
Link
AS 24
ISP Backbone
Enterprise
ISP Edge
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
7
Agenda
• Detecting and Classifying DoS Attacks
• Tracing DoS Attacks
• Reacting to DoS Attacks
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
8
DETECTING AND
CLASSIFYING DOS ATTACK
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
9
Network Baselines
• Network baselines from a variety of sources
• Unexplained changes in link utilization
Worms can generate a lot of traffic, sudden changes in link
utilization can indicate a worm
• Unexplained changes in CPU utilization
Worm scans can affect routers/switches resulting in
increased CPU both process and interrupt switched
• Unexplained syslog entries
• These are examples
Changes don’t always indicate an attack or worm!
Need to know what’s normal to identify abnormal behavior
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
10
Ways to Detect and Classify DoS Attacks
• Customer call
• SNMP: Line/CPU overload, drops
• NetFlow: Counting flows
• ACLs with logging
• Cisco Anomaly Detector (formerly Riverhead)
• Backscatter
• Sniffers
• Third party partner products
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
11
Detecting DoS Through CPU Load
A: CPU Total Utilization
B: CPU at Interrupt Level
router>sh proc cpu
CPU utilization for five seconds: 99%/97%; one minute: 78%; five minutes: 23%
• A: Total CPU load
• B: CPU at interrupt level (note: B <= A)
• A-B: Process switched traffic, CPU processes
(See: http://www.cisco.com/warp/public/63/highcpu.html)
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
12
Abnormal CPU Load
A: CPU Total Utilization
B: CPU at Interrupt Level
router>sh proc cpu
CPU utilization for five seconds: 99%/97%; one minute: 78%; five minutes: 23%
• If A≈B: “Too much traffic to forward”
Interrupts: Packet switching (fast switching)
• If A >> B: “Too much central processing”
Packets to/from the router (e.g. SNMP, ICMPs, vty and console, IPsec
(w/o h/w), routing…)
Process switched packets or switching problem
If attack, targeted at the router itself!
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
13
Netflow: Detection and Classification
• Netflow provides enough data to develop a baseline
What’s normal --> what’s abnormal
• Changes in Netflow indicative of changing traffic
patterns
Might be DoS
SPAM and other mass mailers (e.g. a virus)
• Customers who use Netflow report very high rate of
detection
Partner tools such as Arbor provide a lot of back-end
intelligence
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
14
Using Netflow
• Real-time Netflow display
Show ip cache flow
Use inc command as needed
Look for relevant data
• Data analysis
Export data for external analysis
Find anomalies and changes variation from “normal”
Scripts, Netflow tools, Arbor Networks
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
15
Detecting DoS Attacks with NetFlow
BASIS: HAVE NETFLOW RUNNING ON THE NETWORK
On Border Routers, Every X Min:
Count Flows
with Sampling 1/Y
DANTE Uses:
X=15 Min, Y=200,
Z=10 Sec, N=10
During Z Sec
If
# of Flows
>N
Values Are Empirical
Y
Alarm!
N
End
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
16
Netflow: Real DoS Traffic
Potential DoS Attack (33 Flows) on Router1
Estimated: 660 Pkt/s 0.2112 Mbps
ASxxx Is: …
ASddd Is: …
Real
Real Data
data Deleted
deleted in
in
this Presentation
presentation
src_ip
dst_ip
in out src dest pkts bytes prot src_as dst_as
int int port port
192.xx.xxx.69
192.xx.xxx.222
192.xx.xxx.108
192.xx.xxx.159
192.xx.xxx.54
192.xx.xxx.136
192.xx.xxx.216
192.xx.xxx.111
192.xx.xxx.29
192.xx.xxx.24
192.xx.xxx.39
192.xx.xxx.249
192.xx.xxx.57
…
SEC-2008
194.yyy.yyy.2
194.yyy.yyy.2
194.yyy.yyy.2
194.yyy.yyy.2
194.yyy.yyy.2
194.yyy.yyy.2
194.yyy.yyy.2
194.yyy.yyy.2
194.yyy.yyy.2
194.yyy.yyy.2
194.yyy.yyy.2
194.yyy.yyy.2
194.yyy.yyy.2
…
29
29
29
29
29
29
29
29
29
29
29
29
29
…
9822_05_2004_c2
49
49
49
49
49
49
49
49
49
49
49
49
49
…
1308
1774
1869
1050
2018
1821
1516
1894
1600
1120
1459
1967
1044
…
© 2004 Cisco Systems, Inc. All rights reserved.
77
1243
1076
903
730
559
383
45
1209
1034
868
692
521
…
1
1
1
1
1
1
1
1
1
1
1
1
1
…
40
40
40
40
40
40
40
40
40
40
40
40
40
…
6
6
6
6
6
6
6
6
6
6
6
6
6
…
xxx
xxx
xxx
xxx
xxx
xxx
xxx
xxx
xxx
xxx
xxx
xxx
xxx
…
ddd
ddd
ddd
ddd
ddd
ddd
ddd
ddd
ddd
ddd
ddd
ddd
ddd
…
17
Classifying DoS with ACLs
• Requires ACLs to be in place (for detection)
Extended IP access list 169
permit icmp any any echo (2 matches)
permit icmp any any echo-reply (21374 matches)
permit udp any any eq echo
permit udp any eq echo any
permit tcp any any established (150 matches)
Found:
• Attack type
• Interface
permit tcp any any (15 matches)
permit ip any any (45 matches)
• Watch performance impact
• Used on demand, not pro-active
Looks Like
Smurf Attack
• More used for checking than for detection
• Some ASIC based LCs do not show counters
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
18
Code Red Worm (Aug 2001)
• Infects Microsoft IIS web servers
• Spread: Using real source, random destination
• Attack: accessing a specific server
217.33.138.14- - [07/Aug/2001:01:17:32 +0100] "GET
/default.ida?NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN%u9090%u6858%ucbd3%u7801%u9090%u
6858%ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9090%u9090%u8190%u00c3%u0003%u8b00
%u531b%u53ff%u0078%u0000%u00=a HTTP/1.0" 400 328
Infected Host
(Real IP!)
SEC-2008
9822_05_2004_c2
http get
© 2004 Cisco Systems, Inc. All rights reserved.
Fill Buffer
Unicode Encoded
Assembler Code
19
Code Red:
Detection of Infected Hosts
• Spread: Infected host accesses
random IP addresses (using real IP address)
• Examine Scatter: Every host likely to receive some
“code red” HTTP requests!
• On an ISP’s network:
Route big chunks of unused space analyser
(sniffer, NetFlow, etc.)
Analyzer receives lots of code red http connects
Log IP sources: These hosts are infected!
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
20
SQL Slammer Worm (Jan 2003)
• Very fast: Maximum spread after 10 min!
• No exploit, just spread (lucky!!)
Could have erased disks on all systems!
• Using true source, random destinations
• High pps load on the networks
• Scanning activity is a key indicator
How do we capture it? --> Sink Holes
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
21
Backscatter Analysis
• Sink hole router: Statically announce unused
address space (1/8, 2/8, 5/8, …)
(see http://www.iana.org/assignments/ipv4-address-space)
Note: Hackers know this trick: Use also unused space from
your own ranges (aka DarkIP)
• Or, use default (if running full routing)
• Victim replies to random destinations
• Some backscatter goes to sink hole router,
where it can be analyzed
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
22
Backscatter Analysis
Other
ISPs
Ingress
Routers
RANDOM DESTINATIONS
Target
Sink Hole Router
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
23
Sink Hole Architecture
To ISP
Backbone
Static ARP to
Target Router
SINKHOLE
GATEWAY
TARGET
ROUTER
To ISP
Backbone
SNIFFERS AND
ANALYZERS
To ISP
Backbone
• Dedicated network component to attract traffic
• Can also be used “on demand”: pull the DoS/DDoS attack to
the sinkhole
• Sink Hole design can also incorporate Riverhead scrubbers
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
24
Case Study: Slapper Worm (Sep 2002)
int isreal(unsigned long server) {
...
if (a == 127 || a == 10 || a == 0)
return 0;
if (a == 172 && b >= 16 && b <= 31)
return 0;
if (a == 192 && b == 168) return 0;
return 1;
}
IP Address: a.b.0.0
Worm does not use:
127.x.x.x
10.x.x.x
0.x.x.x
172.16-31.x.x
192.168.x.x
...
if (!isreal(udpclient.in.sin_addr.s_addr)) break;
...
Source: http://isc.incidents.org/analysis.html?id=167
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
25
Sinkholes: Worm Detection
Sinkhole Advertising
Bogon and Dark IP Space
SINKHOLE
NETWORK
May Also Use Netflow
Data from Edge Routers
for This Purpose…
Customer
Computer Starts
Scanning the Internet
SQL
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
26
TRACING DOS ATTACKS
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
27
Tracing DoS Attacks
• If source prefix is not spoofed:
Routing table
Internet Routing Registry (IRR)
Direct site contact
• If source prefix is spoofed:
Trace packet flow through the network
Find upstream ISP
Upstream needs to continue tracing
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
28
The Internet Routing Registry (IRR):
Network Info
• Europe:
whois.ripe.net
madrid% whois -h whois.arin.net 64.103.0.0
OrgName:
OrgID:
Address:
City:
StateProv:
PostalCode:
Country:
Cisco Systems, Inc.
CISCOS-2
170 West Tasman Drive
San Jose
CA
95134
US
NetRange:
CIDR:
[...]
TechHandle:
TechName:
TechPhone:
TechEmail:
64.100.0.0 - 64.104.255.255
64.100.0.0/14, 64.104.0.0/16
OrgTechHandle:
OrgTechName:
OrgTechPhone:
OrgTechEmail:
DN5-ORG-ARIN
Cisco Systems, Inc.
+1-408-527-9223
[email protected]
SEC-2008
9822_05_2004_c2
• Asia-Pac:
whois.apnic.net
• USA and rest:
whois.arin.net
CAH5-ARIN
Huegen, Craig
+1-408-526-8104
[email protected]
© 2004 Cisco Systems, Inc. All rights reserved.
Contact
Information
29
The Internet Routing Registry (IRR):
AS Info
madrid% whois -h whois.arin.net
as109
OrgName:
OrgID:
Address:
City:
StateProv:
PostalCode:
Country:
Cisco Systems, Inc.
CISCOS-2
170 West Tasman Drive
San Jose
CA
95134
US
ASNumber:
ASName:
ASHandle:
[…]
TechHandle:
TechName:
TechPhone:
TechEmail:
109
CISCOSYSTEMS
AS109
OrgTechHandle:
OrgTechName:
OrgTechPhone:
OrgTechEmail:
DN5-ORG-ARIN
Cisco Systems, Inc.
+1-408-527-9223
[email protected]
SEC-2008
9822_05_2004_c2
• Europe:
whois.ripe.net
• Asia-Pac:
whois.apnic.net
• USA and rest:
whois.arin.net
MRK4-ARIN
Koblas, Michelle
+1-408-526-5269
[email protected]
© 2004 Cisco Systems, Inc. All rights reserved.
Also, if domain known:
abuse@domain
30
IP Source Tracker
• Traditional way of tracking DoS:
ACL or NetFlow
Limitation in performance and cross LC support
• Source Tracker:
Across LCs, low performance impact
Line Card
• Availability:
GSR E0,1,2,4: 12.0(21)S
GSR E3: 12.0(26)S
GSR E4+: 12.0(21)S (POS), (23)S (other)
7500: From 12.0(22)S
Other: 12.3(7)T
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
31
IP Source Tracker
GSR
Special CEF
Adjacency
for victim
GRP
Export
Special CEF
Adjacency
for victim
i/f
i/f
IN
IN
Cache
Cache
OUT
OUT
Switch
Packets to the Victim
Other Packets
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
32
IP Source Tracker: Config
Router# ip source-track <victim>
Enable the Feature
Router# show ip source-track 10.1.2.1 (also: … summary)
Address
SrcIF
Bytes
Pkts
Bytes/s
Pkts/s
10.1.2.1
Pos 1/0
2000
20
200
1
Protocol
-------ICMP
Total
Flows
100
Victim
Flows
/Sec
1
Packets Bytes
/Flow
/Pkt
10
100
Packets Active(Sec) Idle(Sec)
/Sec
/Flow
/Flow
10
0
5
Ingress Interface
See: http://www.cisco.com/en/US/products/sw/iosswrel/ps1829/products_feature_guide09186a00800e9d38.html
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
33
Tracing Back with Netflow
Routers Need Netflow Enabled
Victim
router1#sh ip cache flow | include <destination>
Se1
<source>
Et0
<destination>
11 0013 0007 159
…. (lots more flows to the same destination)
The flows come from serial 1
router1#sh ip cef se1
Prefix
Next Hop
Interface
0.0.0.0/0
10.10.10.2
Serial1
10.10.10.0/30
attached
Find the upstream
router on serial 1
Serial1
Continue on this router
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
34
Show IP Cache Flow
router_A#sh ip cache flow
IP packet size distribution (85435 total packets):
1-32
64
96 128 160 192 224 256 288 320 352 384 416 448 480
.000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000
512 544 576 1024 1536 2048 2560 3072 3584 4096 4608
.000 .000 .000 .000 1.00 .000 .000 .000 .000 .000 .000
IP Flow Switching Cache, 278544 bytes
2728 active, 1368 inactive, 85310 added
463824 ager polls, 0 flow alloc failures
Active flows timeout in 30 minutes
Inactive flows timeout in 15 seconds
last clearing of statistics never
Protocol
Total
Flows
Packets Bytes
-------Flows
/Sec
/Flow /Pkt
TCP-X
2
0.0
1 1440
TCP-other
82580
11.2
1 1440
Total:
82582
11.2
1 1440
SrcIf
Et0/0
SrcIf
Et0/0
Et0/0
Et0/0
Et0/0
Et0/0
SEC-2008
9822_05_2004_c2
Flow Details
SrcIPaddress
132.122.25.60
139.57.220.28
165.172.153.65
DstIf
Se0/0
Se0/0
Se0/0
© 2004 Cisco Systems, Inc. All rights reserved.
Source Interface
Flow Info Summary
Packets Active(Sec) Idle(Sec)
/Sec
/Flow
/Flow
0.0
0.0
9.5
11.2
0.0
12.0
11.2
0.0
12.0
DstIPaddress
192.168.1.1
192.168.1.1
192.168.1.1
Pr
06
06
06
SrcP
9AEE
708D
CB46
DstP
0007
0007
0007
Pkts
1
1
1
35
Show IP Cache Verbose Flow
router_A#sh ip cache verbose flow
IP packet size distribution (23597 total packets):
1-32
64
96 128 160 192 224 256 288 320 352 384 416 448 480
.000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000
512 544 576 1024 1536 2048 2560 3072 3584 4096 4608
.000 .000 .000 .000 1.00 .000 .000 .000 .000 .000 .000
IP Flow Switching Cache, 278544 bytes
1323 active, 2773 inactive, 23533 added
151644 ager polls, 0 flow alloc failures
Active flows timeout in 30 minutes
Inactive flows timeout in 15 seconds
last clearing of statistics never
Protocol
Total
Flows
Packets Bytes
-------Flows
/Sec
/Flow /Pkt
TCP-other
22210
3.1
1 1440
Total:
22210
3.1
1 1440
SrcIf
Port Msk AS
Et0/0
5FA7 /0 0
Et0/0
SEC-2008
9822_05_2004_c2
SrcIPaddress
DstIf
Port Msk AS
216.120.112.114 Se0/0
0007 /0 0
175.182.253.65 Se0/0
© 2004 Cisco Systems, Inc. All rights reserved.
Packets Active(Sec) Idle(Sec)
/Sec
/Flow
/Flow
3.1
0.0
12.9
3.1
0.0
12.9
DstIPaddress
NextHop
192.168.1.1
0.0.0.0
192.168.1.1
Pr TOS Flgs Pkts
B/Pk Active
06 00 10
1
1440
0.0
06 00 10
1
36
Tracing Back with ACLs
• Create ACL:
access-list 101 permit ip any <target> log-input
• Apply to interface for a few seconds:
interface xxx
ip access-group 101 in
(wait a few seconds)
no ip access-group 101
• Log shows interface the attack comes from
14:17:21: %SEC-6-IPACCESSLOGP: list 101 permitted tcp 105.12.73.84(0) (FastEthernet0/0
0006.d780.2380) -> 192.168.1.1(0), 1 packet
14:17:22: %SEC-6-IPACCESSLOGP: list 101 permitted tcp 166.159.237.65(0) (FastEthernet0/0
0006.d780.2380) -> 192.168.1.1(0), 1 packet
mac Address
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
src Interface
37
Tracing Back Across an Internet Exchange
Point (IXP) (or Any Other Shared Medium)
• NetFlow: Shows i/f only
Useless if IXP: Lots of routers behind…
• ACLs with log-input:
Shows also the MAC address of the router:
1d00h: %SEC-6-IPACCESSLOGDP: list 101 denied
icmp 11.1.1.18 (Ethernet0 0001.96e6.7641) -> 10.1.2.1
(0/0), 169 packets
router#sh arp | include 0001.96e6.7641
Internet 12.1.1.99
Ethernet0
SEC-2008
9822_05_2004_c2
152 0001.96e6.7641 ARPA
Originating Router
© 2004 Cisco Systems, Inc. All rights reserved.
38
Tip for Logging
• Do not log to console
Slow connection (9.6 kbit/s)
Lots of logging You lose the console
• Logging buffered
Avoids console overload
Automatically wraps
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
39
Trace-Back in One Step: ICMP Backscatter
• Border routers: Allow ICMP (rate limited)
• From sink hole router:
iBGP update to all ingress routers:
“drop all traffic to <victim>” (details later)
• All ingress router drop traffic to <victim>
• And send ICMP unreachables to source!!
• For spoofed sources:
Sink hole router logs the ICMPs!
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
40
How to Detect Drops on a Router
Other
ISPs
Ingress
Routers
DROPS CAN BE SEEN:
• Netflow: destination “null0”
• Interface “null0” counters
(simple!)
• ICMP unreachables
ICMP
Unreachables
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
41
Trace-Back in One Step: ICMP Backscatter
Other
ISPs
Ingress
Routers
iBGP Updates:
“Drop Packets to Target”
Target
Sink Hole Router
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
42
Trace-Back in One Step: ICMP Backscatter
Other
ISPs
Ingress
Routers
iBGP Updates:
“Drop Packets to Target”
Target
ICMP
Unreachables
Sink Hole Router
with Logging
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
43
ICMP Backscatter
ON SINK HOLE ROUTER:
• Static routes for 1/8,2/8,5/8 (will attract 3/256 of packets)
• Access-list 105 permit icmp any any log-input
Access-list 105 permit ip any any
• Border router sends ICMP unreachable for deleted packets,
to source
• If source is random, some will go to 1/8, 2/8, 5/8,…
03:17:22: %SEC-6-IPACCESSLOGDP: list 105 permitted icmp 192.168.0.2
(Serial0/0 *HDLC*) -> 5.52.203.66 (0/0), 1 packet
03:17:38: %SEC-6-IPACCESSLOGDP: list 105 permitted icmp 192.168.0.2
(Serial0/0 *HDLC*) -> 1.167.111.47 (0/0), 1 packet
03:17:52: %SEC-6-IPACCESSLOGDP: list 105 permitted icmp 192.171.12.5
(Serial0/1 *HDLC*) -> 2.153.59.34 (0/0), 1 packet
…
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
44
Summary Tracing DoS Attacks
• Non-spoofed: Technically trivial (IRR)
But: Potentially tracing 100’s of sources…
• Spoofed:
IP Source Tracker: router by router
NetFlow:
Automatic if analysis tools are installed
Manually: Router by router
ACLs:
Has performance impact on some platforms
Mostly manual: Router by router
Backscatter technique:
One step, fast, only for spoofed sources
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
45
REACTING TO DOS ATTACKS
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
46
Response Options
• Wide range of response options exists
Access-control lists
QoS tools such as CAR
IDS, FWs, NBAR, etc.
• At an SP level we need to react with rapid,
widespread solutions:
BGP Triggers
Packet Scrubbing
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
47
Remotely Triggered Black Hole Filtering
• We will use BGP to trigger a network wide response
to an attack
• A simple static route and BGP will enable a
network-wide destination address black hole as fast
as iBGP can update the network
• This provides a tool that can be used to respond to
security related events and forms a foundation for
other remote triggered uses
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
48
Remote Triggered Black Hole
• Configure all edge routers with static route to null0
(must use “reserved” network):
ip route 192.0.2.1 255.255.255.255 null0
• Configure trigger router
Part of iBGP mesh
Dedicated router recommended
• Activate black hole
Redistribute host route for victim into BGP with next-hop set to
192.0.2.1
Use route-map set next-hop 192.0.2.1
Route is propagated using BGP to all BGP speaker and installed
on routers with 192.0.2.1 route
All traffic to victim now sent to null0
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
49
Using Remote Triggered Black Hole
• Is this done today?
Yes! Many service providers use this regularly
• Often the only scaleable answer to handle a large
scale DoS attack
Has proven very effective
• Inter-provider triggers not implemented
Rely on informal/formal channel
• Service: Customer Triggered
Customer trigger the update yourself, SP doesn’t get
involved
Implication: customer detects and classifies, etc.
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
50
Flipping It Around: Triggered Source Drops
• Dropping on destination is very important
Dropping on source if often what we really need
• Reacting using source address provides some
interesting options:
Stop the attack without blackholing real services
Filter command and control servers
Filter (contain) infected end stations
• Must be rapid and scaleable
Leverage pervasive BGP again
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
51
Loose uRPF Check
(Unicast Reverse Path Forwarding)
router(config-if)# ip verify unicast source reachable-via any
i/f 2
i/f 1
S D Data
i/f 3
FIB:
...
S -> i/f x
...
Any i/f:
Forward
SEC-2008
9822_05_2004_c2
i/f 2
© 2004 Cisco Systems, Inc. All rights reserved.
i/f 1
S D Data
i/f 3
FIB:
...
...
...
?
Not in FIB
or route null0:
Drop
52
Source-Based Remote Triggered
Black Hole Filtering
Uses the Same Architecture as DestinationBased Filtering + Unicast RPF
• Edge routers must have static in place
• They also require unicast RPF
• BGP trigger sets next hop—in this case the “victim”
is the source we want to drop
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
53
Source-Based Remote Triggered
Black Hole Filtering
• What do we have?
Black Hole Filtering—If the destination address equals Null
0 we drop the packet
Remote Triggered—Trigger a prefix to equal Null 0 on
routers across the Network at iBGP speeds
uRPF Loose Check—If the source address equals Null 0,
we drop the packet
• Put them together and we have a tool to trigger
drop for any packet coming into the network whose
source or destination equals Null 0!
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
54
Mitigation: Packet “Scrubbing”
• Use the same BGP mechanism to redirect traffic to
scrubbing devices
• Activate redirection:
Redistribute host route for victim into BGP with next-hop
set to scrubbing devices
Route is propagated using BGP to all BGP speaker and
traffic redirected
• When attack is over, BGP route can be removed to
return to normal operation
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
55
Re-Directing Traffic from the Victim
Other
ISPs
Ingress
Routers
• Keeps line to customer clear
• But cuts target host off completely
• Discuss with customer!!!
Target
Sink Hole Router:
Announces Route “target/32”
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
56
Cisco Guard: Packet Scrubbing
BGP Announcement
Riverhead
Guard
3. Divert Only Target’s Traffic
2. Activate: Auto/Manual
1. Detect
Cisco Anomaly
Detector, Cisco IDS,
NetFlow…
Target
Non-Targeted Servers
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
57
Cisco Guard: Packet Scrubbing
Riverhead
Guard
Traffic Destined
to The Target
4. Identify and Filter
the Malicious
Legitimate Traffic
to Target
5. Forward the Legitimate
Cisco Anomaly
Detector, Cisco IDS,
NetFlow,…
6. Non
Targeted
Traffic Flows
Freely
Target
Non-Targeted Servers
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
58
Remote Triggered Rate Limiting
• QPPB (QoS Policy Propagation via BGP)
• On each border router: Define a committed
access rate (CAR) policy, linked to a QoS group
(normally unused)
• To limit an attack, assign the network to the
QoS group (BGP community)
See: http://www.cisco.com/univercd/cc/td/doc/product/software/ios111/cc111/bgpprop.htm
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
59
Summary: Containing DoS Attacks
• Firewalls and such
Can be hard to deploy in SP topologies
More of an edge impact
• ACLs:
High granularity
Manual, potential performance impact
• BGP Triggers
Network-wide rapid reaction
Can be source or destination-based
Foundational techniques: enable others such
as scrubbing and QPPB
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
60
WRAP-UP
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
61
What We Can Do Now
• Detect DoS Attacks (SNMP, NetFlow, ACL)
• Trace back random packet floods
(NetFlow, ACLs, IP source tracker)
• Shun a source (uRPF, ACL)
• Shun a destination (routing, ACL)
• Limit attacking traffic (CAR, PIRC)
• Remote trigger via iBGP
• Use BGP for security in general
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
62
Recommendations for ISPs
• Preventative measures: ACLs, uRPF, CAR…
See ISP Essentials
• Monitor your routers and alarm on:
CPU, line load, memory…
• Use NetFlow plus collector s/w:
Usage statistics, DoS detection, DoS tracing through the
network
• Be prepared:
Technically: Understand the routers
Operationally: Have procedures in place, know your
upstream/downstream contacts, have a CERT
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
63
Some Loose Ends…
• ACLs with “log(-input)”: Nice, but CPU intensive!
• Debug ip packet: Your CPU hates you!!!
Always use with ACL
• Ip route x.x.x.x null0: Configure ICMP
rate-limiting or no ip unreachables!
• Routing security (MD5 auth): Push ISP to do it!!!
(more attacks on routing will come)
• Infrastructure security is critical
Protect your cores
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
64
Tip: Scheduler Allocate
• Schedules CPU time spent on processes versus interrupts
Syntax:
scheduler allocate <interrupt> <processes>
<interrupt>: 3000-60000 Microseconds Handling Network
interrupts
<processes>: 1000-8000 Microseconds Running Processes
Example:
router(config)#scheduler allocate 8000 8000
Default Values: 4000 200
Very Useful under Heavy Load!
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
65
What Will the Future Bring?
• More PCs always online (DSL, Cable)
The vulnerabilities are here!
Need quarantine and containment solutions
• More vulnerabilities and zombies?
• Better integration of detection and reaction
• Improved distinction of “good” from “bad” packets
• Increased infrastructure attacks
• More and more DoS: it pays well
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
66
Other Types of DoS :-)
“The first day of summer vacation is a DoS
attack against the highway system.”
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
67
THANK YOU! QUESTIONS?
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
68
References (Non-Cisco)
• DoS Detection:
“Tackling Network DoS on Transit Networks”: David Harmelin, DANTE, March 2001
(describes a detection method based on NetFlow)
[http://www.dante.net/pubs/dip/42/42.html]
“Inferring Internet Denial-of-Service Activity”: David Moore et al, May 2001;
(described a new method to detect DoS attacks, based on the return traffic from the
victims, analysed on a /8 network; very interesting reading)
[http://www.caida.org/outreach/papers/backscatter/index.xml]
“The spread of the code red worm”: David Moore, CAIDA, July 2001
(using the above to detect how this worm spread across the Internet)
[http://www.caida.org/analysis/security/code-red/]
• DoS Tracing:
“Tracing Spoofed IP Addresses”: Rob Thomas, Feb 2001;
(good technical description of using NetFlow to trace back a flow)
[http://www.enteract.com/~robt/Docs/Articles/tracking-spoofed.html]
• Other:
“DoS attacks against GRC.com”: Steve Gibson, GRC, June 2001 (a real life
description of attacks from the victim side; somewhat disputed, but fun to read!)
[http://grc.com/dos/grcdos.htm]
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
69
References (Cisco)
• Product Security:
Cisco’s Product Vulnerabilities; A page that every SE MUST know!!!
[http://www.cisco.com/warp/public/707/advisory.html]
Security Reference Information: Various white papers on DoS attacks and
how to defeat them [http://www.cisco.com/warp/public/707/ref.html]
• ISP Essentials:
Technical tips for ISPs every ISP should know
[ftp://ftp-eng.cisco.com/cons/isp/]
• Technical tips:
Troubleshooting High CPU Utilization on Cisco Routers
[http://www.cisco.com/warp/public/63/highcpu.html]
The “show processes” command
[http://www.cisco.com/warp/public/63/showproc_cpu.html]
NetFlow Performance White Paper
[http://www.cisco.com/warp/public/cc/pd/iosw/prodlit/ntfo_wp.htm]
• Mailing lists:
Cust-security-announce: All customers should be on this list
Cust-security-discuss: For informal discussions
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
70
Recommended Reading
• Cisco ISP Essentials
ISBN: 1587050412
• CCIE Practical Studies:
Security
ISBN: 1587051109
• Network Security
Principles and Practices
ISBN: 1587050250
AVAILABLE ONSITE AT THE CISCO COMPANY STORE
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
71
Complete Your Online Session Evaluation!
WHAT:
Complete an online session evaluation
and your name will be entered into a
daily drawing
WHY:
Win fabulous prizes! Give us your feedback!
WHERE: Go to the Internet stations located
throughout the Convention Center
HOW:
SEC-2008
9822_05_2004_c2
Winners will be posted on the onsite
Networkers Website; four winners per day
© 2004 Cisco Systems, Inc. All rights reserved.
72
SEC-2008
9822_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
73