Transcript Anukool Lakhina Presentation

Mining Anomalies in
Network-Wide Flow Data
Anukool Lakhina
with Mark Crovella and Christophe Diot
NANOG35, Oct 23-25, 2005
My Talk in One Slide
• Goal: A general system to detect & classify
traffic anomalies at carrier networks
• Network-wide flow data (eg, via NetFlow)
exposes a wide range of anomalies
– Both operational & malicious events
• I am here to seek your feedback 
2
Network-Wide Traffic Analysis
• Simultaneously analyze traffic flows across the
network; e.g., using the traffic matrix
• Network-Wide data we use: Traffic matrix
views for Abilene and Géant at 10 min bins
3
Power of Network-Wide Analysis
Peak rate: 300Mbps;
Attack rate ~ 19Mbps/flow
IPLS
NYC
LA
ATLA
HSTN
Distributed Attacks easier to detect at the ingress
4
But, This is Difficult!
How do we extract anomalies and normal behavior
from noisy, high-dimensional data in a systematic manner?
5
The Subspace Method
[LCD:SIGCOMM ‘04]
• An approach to separate normal & anomalous
network-wide traffic
• Designate temporal patterns most common to all
the OD flows as the normal patterns
• Remaining temporal patterns form the
anomalous patterns
• Detect anomalies by statistical thresholds on
anomalous patterns
6
An example user anomaly
One Src-Dst Pair Dominates:
32% of B, 20% of P traffic
Cause:
Bandwidth Measurement using
iperf by SLAC
7
An example operational anomaly
Multihomed customer CALREN reroutes
around outage at LOSA
8
Summary of Anomaly Types Found
[LCD:IMC04]
False
Alarms
Unknown
Traffic Shift
Outage
Worm
Point-Multipoint
Alpha
Flash
Events
DOS
Scans
9
Automatically Classifying Anomalies
[LCD:SIGCOMM05]
• Goal: Classify anomalies without restricting
yourself to a predefined set of anomalies
• Approach: Leverage 4-tuple header fields:
SrcIP, SrcPort, DstIP, DstPort
– In particular, measure dispersion in fields
• Then, apply off-the-shelf clustering methods
10
Example of Anomaly Clusters
Dispersed
Legend
(DstIP)
Code Red
Scanning
Single source
DOS attack
Multi source
DOS attack
(SrcIP)
(SrcIP)
Concentrated
Summary:
Correctly classified 292 ofDispersed
296 injected anomalies
11
Summary
• Network-Wide Detection:
– Broad range of anomalies with low false alarms
– In papers: Highly sensitive detection, even when
anomaly is 1% of background traffic
• Anomaly Classification:
– Feature clusters automatically classify anomalies
– In papers: clusters expose new anomalies
• Network-wide data and header analysis are
promising for general anomaly diagnosis
12
More information
• Ongoing Work: implementing algorithms in
a prototype system
• For more information, see papers & slides at:
http://cs-people.bu.edu/anukool/pubs.html
• Your feedback much needed & appreciated!
13