Transcript Internet Measurement Initiatives in the Wisconsin

Internet Measurement Initiatives in the
Wisconsin Advanced Internet Lab
Paul Barford
Computer Science Department
University of Wisconsin – Madison
Spring, 2003
Talk Objectives
• Motivate and describe Wisconsin Advanced
Internet Lab (WAIL)
– Internal lab environment
– External lab environment
• Provide some detail on three current projects
– Anomaly detection and characterization
– Distributed intrusion monitoring
– Understanding packet loss
[email protected]
2
Motivation for New Tools
• Any area of scientific research is limited by the tools
available for experimental study
– “If your only tool is a hammer then everything looks
like a nail”
• 2001 NRC report: “network research community is in
danger of ossification due to strictures of experimental
systems”
– Challenge: “Capturing a day in the life of the Internet”
• New experimental tools can open up areas of research
that have not previously been accessible
[email protected]
3
An Internet Instance Lab
• A hands-on test environment designed to recreate
paths and conditions identical to those in the Internet
from end-to-end-through-core
– Requires large amount of routing and end host equipment
• Network and host equipment able to recreate (not
emulate) a wide range of services, configurations and
traffic conditions
– Complete instrumentation of end-to-end paths
– Deployment of disruptive prototypes
[email protected]
4
Key Challenges
•
•
•
•
•
Design
Configurations and management
Traffic generation
Propagation delay
Validation
[email protected]
5
The Wisconsin Advanced Internet Lab
• Our realization of an IIL
• Developed over past 18 months by UW/Cisco team
• Supported by $3.5M equipment grant from Cisco and
UW matching funds
– Used to purchase over 75 pieces of networking equipment
• Phase 1 nearing completion => Abilene recreation
• Other partners: EMC, Spirent, Intel, Fujitsu, Sun
• Research initiatives in many areas…
[email protected]
6
External Environment
• Essential complement to internal environment
• Existing infrastructure
– DOMINO systems (1 class A + 2 class B’s + Dshield)
– Surveyor + WAWM systems (~70 nodes)
• New database and front end by summer ‘03
• Partnerships and other available systems
– Condor/Grid Infrastructures
• Passive flow measurements
– FlowScan data from UW, Internet2, others…
[email protected]
7
Project 1: Detecting Anomalies in IP
Flows
• Motivation: Anomaly detection remains difficult
• Objective: Improve understanding of traffic anomalies
• Approach: Multiresolution analysis of data set that
includes IP flow, SNMP and an anomaly catalog
• Method: Integrated Measurement Analysis Platform for
Internet Traffic (IMAPIT)
• Results: Identify anomaly characteristics using wavelets
and develop new method for exposing short-lived events
[email protected]
8
Our Data Sets
• Consider anomalies in IP flow and SNMP data
– Collected at UW border router (Juniper M10)
– Archive of ~6 months worth of data (packets, bytes, flows)
– Includes catalog of anomalies (after-the-fact analysis)
• Group observed anomalies into four categories
– Network anomalies (41)
• Steep drop offs in service followed by quick return to normal behavior
– Flash crowd anomalies (4)
• Steep increase in service followed by slow return to normal behavior
– Attack anomalies (46)
• Steep increase in flows in one direction followed by quick return to normal
behavior
– Measurement anomalies (18)
• Short-lived anomalies which are not network anomalies or attacks
[email protected]
9
Multiresolution Analysis
• Wavelets provide a means for describing time series
data that considers both frequency and time
– Powerful means for characterizing data with sharp spikes
and discontinuities
– Using wavelets can be quite tricky
• We use tools developed at UW which together make
up IMAPIT
– FlowScan software
– The IDR Framenet software
[email protected]
10
Ambient IP Flow Traffic
[email protected]
11
Flow Traffic During DoS Attacks
[email protected]
12
Deviation Score for Three Anomalies
[email protected]
13
Project 2: Coordinated Intrusion
Detection
• Motivation: Intrusion detection is a moving target
• Objective: Coordinate intrusion monitoring between
multiple sites around the Internet
• Approach: Share data from firewalls, NIDS and tarpits
(on unused IP space)
• Method: Distributed Overlay for Monitoring Internet
Outbreaks (DOMINO)
• Results: Blacklists can be rapidly generated, false
positives can be substantially lowered, new outbreaks
can be easily identified
[email protected]
14
DOMINO: A new approach to DNIDS
• Partnership with dshield.org
– 1600 firewall and NIDS logs
• Tarpits
– Active monitor of unused IP space
– 1 class A (this week), 2 class B’s
• A protocol for node participation, data sharing and
alert clustering
– Chord-based overlay network
– Extension of Intrusion Detection Message Exchange
Format
– Various clustering methods
[email protected]
15
Marginal Utility of Adding Nodes
[email protected]
16
SQL-Sapphire Analysis
[email protected]
17
Project 3: Understanding Packet Loss
• Motivation: Many of the most basic aspects of packet loss are
not understood
– Where, when, how long, how often?
• Focus: Developing a comprehensive understanding of packet
loss in the Internet
• Approach: Combine understanding of protocols and queue
behavior to create a probe train which can accurately measure
delay and loss.
• Implications: End-to-end tools for pin-pointing loss, better
transport protocols, better network management for congestion
[email protected]
18
Active versus Passive Loss Measures
• Hypothesis: Active measures of loss are
correlated with passive measures of loss
• Assessment in Abilene
– SNMP loss measures on all backbone routers
– Active probes via Ping/Zing in Surveyor nodes at
10Hz, 20Hz and 100Hz
– Tests in full mesh over one month period
[email protected]
19
Result: Active <> Passive
[email protected]
20
Summary
• Both internal lab building initiatives and
external measurement initiatives in WAIL
• Internal facilities are intended to be open
• We are seeking partnerships in external
measurement projects.
– DOMINO in particular
[email protected]
21