Transcript PowerPoint2000 - Computer Sciences Dept.

Distributed Network Monitoring in the
Wisconsin Advanced Internet Lab
Paul Barford
Computer Science Department
University of Wisconsin – Madison
Spring, 2002
Motivation
• Many applications that run over the Internet have
minimum performance requirements
• The network is one of the two possible sources of
poor performance
• Wide area network behavior is unpredictable
– IP networks are best effort
– Constant change is normal
• Quality of service capability is not widely deployed
– Will it ever be available?
[email protected]
2
Monitoring is a First Step
•
Accurate monitoring of network state can enable
application adaptivity and improved network
management
– Data provides basis for improved models and protocols
•
There are many challenges in network monitoring
– All features of the Internet make monitoring difficult
– When, where, what, how…
•
Today’s focus
1. Network monitoring efforts at Wisconsin
2. Combining monitoring and analysis to understand network
traffic anomalies
[email protected]
3
The Wisconsin Advanced Internet Lab
• Next generation environment for network
research
– Our focus: performance, management, security
– Platform for testbeds: storage, grid computing , …
• Internal environment
– Instances of end-to-end-through-core Internet paths
• External environment
– Measurement nodes deployed across the Internet
[email protected]
4
WAIL’s External Environment
• Existing infrastructure
– WAWM systems (10)
– Surveyor systems (60)
• Partnership with Advanced Systems
– NIMI systems (45)
• Partnership with PCS and ICIR
– Condor/Grid Infrastructures
• Prototype system is under development
• Passive flow measurements
– FlowScan data from UW, Internet2, others(?)
[email protected]
5
WAIL’s Internal Environment
• Complement to external facilities
• Hands-on test bed which creates paths identical to those in the
Internet from end-to-end-through-core
– Variety of highly configurable equipment
• Why do we need an internal lab?
– Enables instrumentation and measurement of entire end-to-end system
– Enables new systems and protocols to be implemented in places where
access is not possible in wide area
• Vision of internal lab: New means for doing network research
• Status: Significant commitment from industry partners (Cisco,
EMC, Fujitsu) and the university – rev. 1.0 by 5/1/02
[email protected]
6
Distributed Anomaly Detection
• Motivation: Anomaly detection and identification is an
important task for network operators
– Operators typically monitor by eye using SNMP or IP flows
– Simple thresholding is ineffective
– Some anomalies are obvious, other are not
• Focus: Characterize and develop distributed means for detecting
classes of anomalies
– Network outages, Flash crowds, Attacks, Measurement failures
• Approach: Use statistical and wavelet techniques to analyze
anomalies from IP flow and SNMP data from UW and other sites
• Implications: Tools and infrastructure which quickly and
accurately identify and adapt to traffic anomalies
[email protected]
7
Characteristics of “Normal” traffic
[email protected]
8
Our Approach to Analysis
• Analyze examples of each type of anomaly via
statistics, time series and wavelets (our initial focus)
• Wavelets provide a means for describing time series
data that considers both frequency and scale
– Particularly useful for characterizing data with sharp
spikes and discontinuities
• More robust than Fourier analysis which only shows what
frequencies exist in a signal
– Tricky to determine which wavelets provide best
resolution of signals in data
• We use tools developed at UW Wavelet IDR center
• First step: Identify which filters isolate anomalies
[email protected]
9
Analysis of “Normal” Traffic
• Wavelets easily localize familiar daily/weekly signals
[email protected]
10
Example Anomaly: Attacks
• DoS: sharp increase in flows and/or packets in one direction
• Linear splines seem to be a good filter to distinguish DoS attacks
[email protected]
11
Characteristics of Flash Crowds
• Sharp increase in packets/bytes/flows followed by
slow return to normal behavior eg. Linux releases
• Leading edge not significantly different from DoS
signal so next step is to look within the spikes
[email protected]
12
Characteristics of Network Anomalies
• Typically a steep drop off in packets/bytes/flows
followed a short time later by restoration
[email protected]
13
Summary and Conclusion
• Accurate network monitoring is essential for
improving application performance and network
management
• The Wisconsin Advanced Internet Lab provides a
unique environment for network monitoring
• Wavelets are an effective means for identifying
anomalous behavior in data gathered from IP flow
and SNMP interface monitors
– Details on distributed and coordinated monitoring and
analysis available this spring
[email protected]
14