Transcript New Directions in Traffic Measurement and Accounting

New Directions in Traffic
Measurement and Accounting
Cristian Estan
(joint work with George Varghese)
The Problem
• Measurement and monitoring of network
traffic required for Internet backbones.
• Useful for short-term monitoring (e.g., DOS
attacks), traffic engineering (e.g., rerouting),
and accounting (e.g., usage based pricing)
• How can we do so without “tracking
millions of ants to track a few elephants”?
State of the art – Cisco NetFlow
• Sample packets at high speeds;
• Per flow information based on samples;
• Aggregate (based on ASes, prefixes, ports) at
the router;
• Problems: inaccurate (due to sampling and
loss), memory-intensive, slow (needs DRAM).
Towards a NetFlow Alternative
• Small Percentage of flows (elephants)
account for large percentage of traffic.
• Top 9% of flows account for 90% of AS
pair traffic in backbones (Fang-Peterson).
• Can we directly track flows that send say
over 1% of link bandwidth without keeping
track of all flows?
How to identify large flows?
We introduce two new methods for this purpose:
• Sample-Counting: uses sampling only to
decide which flows to watch exhaustively.
• Multistage filter: uses multiple hash tables
allowing large flows to be identified while
only allowing a small number of small
flows (false positives) to pass through filter.
Identify large flows by
sampling
Multistage filters
Operation of Sampled NetFlow
How accurate is Sampled NetFlow?
1 gigabyte/100 megabytes of data
Sampling one in 100 packets
Error
1GB
100MB
1%
39.24%
79.03%
3%
1.07%
41.48%
Operation of our algorithms
Error
1GB
100MB
1%
5.6E-32
0.09%
3%
1.8E-94
6.96E-10
Sampling 1/1000
Error
1GB
100MB
1%
0.08%
49.69%
3%
4.66E-10
12.25%
Filter error: 0%
Comparison
Accuracy
Sampled
NetFlow
Medium
False neg.
Few (high var.) Few (low var.) Never
False pos.
Few
Few
Few
Memory
Big
Small
Very small
Low
Medium
Complexity Low
Identify by
sampling
Good
Multistage
filters
Good
High Speed Implementation?
• John Huber of MMC Networks did a design
of a chip doing filter counting.
• 450,000 transistors, under 1 watt of power,
runs at OC-192 rates, 32 nsec per packet
• Seems easily feasible to implement sample
counting with similar complexity.
Potential Application: scalable
threshold accounting
• Measure flows sending over x% of link
bandwidth using sample/filter counting.
• Bill using flat fee + per byte charge for
flows over x%
• Track aggregates directly to avoid evasion
using several flows, each < x%
• Generalizes usage based (x = 0) and
duration based (x = 100) pricing.
Conclusions
• Paradigm shift for measurement by
concentrating only on “heavy flows”
• Two new techniques (sample and filter
counting) with small memory footprints and
provable performance.
• Techniques make “threshold accounting”
feasible, generalizing usage and duration
based pricing.