Transcript Case studies

Case Studies
Esmaeil Biazar Amlashi
ECE Department, University of Tehran
Fall 2009
Outline


Low-level Monitoring Tools
Individual Toolsets for Network Measurement





Windmill
Click
Dss
Gigascope
Large-scale Measurement Projects




Chapter 10
RIPE
High-energy Physics
CAIDA
PlanetLab
Case Studies
Low-level Monitoring Tools
Commercial off-the-shelf hardware is the preferred way
to capture high-speed traffic in busy locations before
handing the packet collection to software for processing.
Chapter 10
Case Studies
Low-level Monitoring Tools
Set of design issues involve:

Ability of accepting large volumes of output from the
hardware devices.

Filtering the traffic selectively based either on prespecified or dynamically changing set of rules.

Transforming output to enable subsequent processing by a
variety of downstream applications.

Sorting the contents.
Chapter 10
Case Studies
Low-level Monitoring Tools

Interesting way:
Writing an entirely generic piece of software
that can perform all the roles mentioned.
Chapter 10
Case Studies
Low-level Monitoring Tools
Passive monitoring of protocols simultaneously at high
speeds requirements:

Ability to monitor/capture the traffic and write it to disk.

Fine-grained time stamping to facilitate subsequent reassembly
of the traffic.

Limiting amount of CPU resources that can be expended in
transforming the captured traffic.
Chapter 10
Case Studies
Low-level Monitoring Tools
At the lower level:

Choice of operating system

Number of kernel receive buffer

Clock granularity

Scheduling priorities
Chapter 10
Case Studies
Individual Toolsets for Network
Measurement

Windmill

Click

DSS

Gigascope
Chapter 10
Case Studies
Windmill

Windmill is primarily a passive measurement tool
targeted at protocol performance.
Chapter 10
Case Studies
Windmill
It can be used with external active probing tools without
requiring modifications to any substrate.
Functional components of Windmill:



Chapter 10
A protocol filter
Abstract protocol modules
Extensible engine based on dynamic loader
Case Studies
Windmill (protocol filter)
Complex packet matching can be done efficiently at
the lower levels of the machine.
The protocol filter moves code into the kernel.
1.
2.
Chapter 10
The filter constructs an intermediate representation of
the packets.
Allows simultaneous experiments to access this
intermediate representation.
Case Studies
Windmill (set of protocol modules)



One for each of the key internet protocols(IP, TCP,
BGP, HTTP, etc.)
They are set up in a way that network events can be
mapped easily for reconstruction at higher layers.
They are capable of exporting the lower-level events
and contents of data structures to multiple
experiments simultaneously.

Chapter 10
Reducing redundant work and increasing efficiency
Case Studies
Windmill (set of protocol modules)
One example:
TCP module will reassemble the packet stream once
and make the reassembled stream available to
multiple experiments.

Chapter 10
eliminating expensive redundant work.
Case Studies
Windmill (extensible engine)

Extensible experiment engine uses a dynamic
loader which permits modifications during an
experiment and thinning when portions are no
longer required.
Chapter 10
Case Studies
Windmill
Ability to carry out multiple experiments, with
some results shipped to remote locations on
demand, allows this to meet this goal.
Chapter 10
Case Studies
Click

Click router allows for assembling complex configurations
from simple part pieces that carry out specific router
functions such as classifying packets, scheduling, etc.

By opening up the functions of a router and mapping
modules(elements) to each of the major functions.

Click allows for arbitrarily complex router configurations.
Chapter 10
Case Studies
Click
Classes of elements include:




Chapter 10
Packet sources
Sinks
Modifiers
Checkers(checking validity of packets)
Case Studies
Click(Advantages)

There are storage elements for storing packets for later use and
elements to schedule allowing packets to be chosen from
different sources.

Users can create their own libraries of parts of configurations
to be specified.

Unlike commercial routers, new Click modules can be written to
extend the behavior of routers.
Chapter 10
Case Studies
Click(Advantages)

Click has a toolkit for network address translation that can
actually be used for all address and port number rewriting.

Click has a building block approach lending itself to both
passive and active measurements.

Ability to aggregate packets as it processes them and
subsequently analyze flows to produce a digest.
Chapter 10
Case Studies
DSS
Data Stream Scanning tool(DSS) is an example of the
traditional UNIX library approach, playing primarily a
backend role for processing large amounts of data with
minimal effort on the part of the programmer.
Chapter 10
Case Studies
DSS

The essence of DSS is its data abstraction model.

Some components are part of the default dss library,
but most are implemented as independent dss API
dynamically loaded library(DLL).

Applications link against the base dss library and thus
old applications have access to new methods without
requiring recompilation.
Chapter 10
Case Studies
DSS component architecture
Data
Chapter 10
T
R
A
N
S
F
O
R
M
S
D
O
M
A
I
N
S
Case Studies
T
Y
P
E
S
Q
U
E
R
I
E
S
DSS (Components)

Transforms :

Transform class components serve as generic data-independent filters of
raw data.

Compression transforms are available for numerous compression
variants, such as gzip, pzip, bzip and compress.

A transform can be implemented within the main process or as a
separate process.

Transforms allow data to be stored in the most efficient or secure
manner and only need to be converted when accessed.
Chapter 10
Case Studies
DSS (Components)

Domains:

A method describe the data records for a specific data domain.

This included a dictionary of file storage formats and record field types.

The storage format dictionary has and identification function to
automatically determine input formats.

Chapter 10
Record read and write functions.
Case Studies
DSS (Components)

Types:

It provides functions to convert data between internal and external data
representations.

Chapter 10
Types are method independent and are often shared between methods.
Case Studies
DSS (Components)

Queries:

The user visible part of dss, is used to select, filter, and summarize data
stream records.

Interpreted queries are C-style expressions method.

The interpreted queries permit experimentation on small sets of data
with quick answers to different queries. the user can then switch to a
dynamic query for large data sets.
Chapter 10
Case Studies
DSS (Components)
Queries:

A dynamic query is executed within the dss process and has four
functions:

init called once before the first record is read, to allocate private data to be used
by the other functions.

select called for each record which on a successful return calls the act function
and terminates the scan otherwise..

done called once after the last record is read to list reports or summaries, and
release any private resources allocated by the other functions.
Chapter 10
Case Studies
DSS (Components)
Queries:

Chapter 10
Several generic dynamic queries are part of dss:

count : to count the number of selected and total records.

computes count

mean

unbiased standard deviation: for the named numeric field arguments.

Supporting grouping by field value.
Case Studies
DSS (Components)
examples:
ipaddr_t type match function does IP prefix matching:
(src_addr = ~ “12.34/16”)
 Selects all records where src_addr matches the IP prefix
12.34/16.
(path=~ “^123 [456 789] – 8765$)
 Selects all records where path starts with AS 123 followed by 456 or
789 and ends with 8765.
Chapter 10
Case Studies
Gigascope
Gigascope is a data stream management system
targeted at monitoring data streams at high
speeds.
Chapter 10
Case Studies
Gigascope

Gigascope is a combination of tailored hardware and database
software which has been used in passive data capture at high
speeds and active processing of specified standing queries and
user-specified ones.

It has been used in a wide variety of applications such as
analyzing traffic, intrusion detection, and protocol analysis.
Chapter 10
Case Studies
Large-scale Measurement Projects

RIPE

High-energy Physics

CAIDA

PlanetLab
Chapter 10
Case Studies
RIPE

The RIPE Network Coordination Center (RIPE NCC) is a
nonprofit organization established in 1992.

it provides technical coordination for the Internet in Europe,
Middle East, and parts of Central Asia).
Chapter 10
Case Studies
RIPE
RIPE has measurement projects in the following area:

Growth and change of the Internet. RIPE’s hostcount project is relevant
to this question.

Interaction of traffic and Network. This is addressed in RIPE’s Test
Traffic project.

Passive BGP Measurement. These measurements are taken in RIPE’s
Routing Information Service Project.
Chapter 10
Case Studies
Hostcount
The goal of this project is simply to provide
information to the Internet community as a measure
of infrastructure growth.
Chapter 10
Case Studies
Hostcount
The method is used by RIPE starts with each of the TLDs in the RIPE region.
1.
The zone information inside each TLD is collected.
2.
If the are sub-zones within a zone, those are collected as well, until the lowest
level is reached.
3.
Eventually zone downloads consist of A-record, which are the records within
DNS that assign a name to a host.
4.
Only unique names are counted.
5.
multiple names may be assigned to the same host, and these are only counted
once.
RIPE performs this process each month and publishes the results.
Chapter 10
Case Studies
Test Traffic Measurements

The project started in 2000; it currently involves
around 100 hosts.

TTM project measures one-way delays between
hosts(latency), packet losses, path
information(‘traceroute’), and delay variation(jitter).
Chapter 10
Case Studies
Test Traffic Measurements

The goal of the project is to provide participating
organizations with measures of quality of connection
to the Internet, means for diagnosing performance
problems, and measures of long-term trends in
network connectivity.
Chapter 10
Case Studies
Test Traffic Measurements
The project consists of two components:


Chapter 10
A dedicated measurement system(a TTM system) which is
designed and configured by RIPE, sent to a participating
institution or network, and connected to the network by the
receiver.
An operations center at RIPE which provides central storage
of all measurements with secure access to participating
organizations.
Case Studies
Test Traffic Measurements

Periodic measurements

Results are uploaded to the RIPE operations center

Results are available to users in various formats: graphs,
charts,….
Chapter 10
Case Studies
Test Traffic Measurements

At current time the RIPE TTM project is the largest
time-synchronized Internet measurement system.

the RIPE TTM systems are designed to
wide variety of active measurements:
between traffic and network(delay,
topology
measurements,
and
measurements.
Chapter 10
Case Studies
take a very
interactions
loss, etc.),
bandwidth
Routing Information Service

RIS started as a RIPE project in 1999.

The project collects and stores BGP routing
information and makes it publicly available for the
Internet community.
Chapter 10
Case Studies
Routing Information Service
The goal of the RIS project is to help network
operators troubleshoot routing problems by providing
BGP data collected over time, without being limited
to a single BGP veiw.
Chapter 10
Case Studies
Routing Information Service

The system implemented by placing ‘passive’ BGP
listeners at peering points and establishing BGP
connections to BGP routers present amount the
networks that peer at that point.

It operates in a manner similar routeviews project
described in Chapter 5.
Chapter 10
Case Studies
Routing Information Service

They collect two kinds of data:
1.
All BGP packets received: route announcements, withdrawals, and
other messages. This voluminous data is stored in 15-minute files.
2.
The entire BGP routing table maintained by Zebra. These tables are
created every 8 hours.
Chapter 10
Case Studies
Thanks
?
Chapter 10
Case Studies