Transcript Thoughts on TCP benchmarking

A WAN-in-LAB for
Protocol Development
Netlab, Caltech
Lachlan Andrew, George Lee,
Steven Low(PI), John Doyle, Harvey Newman
Outline





What and why is WAN-in-Lab?
What can I do with WiL?
Why would I use WiL?
How do I use WiL?
Future plans
What is WAN-in-Lab?

“Wide Area” Network in a laboratory
• Real fibre delays
• Carrier-class routers, switches, …
Why -- Spectrum of tools
cost
?
UltraLight
PlanetLab
Abilene
NLR
LHCNet
CENIC
etc
DummyNet
EmuLab
ModelNet
NS2
WAIL
SSFNet
QualNet
JavaSim
Mathis formula
Optimization
Control theory
Nonlinear model
Stocahstic model
abstraction
live netwk WANinLab emulation
simulation
maths
All scales are important– WAN-in-Lab fills a gap
What can I do with WAN-in-Lab?
Other groups’ interests

Protocol development
• FAST, delay-based
• MaxNet, explicit signalling
• ADPM, single-bit explicit signalling



Impact of small buffers (U. Pittsburgh)
Test automatic configuration of routers
(MonALISA, Ultralight)
Test distributed file-system (MojaveFS)
TCP Benchmarking

Our current main direction
Evaluating others’ protocols, not ours

Web interface

• Submit kernel patch
• Standard tests automatically performed
• Results mailed back

Explicit or implicit signalling protocols
Physical topology
Svr2
Eth2
10.2.12.2
Svr10
Eth2
10.2.21.2
Svr1
Eth2
10.2.11.2
POS1/1
10.0.12.2
Eth2
10.1.12.2
Svr9
Svr14
Eth2
10.2.24.2
POS9/1
10.0.24.2
POS1/1
10.0.24.4
RB POS2/1
Gi2/2 10.0.23.2
10.2.22.1
Eth2
10.1.12.2
POS1/1
10.0.12.1 POS9/1
10.0.13.1
POS1/1 RD
10.0.24.4
POS1/1
10.0.23.3
Eth2
10.3.11.2
POS9/1
10.0.13.3
Disk4
RA
eth2
10.1.13.2
Svr13
RC
Eth??
10.1.51.2
SA:gi1/0/12
Eth1
10.3.12.2
Eth??
10.1.13.2
Disk1
Svr6
Eth2
10.3.12.18
Svr7
Eth2
10.3.12.17
Svr11
Disk2
Disk3
Capabilities: Delay

24 spools of 100km fibre, many loopbacks
• Set delay by MEMS switching loops in/out

130ms physical delay
• more with IP loopback

2 Dummynets: long delay for cross-traffic
125 ms, 1.8ms steps
External connections

Linked to Ultralight, 10Gbps Physics WAN
Smooth migration testing -> deployment

Delay

• longer
• jitter


Cross traffic
Monitor
data routed
through WiL
Why use WAN-in-Lab?
Why use WiL?

Complement other levels of abstraction,
not replace them

Different ways to use it: reasons for each

Standard platform for TCP benchmarking
• Easier to compare with others’ results
• No need to write your own test suite
Artifacts of software delays


Packets sent on 1ms “ticks”
1Gbps = 83,333 pk/s
83 packets
1ms
How can I use WAN-in-Lab?
Management structure
Wil-ns.cs.caltech.edu
Start script, configure, compile
Servers
Network boot
Read-only FS
virtual /etc
scratch disk
Data plane
Time sharing

Coarse switching between projects
• Servers rebooted, routers reconfigured


Switchover takes ~5 minutes
Book in advance
• For longer bookings, book further in advance
• Also “ad hoc” bookings for individual hosts

Can log in while others have booked
Future plans
Future plans

Benchmarking infrastructure
• Standardise tests
• Use it ourselves
• Develop “indices” of TCP performance


Better control over capacities and buffers
Better cross-traffic generation
• Currently Harpoon


Investigate differences from DummyNet
Integrate DAG cards
Conclusion


WAN-in-Lab fills the gap between
emulation and live network experiments
Seeks to be as realistic as possible
• Long links, simple topology

Focus will be on TCP benchmarking

We welcome people to use it
<http://wil.cs.caltech.edu>
Spare Slides
Case Study: MaxNet
Aim: Wind Tunnel of Networking

WAN in Lab
• Capacity: 2.5 – 10 Gbps
• Delay: 0 – 120 ms round trip

Breakable
• Won’t take down live network

Flexible, active debugging
• Passive monitoring, AQM

Configurable & evolvable
• Topology, rate, delays, route
• Modular design stays up to date

Integral part of R&A networks
• Transition from theory, implementation, demonstration,
deployment
• Transition from lab to marketplace

Global resource
• Part of global infrastructure UltraLight led by Harvey Newman
Equipment







4 Cisco 7609 routers with OC48 line cards
6 Cisco ONS 15454 switches
A few dozen high speed servers
1G switch to routers/servers
Calient switch for OC48
2,400 kilometres of fibre, optical
amplifiers, dispersion compensation
modules
63ms aggregate RTT delay, in two hops
• 120ms using IP loopbacks
Accounts


Mail wil at cs.caltech.edu
Sudo access to “network” commands
• Ifconfig/…/
• Custom commands to set topologies

Login to routers if required

Separate accounts for “benchmark only”
Configuration -- Delays

Want maximum delay from limited fibre
• Signals traverse fibre 16 times

4 WDM wavelengths
4 OC48 (2.5G) MUXed onto OC192 (10G)

Lots of transponders

• WDM amplifier joins 100km spools  200km
Configuration – delays
OC48 slot
16x200km
-------WDM Wavelength--------
Bidirectional 100km
Bidirectional 100km
Amp
Configuration – delays



Delay varied by adjusting the number of
OC48 hops traversed
Calient optical switch selects required
hops
Hop lengths 200km up to 1600km
• Maximise granularity given limited switch ports
Switch
Projects


TCP benchmarking
FAST
• Delay-based congestion control

MaxNet
• Explicit signalling congestion control

MojaveFS
• New distributed file system

University of Pittsburgh
• TCP with small buffers

University of Melbourne
• Single-bit congestion marking
WAN-in-Lab testbed



Dummynet and simulation introduce artifacts
Also need to test on real equipment
WAN with real delays, located in a single room
• Connected to an external WAN (Ultralight)

Open for the community to use for benchmarking
OC-48
OC-48
WAN-in-Lab capabilities
Current
Planned
Two 2.5G bottlenecks
Multiple 1G bottlenecks
Six 2.5G bottlenecks
Two “real” delays
Up to six “real” delays
(Emulate cross traffic delay)
End-to-end RTT, drop
Per-router delay, drop
(movable DAG cards)
Configuration -- delays
OC48 slot
-------WDM Wavelength--------
Bidirectional 100km
Bidirectional 100km
Amp
Using WAN-in-Lab

Contact me – lachlan at caltech . Edu

Coarse timesharing
• Some users set up experiments while others
run experiments

Software setup still being developed
• Your chance to influence our directions to tailor
it to your needs
Sample MaxNet results

Achieves realistic delay at 1Gbit/s