Transcript Networking in Under 30 Minutes - University of Manchester

Networking in Under 30 Minutes !
Richard Hughes-Jones, University of Manchester
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
1
MB - NG
E-science core project
 Project to investigate and pilot:




end-to-end traffic engineering and management over multiple administrative
domains – MPLS in core diffserv at the edges.
Managed bandwidth and Quality-of-Service provision. (Robin T)
High performance high bandwidth data transfers. (Richard HJ)
Demonstrate end-to-end network services to CERN using Dante EU-DataGrid
and to the US DataTAG.
 Partners:CISCO, CLRC, Manchester, UCL, UKERNA plus Lancaster
and Southampton (IPv6)
 Status:






Project is running with people in post at Manchester and UCL.
Project Tasks have been defined and Detailed planning in progress
Kit list for the routers given to Cisco
Test PC ordered
UKERNA organising core network and access links – SJ4 10Gbit upgrade
Strong Links with GGF
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
2
MB - NG
MB – NG SuperJANET4
Development Network (22 Mar 02)
WorldCom
OC48/POSSR-SC
MAN
MCC
OSM-4GEWAN-GBIC
OSM-4GEWAN-GBIC
Leeds
SJ4 Dev
C-PoP
Warrington
12416
OSM1OC48POS-SS
Gigabit Ethernet
2.5 Gbit POS Access
2.5 Gbit POS core
MPLS Admin. Domains
RAL
RAL
OSM-4GEWAN-GBIC
OSM-4GEWAN-GBIC
OSM1OC48POS-SS
SuperJANET4
Production
Network
SJ4 Dev
C-PoP
Reading
12416
OC48/POSLR-SC
SJ4 Dev
C-PoP
London
12416
WorldCom
WorldCom
Dark Fiber (SSE)
POS
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
SJ4 Dev
ULCC
12016
OC48/POSSR-SC
ULCC
UCL
UCL
OSM-4GEWAN-GBIC
OSM-4GEWAN-GBIC
OSM1OC48POS-SS
3
Defining Background Traffic








Regular traffic - constant size packet, regular spaced in time
Poisson traffic - constant size, exponential spacing to form transient queues
IETF traffic mix - different sizes and different probability of each size sent
Play back of real traffic patterns generated from packet headers pre-recorded from suitable
points of the production network. This might include:
 Video Conference traffic -> play back - rude/crude tools
 UCL real conf playback tool
 General traffic captured at edge of a site, e.g. Manchester
 Do tests with a gen to see what gets dropped 0.5 Gbit typical peak UCL
Web-bursty traffic – web mirror – wget
Need to be able to reproduce traffic Statisticaly In general UDP best to understand the net
Consider UDP TCP flows
Need ToS / QoS to be set
How to control



Start Stop
measure load as function of time – links and Queues
Start and end numbers
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
4
Defining the Measurements (10)
 UDP round-trip latency vs packet size & Histograms



Sum of dt/dl transfer rates
Router & stack performance
Indication of network load & Switching / transport effects – spikes
 UDP 1-way delay
 UDP throughput vs Packet size and transmit delay


Throughput behaviour
Offered vs achieved throughput
 UDP Packet Loss vs transmit rate and burst size


Align Metrics with
GGF/IETF
Loss rate Packet loss distribution as function of time
Buffer sizes in the path & Detect packet re-ordering
 UDP Inter-frame Jitter as function of packet transmit spacing


Indication of network load
Behaviour of end system NICs
 TCP round-trip latency vs Message size & Histograms


Sum of dt/dl transfer rates
Stack / protocol performance – detect Packet size dependencies
 TCP throughput vs Message size and transmit delay


Throughput behaviour cf UDP
Packet loss distribution as function of time + Re-transmit rate
 TCP throughput vs Window size / TCP tuning
 TCP throughput vs number of streams


Stream throughput – benefits & effect on the network
Packet loss distribution as function of time + Re-transmit rate
 TCP Protocol behaviour - tcptrace
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
5
Defining the Measurements (11)
 TCP round-trip latency vs Message size




Sum of dt/dl transfer rates
(Router performance)
Stack / protocol performance
Detect Packet size dependencies
 TCP round-trip histograms



Indication of network load
(Switching / transport effects – spikes)
Stack / protocol performance
Align Metrics with
GGF/IETF
 TCP throughput vs Message size and transmit delay




Throughput behaviour cf UDP
Offered vs achieved throughput
Packet loss distribution as function of time + Re-transmit rate
Loss as func of pkt rate eg keep the data rate the same change pkt size – multi-streams
 TCP throughput vs Window size / TCP tuning
 TCP throughput vs number of streams



Stream throughput - benefits
Packet loss distribution as function of time + Re-transmit rate
Effect on Network
 TCP Protocol behaviour - tcptrace


What are the “burst” lengths
Effect of routers / end system NICs
 All this for WRED Wt fair Qing data rate const and ch pkt size – chack how well the routers do the
Qing
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
6
The EU DataTAG project





EU Transatlantic Girid project.
Status: Well under way – People in post, Link expected Jul 02
Partners: CERN/PPARC/INFN/UvA. IN2P3 sub-contractor
US Partners: Caletch, ESnet, Abilene, PPDG, iVDGL …
The main foci are:


Grid Network Research including:
• Provisioning (CERN)
• Investigations of high performance data transport (PPARC)
• End-to-end inter-domain QoS + BW / network resource reservation
• Bulk data transfer and monitoring (UvA)
Interoperability between Grids in Europe and the US
• PPDG, GriPhyN, DTF, iVDGL (USA)
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
7
DataTAG Possible Configuration
multi-platform multi-vendor
SLAC
UK
Fermi
NL
SuperJANET4
SURFne
t
Abilene
ESNET
GEANT
IT
GARRB
Alcatel
Cisco
Juniper
Giga
Switch
Juniper
Light
Switch
Juniper
Light
Switch
Alcatel
Giga
Switch
Cisco
6509
Cisco
Juniper
Starlight (Chicago)
2.5 Gbit PoS lambda
CERN (Geneva)
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
8
The SuperMicro P4DP6 Motherboard
 Dual Xeon Prestonia (2cpu/die)
 400 MHx Front side bus




Intel® E7500 Chipset
6 PCI-X slots
4 independent PCI buses
Can select:



64 bit 66 MHz PCI
100 MHz PCI-X
133 MHz PCI-X
 2 100 Mbit Ethernet
 Adaptec AIC-7899W dual channel SCSI
 UDMA/100 bus master/EIDE channels

data transfer rates of 100 MB/sec burst
 Collaboration:
Boston Ltd. (Watford) – SuperMicro Motherboards, CPUs, Intel GE NICs
Brunel University – Peter Van Santen
University of Manchester – Richard Hughes-Jones
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
9
Motherboard: SuperMicro P4DP6 Chipset: Intel E7500 (Plumas)
CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz Slot 4: PCI, 64 bit, 66 MHz
RedHat 7.2 Kernel 2.4.14
IntelPro1000 : P4DP6 64bit 66MHz PCI slot4
 Latency high but smooth
 Indicates Interrupt coalescence
 Slope 0.0136 us/byte, Expect:
PCI 0.00188
GigE0.008
PCI 0.00188
0.0118 us/byte
250
y = 0.0135x + 174.05
200
y = 0.0121x + 178.31
Latency us



Latency & Throughput:
Intel Pro/1000 on P4DP6
150
100
50
0
0
500
1500
2000
2500
3000
50 bytes
100 bytes
200 bytes
400 bytes
600 bytes
800 bytes
1000 bytes
1200 bytes
1400 bytes
1472 bytes
UDP IntelPro1000 : P4DP6 64bit 66MHz PCI slot4
1000
900
Recv Wire rate Mbits/s
 Max throughput 950Mbit/s
 Some throughput drop for packets
>1000 bytes
1000
Message length bytes
800
700
600
500
400
300
200
100
www.hep.man.ac.uk/~rich/net/nic/GigEth_
tests_Boston.ppt
0
0
5
10
15
20
25
30
35
40
Transmit Time per frame us
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
10
New External Connectivity
6 * 155 Mbit links
2.5Gbit line installed
IP commodity peer in London
Research traffic over 2.5G bit
Peer in Hudson St.
622 Mbit to Esnet
622 Mbit to Abilene.
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
11
Connectivity to Europe : Geant
Start mid November 2001
UKERNA switched off TEN-155 3 Dec 2001
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
12
Connectivity to Europe
ICFAMON Plot from DL to CERN for 18th Feb to 3rd Mar 2002
UK Dante Access link 2.5 Gbit POS
Remember 19th Oct to 1st Nov 2001
155 Mbit Access link over loaded
Sustained rate 130 Mbit
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
13
Monitoring: US Traffic
UKERNA Traffic data Kbit/s. Blue Traffic from US; Maroon Traffic to US
7 day periods 1 hour averages
14 Jan 2002 (800Mbit/s)
peak 86% of total 930 Mbit
17 Jan 2002
Peering altered 22 Jan
Weekend-Before
Weekday-After
Weekday-Before
22 Jan 2002
Weed day peak 175 Mbit/s
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
14
Monitoring: US Traffic
UKERNA Traffic data Kbit/s. Blue Traffic from US; Maroon Traffic to US
7 Dec 2001 (900kbit/s)
29 Jan 2002 (175kbit/s)
peak is 88% of total BW 930 Mbit
10 minute averages
10 minute averages
7 days 1 hour averages
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
15
DataGrid Network Monitoring
 Several tools in use – plugged into a coherent structure:
 PingER, RIPE one way times, iperf, UDPmon, rTPL, GridFTP, and
NWS prediction engine
 continuous tests for last few months to selected sites:
 DL Man RL UCL CERN Lyon Bologna SARA NBI SLAC …
 The aims of monitoring for the Grid:







to inform Grid applications, via the middleware, of the current status of the
network – input for resource broker and scheduling
to identify fault conditions in the operation of the Grid
to understand the instantaneous, day-to-day, and month-by-month
behaviour of the network – provide advice on configuration etc.
Network information published in LDAP schema – new self-defining
Cost Function in development – collaboration with WP1 & WP2
Will be used by UK GridPP and e-science sites and non HEP WPs
Links to the US
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
16
Network Monitoring Architecture
LDAP
Schema
Grid Apps
GridFTP
PingER
(RIPE TTB)
iperf
rTPL
NWS
etc
Local Network
Monitoring
Store & Analysis
of Data (Access)
Backend LDAP script to fetch metrics
Monitor process to push metrics
local
LDAP
Server
Grid Application access via
LDAP Schema to
- monitoring metrics;
- location of monitoring data.
Access to current and historic data
and metrics via the Web, i.e. WP7
NM Pages, access to metric forecasts
Robin Tasker
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
17
Network Monitoring Components
Clients WEB Display
Predictions
LDAP
Web I/f
LDAP
Table
plot
Grid Broker
LDAP
LDAP
Table
raw
raw
plot
Analysis
LDAP
Table
raw
plot
raw
raw
Scheduler
Cron
script
control
Cron
script
Cron
script
control
Tool
Ping
Netmon
UDPmon
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
iPerf
Ripe
18
Network Monitoring
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
19
Network Monitoring: Ping
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
20
Network Monitoring: Iperf (TCP)
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
21
HI Performance UDP Man - RAL Gigabit I/f
Manc 64bit 66 MHz PCI RedHat 7.1 Kernel 2.4.14 NIC: NetGear
RAL RedHat 7.1 Kernel 2.2.16-3 NIC: Intel pro 1000
 Latency 5.2 mS
 Slope 0.0904 us/byte
 or 2 PCs expect:
PCI
0.00188
GigE
0.008
PCI
0.00188
Total 0.0118 us/byte
Man-RAL Gig 20 Apr 02
7000
y = 0.0904x + 5271.8
6000
5000
4000
3000
2000
1000
 7 routers extra links
0.024
0.0064
0.012
0.0424
 Structure seen:


Period 88 bytes
Variation 150 – 190 us
 Max throughput 550Mbit/s
 Some throughput drop for
packets < 20 us spacing
0
0
500
1000
1500
2000
2500
3000
Message length bytes
50 bytes
100 bytes
200 bytes
400 bytes
600 bytes
800 bytes
1000 bytes
1200 bytes
1400 bytes
1472 bytes
UDP Man-RAL Gig 21 Apr 02
1000
900
Recv Wire rate Mbits/s
3 GigE
2 2.5G PoS
1 622Mbit
Total
y = 0.018x + 5409.1
Latency us


800
700
600
500
400
300
200
100
0
0
5
10
15
20
25
30
35
40
Transmit Time per frame us
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
22
HI Performance UDP Man - UvA Gigabit I/f
Manc Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset
CPU: PIII 800 MHz PCI:64 bit 66 MHz
RedHat 7.1 Kernel 2.4.14
NIC: NetGear
Man-UvA Gig 28 Apr 02
UvA RedHat 7.1 Kernel 2.2.16-3 ? 14850
14800
NIC: NetGear?
 Latency 14.68 mS
 Slope 0.006 us/byte
 For 2 PCs expect:
PCI
0.00188
GigE
0.008
PCI
0.00188
Total
0.0118 us/byte
 n routers extra links ??
 No Structure seen:
14750
Latency us
y = 0.0061x + 14675
14700
14650
y = 0.001x + 14681
14600
14550
14500
0
500
1000
1500
2000
2500
3000
Message length bytes
50 bytes
100 bytes
200 bytes
400 bytes
600 bytes
800 bytes
1000 bytes
1200 bytes
1400 bytes
1472 bytes
UDP Man-UvA Gig 28 Apr 02
1000
900
 Throughput 825Mbit/s 1400 bytes
 Some throughput drop for
packets < 20 us spacing
 Throughput 725Mbit/s 1200 bytes
Recv Wire rate Mbits/s






800
700
600
500
400
300
200
100
0
0
5
10
15
20
25
30
35
40
Transmit Time per frame us
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
23
iperf TCP & UDP throughput MAN-SARA
From 20 Oct 01
Iperf TCP throughput Mbit/s
ucl – sara
262144 byte buffer
Forecast
UDPmon throughput Mbit/s
man – sara
300 * 1400 byte frames
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
24
iperf & Pinger UK-Bologna
From 20 Oct 01
Iperf throughput
ucl – Bologna
262144 byte buffer
Forecast in green
PingER rtt (ms)
dl – Bologna
1000 byte packet
Forecast
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
25
UDPmon Loss
iperf throughput UCL-SARA
From 1 Nov 01 – Geant Operational
Throughput
Mbit/s
MAN – SARA
Iperf Throughput
Mbit/s
UCL – SARA
262144 byte
buffer
Geant Enabled
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
Routing Stable
26
100
Iperf mem-mem vs file copy disk to disk
Les Cottrell SLAC
Fast Ethernet
OC3 Over 60Mbits/s iperf >> file copy
Disk
limited
0
Iperf TCP Mbits/s
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
400
27
Don’t Forget Involvement with:
 GGF
 US Grids PPDG, iVDGL
 UKQCD UKDMC (dark matter) MINOS
 AstroGRID
 AccessGRID
 E-science Centres
 Optical “Lambda Switching” Projects
 Collaborations with UKERNA, Dante, Terena …
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
28
More Information Some URLs
PPNCG Home page with Stop Press:
http://ppncg.rl.ac.uk/
and e-mail:
[email protected]
DataGrid WP7 Networking:
http://www.gridpp.ac.uk/wp7/index.html
DataGrid WP7 EDG Monitoring:
http://ccwp7.in2p3.fr/
IEPM PingER home site:
http://www-iepm.slac.stanford.edu/
IEPM-BW site:
http://www-iepm.slac.stanford.edu/bw
GridPP Collaboration Meeting May 2002
R. Hughes-Jones Manchester
29