Transcript No Slide Title

Testing and Modeling Ethernet
Switches and Networks for Use
in ATLAS High-level Triggers
R.W. Dobinson, S. Haas, K. Korcyl, M.J. LeVine,
J. Lokier, B. Martin, C. Meirosu, F. Saka, K. Vella
Copyright© 2000 OPNET Technologies, Inc.
Overview
• Ethernet as a candidate technology for ATLAS LVL2
• ATLAS - the networking requirement
• Parameterized model of an Ethernet switch
• Large network modeling results using the parameterized switch model
• GBE measurements
• Fast Ethernet tester/ROB emulator
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
2
ATLAS LVL2 system network
1564 buffers (distributed 2 Mbyte image)
Readout buffers
(ROB)
Fast Ethernet
Concentrating
switches
Gigabit
Ethernet
CENTRAL Gigabit switch
Trigger rate
100 kHz
Concentrating
switches
Fast Ethernet
Processing nodes
~500 500MIPS processors analyze data from 5% of buffers
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
3
ATLAS LVL2 network traffic
• Network nodes
• 1564 ROBs (Readout Buffers )
• 500-600 Processor CPUs
• Message scenario
• LVL1 accepts @ 75 kHz [scalable to100 kHz]
• Supervisor CPU selects processor CPU for each event accepted by LVL1
• Processor CPU sends ROB_REQUEST to selected (~5%) of ROBs
• ROBs reply with ROB_REPLY containing designated ROI data
• Traffic @ ROB: ~11000 request-response/sec
• Worst case data rate: 4 MB/s
• Traffic @ processor node: ~14000 request-response/sec
• Average data rate: 6.6 MB/s
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
4
Modeling
• Parameterized model of a switch

Motivation • Limited number of physical parameters
• Fast execution
• Direct determination by measurement
 commodity Ethernet switches
 parameterized model of a switch
 comparison of parameterized model and results from measurements
• Parameterized model in large Ethernet network


ATLAS HEP experiment at LHC in CERN
second level trigger (LVL2) for ATLAS experiment
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
5
Commodity Ethernet switches
Most of the Ethernet switches on the
market today have a hierarchical
architecture
• ports
• modules
• backplane
chassis
backplane
Store-and-forward mode of operation
• frame is fully stored in module with
•
an input port
for inter-module transfer the frame is
subsequently stored in a module with
an output port
module
module
module
ports
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
6
Parameterized model for switch:
inter-module communication
P3
Input buffer P1
MAC
P4
P7
P7
Output buffer P2
MAC
Backplane
Buffer manager
P6
Input module
Buffer manager
Output module
Parameters:
P1 - Input Buffer Length [ #frames]
P2 - Output Buffer Length [ # frames]
P3 - Max ToBackplane Throughput [MB/s]
P4 - Max FromBackplane Throughput [MB/s]
P6 - Max Backplane Throughput [MB/s]
P7 - Inter-module Transfer Bandwidth [ MB/s]
P9 - Inter-module Fixed Overhead [µs] (not shown)
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
7
Switch parameters from latency and rate
measurements
•Measure latency as function of packet size (ping-pong)
•Measure packet rate as function of packet size
•No switch, intra-module, inter-module
Setup of the
mea surement
Direct connect
What is
mea sured
Latency L
Intra-m odule
Latency L
Inter-m odule
Latency L
Direct connect
A chieve drate
Intra-m odule
A chieve drate
Inter-m odule
A chieve drate
DAQ 2000 Oct 2000
What can be learned from the
mea surements
The PC s’ softw are overhe ad
Con stant overh ead P10
and transfer bandwidth P8:
Con stant overh ead P9
and transfer bandwidth P7:
Ma ximu m throughput
th e PC s c an g enerate/ab sorb
Ma ximu m intra-m odule
throughput: P5
Ma ximu m inter-m odule
throughput: P3, P4, P6
Testing & Modeling Ethernet Switches for ATLAS HLT
8
Modeling calculations
• Switch parameters determined from simple ping-pong and
streaming measurements described
• Switch model implemented as C++ code
• Event-driven simulations
• Object-oriented switch model embedded in OPNET (commercial package)
• Calculations also carried out using the same model within Ptolemy
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
9
parameterized model vs measurement
latency vs throughput
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
10
parameterized model vs measurement
lost frame rate vs input data rate
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
11
ATLAS LVL2 network performance studies
Central switch: generic - fully non-blocking
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
12
Modeling results
• The parameterized model uses a limited number of parameters,
which can be measured on COTS switches
• The parameterized model has been successfully verified on a
number of switches
• The parameterized switch model was used in modeling the large
Ethernet network for the ATLAS second level trigger system
• The model of the network can be used to verify applicability of
different switches currently on the market and their impact on the
network performance
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
13
CPU utilization measurements
• Ping-pong configuration at different packet rates
• Measure CPU utilization for various packet sizes, rates
ATLAS LVL2
REQUIREMENT
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
14
Limitations of PC-based measurements
• PC with standard software unable to drive switches at line speed



100% for FE only for packet size > 500 Byte [half duplex]
35% for GBE for any packet size - limited by host transfer
Parameterization of a single switch needs to be tested over the entire operational
envelope
• Number of nodes limited to 40 in present tests


Scaling up to ~2000 nodes is not credible
Model needs to be verified on a larger test bed
Conclusion:
• Dedicated traffic generators needed for FE, GBE in order to fully characterize
switches to be used in ATLAS HLT
• Need larger test bed for greater confidence in model applied to full ATLAS
HLT system
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
15
Rates: GB Ethernet NIC development
• Commercial NIC - Alteon ACENIC
• Inadequate performance using manufacturer’s firmware
• Modifications -
• Custom software developed
• All traffic local to NIC (host memory not involved)
• Full duplex line speed achieved for any size packet
PCI BUS
TIGON CHIP
TIGON
CHIP
PCI
interface
MAC
DMA1
DMA2
PCI BUS
Phy
CPU 1
Mem
CPU 2
Mem
GE
Ext Mem
0.5/1 Mbyte
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
16
8 Gigabit switch tester in a single chassis
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
17
Latency vs throughput for 8-port GBE switch
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
18
Fast Ethernet tester
•
•
•
•
Custom board, built at CERN
Programmable network traffic to characterize network switches
IP, Ethernet, including QoS
Why build it instead of buying?
• Economics
• Reprogrammable for
ATLAS purposes
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
19
FE tester: capabilities & status
•
•
•
•
•
•
•
•
32 ports full-duplex fast Ethernet
Parallel port connection to host
MAC function: FPGA
All FPGAs programmed in Handel C
• Handel C is a high-level language with ‘C’ syntax
• Enhancements for FPGAs (e.g., arbitrary width bit fields)
• Simulation facility built into implementation
Output packets generated, time stamped
Incoming packets time stamped, CRC checked
Queue dwell times histogrammed
All of this full duplex @ full line speed
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
20
Fast Ethernet tester board
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
21
FE tester as ROB emulator
• Accepts MESH ROB_DATA_REQUESTs
• Generates MESH ROB_DATA_REPLYs
• Programmable latency in ROB
• Response size depends on ROI size and detector type
• Response contents not meaningful
• Measure latencies and queue depths
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
22
Large scale test bed using ROB emulator
•
Use 8 of these boards [11K CHF each] to provide 256 emulated ROB ports
•
Add: -
• 64 PCs - supervisors + farm nodes
• Switch fabric
•
We have a ~15% scale test bed of Atlas LVL2
• Previous test bed: order of magnitude smaller
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
23
Future activity
• Measurements -
• Improved switch characterization
• Modeling -
• Parameterized model of GE switch (central switch)
• Detailed modeling of ATLAS LVL2
• Modeling of unified network to handle LVL2 traffic + Event Filter (LVL3 trigger)
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
24
Summary
• TCP/IP is very expensive in terms of CPU utilization at rates required
• MESH will likely be used in ATLAS LVL2 trigger
• GBE needs custom firmware in NIC to get required performance
• Modeling in good agreement with single switch measurements
• Large scale test bed minimizes uncertainty associated with behavior
extrapolated from a few nodes
• Hardware ROB emulation provides opportunity to see network behavior under
realistic conditions
DAQ 2000 Oct 2000
Testing & Modeling Ethernet Switches for ATLAS HLT
25