Transcript Poster - Indico

Forschungszentrum Karlsruhe
in der Helmholtz-Gemeinschaft
Forschungszentrum Karlsruhe GmbH, Institut für Wissenschaftliches Rechnen,
Postfach 36 40, 76021 Karlsruhe
Bruno Hoeft, Marc Garcia Marti
LHC-OpticalPrivateNetwork NETWORK AT GRIDKA
10Gbit LAN/WAN evaluations
NIC – tuned for throughput
Intel
Chelsio
German / European
1)
IP via
MPLS Tunnel
International IP
RWTH-Aachen
Uni-Dresden
Fnal
Uni-Freiburg
MPI u. Uni-München
SLAC
Uni-Wuppertal
CNAF
SARA
GSI /
FZU Prague /
Desy PSNC Poland
(2007 10Gbit)
(2007 10Gbit)
CERN
(2006/7/8
(2007
1/10Gbit) 1Gbit)
2007 10Gbit
240Mbyte netto
}
HEP
Standard mtu
8160 Bytes mtu
8180 Bytes mtu
9216 Bytes mtu
Tier-0/Tier-1 (T0/T1)
LHC-OPN Lightpath
Tier-2
LHC Lightpath
T2
T2
T2
T1
T2
T0
T0‘
Standard mtu
8180 Bytes mtu
9216 Bytes mtu
DFN
X-WiN
- PCI-X Bus: theoretical limit 8.5 Gbps approx.
- tested max. rate 7.8 Gbps (linux kernel packet generator)
10G
E
AR
DWDM
10GE
10GE
FZK-Internet
Router
…
Cache Headnode
LoginServer
ALICE/Atlas/CMS/LHCb
LoginServer
- Intel 0.016ms
- The throughput of the Intel NICs is neglectable after tuning
for latency
d
…
router
private network
FS
FS
…
…
SAN
switch
…
FS
FS NAS
FS NAS
…
ComputeNode
ComputeNode
ComputeNode
…
- Chelsio 0.020ms
-The throughput of the Chelsio NIC is still normal after
tuning for latency
dcache2)
pools
FS NAS
…
NIC – tuned for Latency
LCG(se/ce/ui)
LCG(RGMA/MON/BDII)
LCG(RB/LFC/DC-SRM)
…
Ethernet 1 Gbit
FiberChannel 2Gbit
Ethernet 10 Gbit
T0 DWDM lightpath
T0 DWDM lightpath
Backup (end 2006)
T1 DWDM lightpath
Bandwidth 10GE
T2 DWDM lightpath
Bandwidth 1/10GE
T2 L2-IP-Tunnel
(MPLS)
Babar/CDF/DZero/Compass
…
FZK-LAN
dcache2)
pools
intern
Standard mtu
8160 Bytes mtu
8180 Bytes mtu
9216 Bytes mtu
Standard mtu (server side)
8160 Bytes mtu (server side)
8180 Bytes mtu (server side)
9216 Bytes mtu (server side)
Firewall
extern
Standard mtu
8180 Bytes mtu
9216 Bytes mtu
Standard mtu (server side)
8180 Bytes mtu (server side)
9216 Bytes mtu (server side)
GridKa-Internet
Router
10GE
ComputeNode
WAN setup -- GridKa - CERN
Cisco 12416
GEANT
Juniper M160
DFN
Cisco 12410
Juniper M160
Routing:
- External Routing via BGP
-eBGP with the Forschungszentrums AS-No.: 34878
exterior Boarder Gateway Protocol
-iBGP synchronising the eBGP router
interior Boarder Gateway Protocol
- Internal redundance secured via OSPF
- Layer-2 tunnel for distributed Storage Server via
Policy Based Routing
- Private network space separated
into switched VLans
Cisco 6509
Cisco 7609
Gridka (Karlsruhe)
CERN
oplapro96
10gtk113
WAN throughput
/
TCP cloning
AR
eBGP
DFN
DWDM
eBGP
Juniper M160
R-inet-GridKa
R-inet-IWR
FAN
STATUS
9
Window size 14.0 MByte
Window size 2.0 MByte
Window size 8.0 MByte
Window size 8.0 MByte
Window size 8.0 MByte
8
7
6
5
4
3
2
1
FAN
STATUS
TxIntDelay=500 RxIntDelay=500
TxIntDelay=500 RxIntDelay=500
TxIntDelay=250 RxIntDelay=250
TxIntDelay=74 RxIntDelay=74
TxIntDelay=15 RxIntDelay=15
9
8
7
6
5
4
3
2
1
iBGP
Power Supply 1
Power Supply 2
WS-C6509-NEB
Power Supply 1
Power Supply 2
Mmrbc set to 512 bytes
Mmrbc set to 2048 bytes
Mmrbc set to 4096 bytes
PolicyBasedRouting
WS-C6509-NEB
VLAN 192.108.45.0 /26
OSPF +
FAN
STATUS
FAN
STATUS
9
8
7
6
5
4
3
2
9
1
8
7
6
5
4
3
2
1
Layer-2 Tunnel
Power Supply 1
Power Supply 1
Power Supply 2
Power Supply 2
WS-C6509-NEB
WS-C6509-NEB
...
...
1GE versus 10GE Server Nodes
VLAN 33_x 10.33.x.x /24
LCG-se/ce/ui/bdii/...
45000
40000
Total Cost of Ownership (TCO):
- Dramatically reduced costs (per NIC/high end server)
- the troughput increased from
2.8Gbit to 7.0Gbit with TOE (standard MTU)
- eight 1GE standard server = one 10GE Server
- higher administration needs to be taken into account
- PCI-Express will allow for the full rate of 10GE
- 250/6000 Euro per 1GE/10GE router port
35000
Euro
30000
25000
20000
15000
10000
5000
0
2003
Euro / Server
Euro / NIC
2004
2005
VLAN / Switched
2006
Year
Email: [email protected]; [email protected]
Phone: +49 7247 82 8172
Web: www.gridka.de
www.fzk.de/iwr
1)
„Connecting WLCG Tier-2 Centres to GridKa“
of Dr. Andreas Heiss presents more details about the
at GridKa assigned Tier-2 sites
2)
Dr. Doris Ressmann will present in her talk more detailed
information at the dCache setup of GridKa.
„using TSM to create a high performance Tape connection“
Track: Computing Facilities and Networking, Monday 14:20-14:40