Transcript document
Network to and at CERN Getting ready for LHC networking Jean-Michel Jouanigot and Paolo Moroni CERN/IT/CS Summary Current situation T0-T1 planning: LAN T0-T1 planning: WAN Current situation General purpose network Technical network Experimental areas (pre-production) External network (firewall / HTAR) 21 January 2005 T0/1 network meeting 3 General-purpose network COMPUTER CENTER Server Farms Technical Network ..etc.. Firewall CIXP,Internet 21 January 2005 T0/1 network meeting REMOTE MAJOR STARPOINTS ..etc.. 4 Technical network SR1 SR2 CCR General Purpose Network SR3 PCR SR4 MCR SR5 SR6 TCR SR7 SR8 21 January 2005 T0/1 network meeting 5 External network GÉANT +SWITC H Internet Tests + LHC preproduction CIXP Chicago PoP … General . purpose network 21 January 2005 T0/1 network meeting … . 6 Firewall This slide is intentionally left blank 21 January 2005 T0/1 network meeting 7 T0-T1 planning (LAN) New 2.4 Tb/s backbone to interconnect LHC experiments (CERN Tier0) general purpose network CERN Tier1 T0-T1 WAN (regional Tier1’s) Based on 10GE technology Layer 3 interconnections No central switch(es) Redundancy via multiple 10GE paths (OSPF) 21 January 2005 T0/1 network meeting 8 More about T0-T1 LAN Random paths through the backbone for load balancing (OSPF) IP addressing: depends on the LHC WAN implementation, RFC1918 addresses are likely for a lot of end systems a data mover facility can help a lot (already successfully implemented for the BABAR experiment at IN2P3) Default route? Maybe not necessary Call for tender for the equipment being issued 21 January 2005 T0/1 network meeting 9 T0-T1 network at CERN (LAN) multiple 10GE T0-T1 WAN 10GE Externa l network GbE 4 LHC experimental areas GPN Raw LHC data CERN Tier1 10GE->88*GE ~6000 CPU servers 10GE->88*GE ..88.. 21 January 2005 ..88.. … . 10GE->88*GE ..88.. 10GE->32*GE ..32.. T0/1 network meeting … . 10GE->n*10GE ..10.. ~2000 Tape and Disk servers 10 Tier0 network (LHC experimental areas) Low speed (management) High speed: redundant 10GE (data) CER N Tier1 T0-T1 WAN LHC experiment LHC experiment LHC experiment T0-T1 LAN LHC experiment control network DAQ GPN 21 January 2005 T0/1 network meeting 11 T0-T1 WAN: progress A lot of progress has been made: 10 Gb/s equipment is commonly available (although not yet cheap): STM-64 (10GE WAN PHY), 10GE LAN 10 Gb/s capacity (SDH, wavelength, WDM over dark fibre) is affordable long-distance, high-speed TCP is feasible, although with special Linux tuning 21 January 2005 T0/1 network meeting 12 T0-T1 WAN: progress (continued) More progress being made: GN2 is coming in Europe with new services and research activities Several interesting initiatives in North America and in Europe (dark fibre-based networks, etc.) Several interesting monitoring tools exist or are being developed Pre-production simulation (robust data challenge): a useful ongoing experience Firewall with HTAR works for non-LHC traffic and for some preproduction 21 January 2005 T0/1 network meeting 13 T0-T1 WAN: issues Still several open questions: how will Tier1’s connect to Tier0 (directly, one upstream, layered upstreams, …)? backup routing ? non-homogeneous Tier1 requirements? any Tier1-Tier1 traffic via Tier0? IP addressing: routable or RFC1918 ? does every Tier1 have enough routable addresses? and … 21 January 2005 T0/1 network meeting 14 T0-T1 WAN: more issues …what about security ? Tier2’s ? compatibility between GRID middleware and network design? special tuning for WAN data transfers? compatibility between high speed flows and some network devices (Juniper M160)? management, monitoring, troubleshooting? Anything else? 21 January 2005 T0/1 network meeting 15 Recommendations (I) Allow for diverse regional requirements, but standardise NOW on the T0-T1 physical interface: 10GE LAN PHY (LR/SR ?) STM-64/OC192 10GE WAN PHY (?) Other interfaces also possible in the pre-production phase (GbE, multiple GbE, STM-16) Take advantage of useful experience (robust data challenge) Define clearly the operational responsibilities across multiple administrative domains 21 January 2005 T0/1 network meeting 16 Recommendations (II) Select equipment which is expected to work reliably for some years A data mover facility (spooling system) helps with several issues: IP addressing needs security WAN data transfer optimisation Select proven and stable technology: smooth network operations and easy troubleshooting are essential 21 January 2005 T0/1 network meeting 17 Recommendations (III) Security is essential Monitoring is essential Allocate suitable (routable) subnets, dedicated to LHC production purposes If not enough routable IP addresses, ask RIPE-NCC for more, via the appropriate upstream LIR, and do so NOW (or ask ARIN, or APNIC, according to the region) 21 January 2005 T0/1 network meeting 18 Recommendations (IV) Never mind if the network is just a boring production tool: being at the bleeding edge is not essential in this situation LHC physics is the research target, not LHC networking 21 January 2005 T0/1 network meeting 19 LHC WAN: a possible design Assumptions: if … Tier1’s connect at layer 3 backup routing is a requirement and it is acceptable via research IP networks (not more than two-three Tier1’s down at the same time) Tier1-Tier1 traffic is allowed via Tier0 (although this would not be Tier0’s preference…) Tier1 and Tier0 addresses are publicly routable and every Tier1 has allocated a SMALL number of subnets for inter-Tier0/1 traffic BGP routing using the “natural” ASN and routable prefixes no default route (or no default route towards T0): is it possible? … 21 January 2005 T0/1 network meeting 20 A possible design (continued) …and if … basic security is provided via layer 3 ACLs (allowed subnets and, if possible, port numbers) Tier1’s may have some non-homogeneous requirements no Tier2 directly connected to Tier0, but some may be allowed to exchange traffic at less that 10 Gb/s alternatively, some T0-T2 traffic may transit via an intermediate T1 a spooling system (data mover) is used as buffer between sites to optimise long-distance data transfer and reduce public IP addresses needs … then … 21 January 2005 T0/1 network meeting 21 Tier 2 A possible T0-T1 WAN network multiple 10GE Tier1 Tier1 10GE or STM-64 Tier1 10GE or multiple GbE Tier 2 Tier1 Externa l network Tier1 Tier1 Data mover (spool) Tier1 LHC LAN 21 January 2005 T0/1 network meeting Tier1 … Tier1 . 22 Thank you Questions?