Transcript GridKa - Indico

Gridka 2005 – Status and Plans,
Network projection
Forschungszentrum Karlsruhe GmbH
Institute for Scientific Computing
P.O. Box 3640
D-76021 Karlsruhe, Germany
http://www.gridka.de
Bruno Hoeft
Bruno Hoeft
T1/T2 Workshop - GridKa , 19. Oct. 2005
The Grid Computing Centre Karlsruhe GridKa
Requested as a Regional Data and Computing Centre by 41 HEP
user groups in 19 German universities and research institutions.
Founded in 2001 as part of the computing centre of
Forschungszentrum Karlsruhe.
Main goals
• test environment for LHC (ALICE, ATLAS, CMS, LHCb)
• LHC Tier-1 in 2007+
• production environment for non-LHC (BaBar, CDF, D0, Compass)
• environment for Grid R&D (CrossGrid, LCG, EGEE, …)
• user support, services, education & training
• grid environment for other sciences (astrophysics, bio-informatics...)
Bruno Hoeft
T1/T2 Workshop - GridKa , 19. Oct. 2005
Stepwise extension of GridKa resources
•
•
•
•
•
Bruno Hoeft
Oct 2004
Apr 2005
Oct 2005
% of 2008
Processors
1070
1280
1550
40 %
Compute power / kSI2k
920
1200
1500
18 %
Disk [TB]
220
270
310
8%
Tape [TB]
375
475
500
11 %
Internet [Gb/s]
10
10
10
50 %
extension of resources every 6 months (2001-2005)
heterogeneous environment, PIII@1,26 GHz, …, PIV@3,06, … Opteron 246
310 TB Disk is usable disk space, ~2550 disk drives
installed with SC3.0.x, LCG 2_6_0
available via Grid together with ~130 other installations in Europe (EGEE)
T1/T2 Workshop - GridKa , 19. Oct. 2005
Requested resources at GridKa (LHC + non-LHC)
2005
2006
2007
2008
2009
CPU [kSI2k]
1440
2020
3080
8300
12780
Disk [TB]
310
640
1390
3860
5880
Tape [TB]
500
960
1830
4460
8700
WAN [Gb/s]
10
10
10
20
20
2005
SC2
SC3
2006
2007
cosmics
First physics
First beams
Full physics run
SC = Service Challenge (between Tier-0/1/2)
Bruno Hoeft
2008
T1/T2 Workshop - GridKa , 19. Oct. 2005
Service Challenge 3 (SC3) @ GridKa
Data transfers
Throughput phase (Juli):
•
FTS transfers from Cern to FZK with approx. 50 MB/s average / 100 MB/s
peak to dCache disk. (SC3 Goal: 150 MB/s)
-
Cern FTS configuration problem
dCache disk file system (ext3) problem at GridKa
•
No tape connection available
•
CMS PhEDEx transfers to dCache disk with up to 100 MB/s peak
Most T1 centers did
not achieve the SC3
throughput goal in Juli.
Bruno Hoeft
T1/T2 Workshop - GridKa , 19. Oct. 2005
GridKa SC3 dCache pool nodes got SAN disks with GPFS file system.
→ approx. 20 TB disk attached to 8 dCache pool machines.
● dCache Tape connection installed (but not yet stable).
●
●
Bruno Hoeft
In September - after official
throughput phase –
the SC3 goal (150 MB/s to
disk) was achieved on
several days.
Peak rate up to 280 MB/s
hourly average.
T1/T2 Workshop - GridKa , 19. Oct. 2005
Tier 1 ↔ Tier 2 transfers:
●
●
●
PhEDEx transfers (CMS) from CERN to FZK and FZK to DESY (Jens Rehn)
gridftp and srm transfers tested to/from DESY, FZU, GSI.
FTS server installed and tested. FZK ↔ GSI transfer channel available.
LCG 2.6.0 setup for service phase
●
●
●
●
●
Experiment's login nodes are used as “VO box“.
BDII, RB, PX, MON and SE (classic) used from production system
Separate SC3 setup with UI, CE (all GridKa WNs available), LFC.
dCache SE is currently being installed.
FTS server for T1 ↔ T2 transfers operational.
CMS and ALICE started their service phase in September
● No major problems reported except for “Cern blockade incident“ on
Friday Sept. 16th.
Bruno Hoeft
T1/T2 Workshop - GridKa , 19. Oct. 2005
Planing SC4 hardware expansion GridKa
Resource
Delta
In service
Status
Network
Lightpath to CERN
Jan. 2006
Waiting for GEANT2 progress
CPU
Expansion with 500 kS of existing 530 kSI
April 2006
Procurement started
Disk
Expansion with 200 TB for dCache of existing
45 TB
April 2006
Procurement starts Nov 2006
Tape
Tape Expansion with 250 TB of existing 620 TB
Jan. 2006
Procurement started
Tape IO
Expansion with 320 MB/s of existing 300 MB/s
Jan. 2006
Procurement started
Expansion with 640 MB/s
April 2006
Procurement starts Jan 2006
Write Pool Expansion with 100 MB/s to 250
MB/s
April 2006
Procurement starts Jan 2006
Write Pool
•The SC3 started the integration into the LCG production environment. The integration and
migration to a single production environment is in progress and will be finished in January 2006.
•Several Services installed and operational (SRM, FTS server, LFC, PhEDEx)
•We plan to operate a single database engine (Oracle) to consolidate several separate databases
within the LCG frame. Planning is dependent on progress of LCG 3D working group/workshop.
Bruno Hoeft
T1/T2 Workshop - GridKa , 19. Oct. 2005
Network installation
DFN
10Gbit/s
service challenge only
router
…
SAN
sc nodes
sc nodes
1Gbit/s
GridKa
1 Gbit/s
sc nodes
PIX
LoginServer
LoginServer
Storage
direct
internet access
…
1 Gbit/s
router
FS
FS
…
…
FS NAS
SAN
FS
…
ComputeNode
ComputeNode
ComputeNode
FS NAS
FS NAS
…
switch
…
…
…
private network
ComputeNode
Bruno Hoeft
T1/T2 Workshop - GridKa , 19. Oct. 2005
Ethernet 320 Mbit
Ethernet 100 Mbit
Ethernet 1 Gbit
FiberChannel 2Gbit
Ethernet 10 Gbit
Network installation
incl. Management Network
DFN
10Gbit/s
service challenge only
router
…
SAN
sc nodes
sc nodes
1Gbit/s
GridKa
1 Gbit/s
sc nodes
PIX
direct
internet access
RM
switch
1 Gbit/s
…
LoginServer
LoginServer
Master Controller
Storage
router
switch
…
RM
FS
FS
FS
RM
SAN
RM
RM
…
ComputeNode
ComputeNode
ComputeNode
RM
FS NAS
…
…
FS NAS
FS NAS
…
…
…
private network
ComputeNode
Bruno Hoeft
-Ganglia
-Nagios
-Cacti
T1/T2 Workshop - GridKa , 19. Oct. 2005
Ethernet 320 Mbit
Ethernet 100 Mbit
Ethernet 1 Gbit
FiberChannel 2Gbit
Ethernet 10 Gbit
KR
IWR—R-internet
GridKa-InternetRouter
10GE
10GE
2005
2006
FS
Layer2
GridKa DMZFSFS
FZK-LAN
2007
2008
SAN
external
internal
WN
WN
WN
Bruno Hoeft
FS
FS
FS
T1/T2 Workshop - GridKa , 19. Oct. 2005
SAN
dCache
TSM
IP Backbone
KR
IWR—R-internet
DWDM
DFN
GridKa-InternetRouter
10GE
10GE
2005
FZK-LAN
2006
GridKa
2007
FS
FS
FS
2008
SAN
external
internal
WN
WN
WN
Bruno Hoeft
FS
FS
FS
T1/T2 Workshop - GridKa , 19. Oct. 2005
SAN
dCache
TSM
https://uimon.cern.ch/twiki/bin/view/LHCOPN/LHCopnTables
Bruno Hoeft
T1/T2 Workshop - GridKa , 19. Oct. 2005
T2
T2
T0‘
HEP
T0
T1
T2
X-WiN
AR
10GE
DWDM
T0 DWDM lightpath
10GE
10GE
1GE?/?10GE
KR
IWR—R-internet
10GE
GridKa-InternetRouter
T0 DWDM lightpath
Backup (end 2006)
T1 DWDM lightpath
Bandbreite ntd
T2 DWDM lightpath
Bandbreite ntd
10GE physikalische
verbindung
1GE physikalische
verbindung
T2 L2-IP-Tunnel
(MPLS)
FZK-LAN
Bruno Hoeft
GridKa
T1/T2 Workshop - GridKa , 19. Oct. 2005
Conclusion
•
•
•
•
•
•
GridKa will tripple the instalation 2007/2008
SC3 goal reached 150MBps (peak 280MBps)
Services operational (FTS, SRM,…)
Divide storage in Ext/Int
LHC OPN T0/T1
MPLS Layer-2 Tunnel / LHC OPN T1/T2
Bruno Hoeft
T1/T2 Workshop - GridKa , 19. Oct. 2005
Forschungszentrum Karlsruhe
in der Helmholtz-Gemeinschaft
Bruno Hoeft
T1/T2 Workshop - GridKa , 19. Oct. 2005