Transcript scotgrid: a prototype tier 2 centre

SCOTGRID: A PROTOTYPE
TIER-2 CENTRE
Steve Thorn
Authors: A. Earl, P. Clark, S. Thorn
University of Edinburgh
ScotGrid: a Prototype Tier-2 Centre – Steve Thorn, Edinburgh University
Overview
• What is ScotGrid?
• ScotGrid resources
• LCG2 experiences (at Edinburgh)
– System administrator
– Particle physics user
• Future plans
• Conclusions
ScotGrid: a Prototype Tier-2 Centre – Steve Thorn, Edinburgh University
What is ScotGrid?
•
•
•
•
•
•
•
•
•
•
One of four Tier-2 sites in the UK
Durham, Edinburgh & Glasgow
Member of GridPP collaboration
Over 100 users
Supports: ATLAS, BaBar,
Bioinformatics, CDF, Computing
Science, Engineering, LHCb,
Medipix, MICE, PhenoGrid, UKQCD,
ZEUS
First UK Tier-2
Over 200 CPUs
~40 TB disk
Initially funded by SHEFC (Scottish
Higher Education Funding Council)
www.scotgrid.ac.uk
ScotGrid: a Prototype Tier-2 Centre – Steve Thorn, Edinburgh University
Hardware overview
ScotGrid: a Prototype Tier-2 Centre – Steve Thorn, Edinburgh University
Edinburgh University
Front Ends:
CE & SE: IBM x205: P-IV 1.8 GHz, 256 MB RAM
LCFG: IBM x340: dual P-III 1 GHz, 2 GB RAM
Storage (backend):
1 × IBM x440: 8 × Xeon 1.9 GHz, 32 GB RAM
2 × IBM FAStT900 providing 24 TB RAID
10 TB of 155 TB SAN
Worker Nodes: 4 × dual Xeon 2.8 GHz, 2 GB
RAM, 200 GB disk
Housed at the multi-million pound Advanced
Computing Facility. 20 Gbit/s link to campus.
ScotGrid: a Prototype Tier-2 Centre – Steve Thorn, Edinburgh University
Benchmarking
IBM x440: 8 × Intel Xeon 1.9 GHz, 32 GB RAM
High Performance Linpack benchmark using ATLAS libraries. Matrix calculations
to test processor and memory performance.
Achieved ~75% of theoretical max performance with un-optimised system
Full results at www.ph.ed.ac.uk/~aearl/hpl/
ScotGrid: a Prototype Tier-2 Centre – Steve Thorn, Edinburgh University
Glasgow University
Front Ends (CE,SE,LCFG,UI):
4 × Transtec 1001L: P-IV 2.8 GHz, 512 MB
RAM
Storage:
1 × IBM x370: P-III Xeon 700 MHz, 16 GB
RAM
1 × IBM FAStT500 providing 5 TB RAID
Worker Nodes:
59 × IBM x330: dual P-III, 1 GHz, 2 GB RAM
34 × IBM blades: dual Xeon 2.4 GHz, 1.5 GB
RAM
5 × IBM x340: dual P-III 1 GHz, 2 GB RAM
ScotGrid: a Prototype Tier-2 Centre – Steve Thorn, Edinburgh University
Usage statistics
Glasgow CPU farm
ATLAS
LHCb
1498570 CPU hours
BaBar
4471 kilo SPECint2000 weeks
(23 September 2004)
CDF
Grid Data Management
Bioinformatics
ZEUS
UKQCD
Medipix
Information Retrieval
Device Modelling
MICE
ScotGrid: a Prototype Tier-2 Centre – Steve Thorn, Edinburgh University
Durham University
New ScotGrid member since September
2004
Worker Nodes: 40 × dual P-IV 2.2 GHz, 2
GB RAM, 30 GB disk
Based at the Institute for Particle Physics
Phenomenology
Using Sun Grid Engine
ScotGrid: a Prototype Tier-2 Centre – Steve Thorn, Edinburgh University
LCG2 experience: sys admin
•
•
•
•
•
•
Storage emphasis – too few Front End machines
Installed with LCFGng
Storage limited to 1 TB until SRM based deployment
Hardware moved to Advanced Computing Facility – successful
Worker Node 15 miles (24 km) from Front Ends
Participating in LCG2 Test Zone:
ScotGrid: a Prototype Tier-2 Centre – Steve Thorn, Edinburgh University
Installation problems
•
Multiple network interfaces
– CE(2) and SE(3)
– eth0 (external) is automatically
configured by LCFGng
– Others require manual config. of
driver, network parameters (ip,
netmask etc.) and routing
– SE eth0 not used so eth1 & eth2
manually defined
•
SCSI and RAID controllers
– SE only
– Custom boot kernel
– Full configuration of
/etc/modules.conf required
/* Second network interface eth1 */
EXTRA(hardware.modlist)
eth1
+hardware.mod_eth1
alias eth1 tg3
EXTRA(network.interfaces)
network.ipaddr_eth1
network.netmask_eth1
network.hostname_eth1
network.network_eth1
network.broadcast_eth1
eth1
192.168.1.6
255.255.255.0
glenlivet
192.168.1.0
192.168.1.255
REPLACE(install.installmethods,network,network
route)
install.imethod_route
route install/root
ScotGrid: a Prototype Tier-2 Centre – Steve Thorn, Edinburgh University
LCG2 Experience: user
Aim: to get a physicist with no grid
experience to use LCG2
1. Applied for Grid certificate
Well documented but time consuming
2. Joined LHCb VO
3. Install UI software on user’s desktop
Not possible due to OS mismatch and
firewall
4. Used Glasgow’s UI to successfully run
jobs
ScotGrid: a Prototype Tier-2 Centre – Steve Thorn, Edinburgh University
Future plans
• Edinburgh
– Deploy new Worker Nodes: 4 × dual processor from GridPP
– Storage software development – SRM to SRB interface
– Increase LCG storage to full capacity including SAN
• Glasgow
– Upgrade older Worker Nodes
– Migrate to Scientific Linux for Front Ends
• Durham
– Deploy new Front End systems: 4 × dual processor from GridPP
– Join LCG2 Test Zone
• Seeking funding
– to increase CPU at Edinburgh and storage at Glasgow
– Research and support staff
• Other institutes?
ScotGrid: a Prototype Tier-2 Centre – Steve Thorn, Edinburgh University
Conclusions
•
•
•
•
Successfully proven Tier-2 concept
Reached million CPU hour landmark
Over 100 users within 13 research groups/collaborations
Gained valuable LCG2 knowledge (installation and
maintenance)
• Edinburgh & Glasgow participating in LCG2 Test Zone
• Next steps: prototype → production
– Encourage more particle physics analysis
– Continue with data challenges
– More resources and funding
ScotGrid: a Prototype Tier-2 Centre – Steve Thorn, Edinburgh University
Questions?
ScotGrid: a Prototype Tier-2 Centre – Steve Thorn, Edinburgh University