Transcript The European DataGrid Project

The European DataGrid Project
Fabrizio Gagliardi
EU DataGrid Project Leader
CERN
[email protected]
The Grid metaphor
Mobile Access
Workstation
G
R
I
D
M
I
D
D
L
E
W
A
R
E
Supercomputer, PC-Cluster
Data-storage, Sensors, Experiments
Visualising
Internet, networks
Fabrizio Gagliardi EDG5 Piliscsaba
Technology Trends
• Very fast networks can be built and are becoming
much cheaper
• Archival and disk storage becoming much cheaper
• Sensors and instruments becoming ubiquitous and
have much higher resolution => oceans of data
• Computing power continues to increase
– so many groups can analyze lots of data
• Middleware, while still developing, is not vaporware
• Everyone uses the web, so concept is not foreign
• Some demonstration projects have shown the
benefits (EU DataGrid test bed among others)
Fabrizio Gagliardi EDG5 Piliscsaba
Grids Leverage Major
Investments in Experimental
Research Facilities
• Research instruments, satellites,
particle accelerators, etc., cost a great
deal
• Data from those devices can be
accessed and analyzed by many more
scientists from anywhere
– not just the team that gathered the data or
institutes from privileged countries
• Easier international collaboration
Fabrizio Gagliardi EDG5 Piliscsaba
The EU DataGrid Project
•
•
•
•
•
21 Partners
9.8 M Euros EU funding over 3 years
90% for middleware and applications (HEP, EO
and Biomedical)
Three year phased developments & demos (20012003)
Spin-off:
– DataTAG (2002-2003)
– CrossGrid (2002-2004)
– GridStart (2002-2004)
Fabrizio Gagliardi EDG5 Piliscsaba
EU DataGrid Project
Objectives
•
Use Grid technology to develop a sustainable
computing model for effective share of
computing resources and data for large
scientific communities
•
Specific project objectives:
– Middleware for fabric & Grid management (mostly
funded by the EU)
– Large scale testbeds (mostly funded by the partners)
– Production quality demonstrations (partially funded by
the EU)
Fabrizio Gagliardi EDG5 Piliscsaba
EU DataGrid Project
Objectives
•
•
•
Collaborate with and complement other
European and US projects
Test and demonstrator of EU RN/Geant
Contribute to Open Standards and international
bodies:
– Co-founder of Global GRID Forum and host of GGF1
and GGF3, co-founder of GRIDSTART
•
Dissemination
– Industry and Research Forum for dissemination of
project results
•
Education: Tutorials, CSC02, Road Show
Fabrizio Gagliardi EDG5 Piliscsaba
Project Schedule
• Project started on 1/1/2001
• TestBed 0 (early 2001)
– International test bed 0 infrastructure deployed
• TestBed 1.0 ( March 2002 )
• Project successfully reviewed by EU on March 1st 2002
• TestBed 1.2 ( now )
– First release of EU DataGrid software to defined users within the project:
• HEP experiments, Earth Observation, Biomedical applications
• TestBed 2 (end of 2002)
– Builds on TestBed 1 to extend facilities of DataGrid
• TestBed 3 (Fall 2003)
• Project completion expected by end 2003
Fabrizio Gagliardi EDG5 Piliscsaba
DataGrid Main Partners
• CERN – International (Switzerland/France)
• CNRS - France
• ESA/ESRIN – International (Italy)
• INFN - Italy
• NIKHEF – The Netherlands
• PPARC - UK
Fabrizio Gagliardi EDG5 Piliscsaba
Assistant Partners
Industrial Partners
•Datamat (Italy)
•IBM-UK (UK)
•CS-SI (France)
Research and Academic Institutes
•CESNET (Czech Republic)
•Commissariat à l'énergie atomique (CEA) – France
•Computer and Automation Research Institute,
Hungarian Academy of Sciences (MTA SZTAKI)
•Consiglio Nazionale delle Ricerche (Italy)
•Helsinki Institute of Physics – Finland
•Institut de Fisica d'Altes Energies (IFAE) - Spain
•Istituto Trentino di Cultura (IRST) – Italy
•Konrad-Zuse-Zentrum für Informationstechnik Berlin - Germany
•Royal Netherlands Meteorological Institute (KNMI)
•Ruprecht-Karls-Universität Heidelberg - Germany
•Stichting Academisch Rekencentrum Amsterdam (SARA) – Netherlands
•Swedish Research Council - Sweden
Fabrizio Gagliardi EDG5 Piliscsaba
EDG Highlights
•
The project has passed the first EU review (last
March)
– All 21 partners have contributed at least at contractual level
– All EU deliverables (40, >2000 pages) submitted in time
(available, code & documents via www.edg.org)
– 60 EU funded FTEs and 106 un-funded FTEs
– One of the largest Grid technology development and
demonstrator
•
Real applications being deployed now
Fabrizio Gagliardi EDG5 Piliscsaba
Particle Physics Applications
CMS
Storage –
ATLAS
Raw recording rate 0.1 – 1 GByte/sec
Accumulating data at 8-10 PetaBytes/year
10 PetaBytes of disk
LHCb
Processing –
100,000 of today’s fastest PCs
Fabrizio Gagliardi EDG5 Piliscsaba
Biomedical applications
• Data mining on genomic databases
(exponential growth)
• Indexing of medical databases
(Tb/hospital/year)
• Collaborative framework for large scale
experiments (e.g. epidemiological
studies)
• Parallel processing for
–Databases analysis
–Complex 3D modelling
Fabrizio Gagliardi EDG5 Piliscsaba
Earth Observations
ESA missions:
• about 100 Gbytes of data per
day (ERS 1/2)
• 500 Gbytes, for the next
ENVISAT mission (launched
March 1st)
EO requirements for the Grid:
• enhance the ability to access high
level products
• allow reprocessing of large
historical archives
• improve Earth science complex
applications (data fusion, data
mining, modelling …)
Fabrizio Gagliardi EDG5 Piliscsaba
DataGrid Testbed
Testbed Sites(>40)
HEP sites
ESA sites
Dubna
Lund
Moscow
RAL Estec KNMI Berlin
IPSL
Paris
Santander
Lisboa
CERN
Prague
Brno
Lyon
Grenoble
Milano
PD-LNL
Torino
Madrid
Marseille Pisa BO-CNAF
Barcelona
ESRIN
Roma
Valencia
Catania
[email protected]
- [email protected]
Fabrizio Gagliardi EDG5
Piliscsaba
Project Dissemination
www.eu-datagrid.org
CERN School of Computing 2002
Programme includes
Grid Lectures by
Ian Foster
Carl Kesselman
Hands-on tutorial
DataGrid
Vico Equense, Italy, 15-28 September 2002
The 2002 CERN School of Computing is organised by CERN,
with the Institute of Composite and Biomedical Materials, National
Research Council, Naples, Italy
Fabrizio Gagliardi EDG5 Piliscsaba
Future Plans
• Concentrate on production quality and real
applications
• Complete the programme of work till end of
2003
• Educate new users and disseminate results
• Make plans to conserve momentum and
assets in the EU FP6
Fabrizio Gagliardi EDG5 Piliscsaba
A proposal to the EU
• EDG has already demonstrated the viability of
this technology
• EU had a fast start in supporting 18 Grid
projects in the last two years!
• RN Geant offers an excellent basis for a large
European Grid infrastructure
• The EU FP6 programme should encourage
and support the deployment and production
quality operation of a large international Grid
infrastructure open to research and industry
in Europe
Fabrizio Gagliardi EDG5 Piliscsaba
The EGEE EoI
• EGEE: Integrated Infrastructure Initiative (III) to
support ERA (European Research Area) 300 M €
planned (100 M € requested)
• Vision: to create and deploy Grid technologies to
enable the widespread uptake of e-Science
applications throughout the European Research Area
• Four key objectives:
– integrating Grid technological developments from across Europe;
– establishing a Europe-wide Grid infrastructure for science and
industry with a focus on heterogeneity and interoperability;
– enabling the creation of e-Science applications from across the
scientific and industrial spectrum;
– ensuring the timely delivery of the project’s programme of work,
guided by the needs of academic and industrial partners.
Fabrizio Gagliardi EDG5 Piliscsaba
The EGEE EoI continued
• Approach: coordinated submission with RN
Geant follow-up (200 M € requested)
• Focus on the Grid production infrastructure to
support vertical and horizontal applications
(vertical: Bio, EO, PP, Engineering,
Commerce? Industry? horizontal: CS and
added value services)
• Special emphasis of the EU programme on
SMEs
• Distributed managerial structure by
coordinating national and regional Grid
initiatives
Fabrizio Gagliardi EDG5 Piliscsaba
Final word…
• EU DataGrid is a good first step together with
the other related Grid initiatives
• Need not to lose momentum but aggregate
interest and resources in science, industry
and funding agencies for proposals in the EU
FP6
• Significant contribution from the Hungarian
partners so far
• Important step this week here to review the
progress and make plans for the future
Fabrizio Gagliardi EDG5 Piliscsaba
Many thanks for the kind
hospitality and excellent
support!
Fabrizio Gagliardi EDG5 Piliscsaba