Transcript transparencies - Indico

Experience with the EU Data
Grid Project
From a Computer Scientist’s point of view
Heinz Stockinger
EDG Education & Outreach Manager
http://www.eu-datagrid.org
DataGrid is a project funded by the European Union
The EU DataGrid (EDG) Project

9.8 M Euros EU (IST) funding over 3 years

Three year phased developments & demos (2001-2003)

Project objectives:

Middleware for fabric & Grid management (mostly funded by the EU)

Large scale testbed (mostly funded by the partners)

Production quality demonstrations (partially funded by the EU)

To collaborate with and complement other European and US projects

Contribute to Open Standards and international bodies:


 Total


Co-founder of Global Grid Forum and host of GGF1 and GGF3
Industry and Research Forum for dissemination of project results
of 21 partners
Research and Academic institutes as well as industrial companies
Main partners:
Experience with EU DataGrid Project – from a CS point of view – Heinz Stockinger - n° 2
DataGrid Scientific Applications
Developing Grid middleware to enable large-scale
usage by scientific applications
Bio-informatics


Data mining on genomic databases (exponential
growth)
Indexing of medical databases (Tb/hospital/year)
Earth Observation
•about 100 Gbytes of data per day
(ERS 1/2)
•500 Gbytes, for the ENVISAT mission
Particle Physics

Simulate and reconstruct complex physics
phenomena millions of times

LHC experiments will generate 6-8
PetaBytes/year
Experience with EU DataGrid Project – from a CS point of view – Heinz Stockinger - n° 3
EDG structure : work packages

The EDG collaboration is structured in 12 Work Packages:

WP1: Work Load Management System

WP2: Data Management

WP3: Grid Monitoring / Grid Information Systems

WP4: Fabric Management

WP5: Storage Element

WP6: Testbed and demonstrators

WP7: Network Monitoring

WP8:
High Energy Physics Applications

WP9:
Earth Observation

WP10: Biology

WP11: Dissemination

WP12: Management
}
Applications
Experience with EU DataGrid Project – from a CS point of view – Heinz Stockinger - n° 4
EDG Middleware Architecture
Local Computing
Local Application
Local Database
APPLICATIONS
Grid
Grid Application Layer
Data
Management
Job
Management
Metadata
Management
Collective Services
Grid
Scheduler
Information
&
Monitoring
Replica
Manager
Underlying Grid Services
SQL
Database
Services
Computing
Element
Services
Storage
Element
Services
Replica
Catalog
Authorization
Authentication
and Accounting
Service
Index
M/W
Grid
Fabric
Fabric services
Resource
Management
Configuration
Management
Monitoring
and
Fault Tolerance
Node
Installation &
Management
Fabric Storage
Management
GLOBUS
CondorG
(via VDT)
Experience with EU DataGrid Project – from a CS point of view – Heinz Stockinger - n° 5
Workflow Overview
Testbed
Workload
Management
Security
Data
Management
Information &
Monitoring
Applications
Fabric Management
Experience with EU DataGrid Project – from a CS point of view – Heinz Stockinger - n° 6
Early Days of EU DataGrid (EDG)
 EDG
officially started:1 January 2001
 In
early 2000, first working groups were formed to investigate
the Grid research domain

Evaluation of Globus (partly Legion)

First contacts with the leading Grid community

Data Grids were just emerging in the Grid Forum
 CERN’s
data intensive LHC Computing projects were soon
accepted as example Data Grids


Originally only a few technical people were involved in the Grid
activity at CERN (in particular master’s and Ph.D. students)
Early Grid software prototypes written plus first research papers
Experience with EU DataGrid Project – from a CS point of view – Heinz Stockinger - n° 7
Research Challenges
 Data
Grids are still rather new and very complex
 In
EDG, strong interactions with application as well as related
Grid projects all over the world

As a scientist, you are not alone in the field

Lots of co-operation and competition

Challenge for every scientist
 Main

search domains
Workload & data management, information systems, networking,
security
 EDG
has more than 150 project members:

Several of them are students (who often do most of the work )

Several Masters and Ph.D.s already “born” within EDG
Experience with EU DataGrid Project – from a CS point of view – Heinz Stockinger - n° 8
CERN’s Ph.D. Student Programme
 CERN
provides a student programme for Ph.D. as well as
technical students (mainly BSc or MSc):
http://humanresources.web.cern.ch/HumanResources/external/recruitment/Stu
dents/students.asp
 Student
programmes have close co-operation between:

Home University

CERN group (incl. CERN supervisor)

CERN related project (e.g. EDG)
 EDG
in particular provides several challenges and very good
opportunities for students

In particular in the earlier phases of the projects

In later phases more engineering is/was required
Experience with EU DataGrid Project – from a CS point of view – Heinz Stockinger - n° 9
Search for Further Challenges
 EDG
will officially end in Dec. 2003
However, the Grid is not ready yet !
 EGEE
will “take over” in 2004 (see Bob Jones’ slides)
 Several
other EU funded Grid projects in Europe, U.S., etc.
 GRIDSTART
www.gridstart.org provides a good forum for
projects EU projects:

Inventory of existing projects and open challenges:

http://www.gridstart.org/download/GRIDSTART-IR-D2.2.1.2-V1.2.doc
Experience with EU DataGrid Project – from a CS point of view – Heinz Stockinger - n° 10
Conclusion
 EU
DataGrid has been and is one of the leading Grid projects
that drives Grid technology


Several research aspects have been covered but still many open
challenges
Grid community has still several challenges to meet in order
to make Grids really production ready


R&D still required in many fields
More and more need for highly qualified Computer and
Application Scientists to engineer parts of the Grid software &
infrastructure
 Learn
more about EDG:

www.eu-datagrid.org

www.cern.ch/edgtutorial
Experience with EU DataGrid Project – from a CS point of view – Heinz Stockinger - n° 11