Transcript European and German Grid Projects

Forschungszentrum Karlsruhe
in der Helmholtz-Gemeinschaft
European and German
Grid Computing Projects
Marcel Kunze
Forschungszentrum Karlsruhe
[email protected]
DESY Seminar
December 2002
Grid Computing Potential
1980
2005
Marcel Kunze, DESY Seminar
What is Grid Computing?
Resource sharing & coordinated problem
solving in dynamic, multi-institutional virtual
organizations
Marcel Kunze, DESY Seminar
Grid Computing:
Diverse Areas of Application
• Computational Grid
High Performance Computing (HPC)
Perspective: Parallelization of programs, as fast as possible
• Data Grid
High Throughput Computing (HTC)
Perspective : Parallelization of data, as much as possible
• Gaming Grid
Communication between players
Perspective: LAN + WAN-Party,
as entertaining as possible
Marcel Kunze, DESY Seminar
Grid Computing:
Commercial Aspects
• Huge potential of Grid Computing in the field of
information technology:
– Cost reduction through more efficient/diverse use of system resources
– „Resource on demand“
Added flexibility, e.g. dynamically satisfy increased computing demand
in bank accounting sector at the end of an accounting period
– „Business continuity“
Proliferation of mission-critical IT-Services to ease disaster recovery
– Possibility of system consolidation:
Reduction of system complexity by standardization of services
Marcel Kunze, DESY Seminar
Grid Computing:
Scientific Aspects
• Enables formation of „critical mass“ of resources and
expertise to allow for interdisciplinary projects in socalled “Virtual Organizations” (VO)
–
–
–
–
Tele-Collaboration
Instrument Sharing
Computing Resource Sharing
Data Sharing
• Potential fields of application
–
–
–
–
Bio-Informatics and Medicine
Astronomy
Earth Sciences
High Energy Physics (HEP)
Marcel Kunze, DESY Seminar
Biomedical Applications
• Data mining on genomic databases
(exponential growth)
• Indexing of medical databases
(Tb/hospital/year)
• Collaborative framework for large scale
experiments
(e.g. epidemiological studies)
• Parallel processing for
–Databases analysis
–Complex 3D modelling
Fabrizio Gagliardi
Marcel Kunze, DESY Seminar
Astronomy:
Past and Future of the Universe
New phase of astronomy,
storing, searching and
analysing Petabytes of data:
• Virtual Observatories –
GAVO, NVO, AVO, AstroGrid
– Store all wavelengths, need
distributed joins
– NVO 500 TB/yr from 2004
• Grid Computing might help to
– Master the data streams
– Federate databases with different
schema
– Maintain meta data (information)
and provenance data (history)
Marcel Kunze, DESY Seminar
Crab Nebula viewed
At four different
wavelengths: X-ray,
optical, infrared, radio.
Earth Observation
ESA missions:
• about 100 Gbytes of data per day
(ERS 1/2)
• 500 Gbytes, for the next ENVISAT
mission (launched March 1st)
EO requirements for the Grid:
Marcel Kunze, DESY Seminar
Fabrizio Gagliardi
• enhance the ability to access high level
products
• allow reprocessing of large historical
archives
• improve Earth science complex
applications (data fusion, data mining,
modelling …)
What is Grid Computing?
Resource sharing & coordinated problem
solving in dynamic, multi-institutional virtual
organizations
Marcel Kunze, DESY Seminar
Elements of the Problem
• Resource sharing
– Computers, storage, sensors, networks, …
– Heterogeneity of device, mechanism, policy
– Sharing conditional: negotiation, payment, …
• Coordinated problem solving
– Integration of distributed resources
– Compound quality of service requirements
• Dynamic, multi-institutional virtual orgs
– Dynamic overlays on classic org structures
– Map to underlying control mechanisms
Marcel Kunze, DESY Seminar
The Grid World: Current Status
• Dozens of major Grid projects in scientific & technical
computing/research & education
– Deployment, application, technology
• Considerable consensus on key concepts and
technologies
– Open source Globus Toolkit™ a de facto standard for major
protocols & services
– Far from complete or perfect, but out there, evolving rapidly, and
large tool/user base
• Global Grid Forum a significant force
• Industrial interest emerging rapidly
Marcel Kunze, DESY Seminar
“Web Services”
• Increasingly popular standards-based framework for
accessing network applications
– W3C standardization; Microsoft, IBM, Sun, others
• WSDL: Web Services Description Language
– Interface Definition Language for Web services
• SOAP: Simple Object Access Protocol
– XML-based RPC protocol; common WSDL target
• WS-Inspection
– Conventions for locating service descriptions
• UDDI: Universal Description, Discovery & Integration
– Directory for Web services
Marcel Kunze, DESY Seminar
Open Grid Services Architecture
(OGSA)
• Platform independent industrial and scientific standard
to construct Grid computing components
• Service orientation to virtualize resources
• From Web services:
– Standard interface definition mechanisms: multiple protocol bindings,
multiple implementations, local/remote transparency
• Building on Globus Toolkit:
–
–
–
–
Grid service: semantics for service interactions
Management of transient instances (& state)
Factory, Registry, Discovery, other services
Reliable and secure transport
• Multiple hosting targets: J2EE, .NET, “C”, …
Marcel Kunze, DESY Seminar
The Grid Service
• A (potentially transient) Web service with
specified interfaces & behaviors, including
–
–
–
–
–
–
–
–
Creation (Factory)
Global naming (GSH) & references (GSR)
Lifetime management
Registration & Discovery
Authorization
Notification
Concurrency
Manageability
Marcel Kunze, DESY Seminar
Grid Projects
Marcel Kunze, DESY Seminar
GRID Projects World Wide
• EU
– EDG
(EU-IST) – R&D EU GRID project [ www.edg.org ]
– CrossGRID QoS – Real Time apps. [ www.crossgrid.org ]
– DataTAG
GLUE (EU-USA) [ www.datatag.org ]
– LCG
The LHC Computing GRID – Deployment [ cern.ch/lcg ]
– The new 16,2 B Euro EU VI Framework Prog. GEANT based GRID projects
• USA
– GriPhyN
[ www.griphyn.org ]
iVDGL-VDTv1
[ www.idvgl.org ]
• Asia
– ApGrid
[ www.apgrid.org ]
Pragma (USA-Asia)
Marcel Kunze, DESY Seminar
PPDG ( NSF, DoE )
[ www.ppdg.org ]
IST Grid Project Space
- Links with European National efforts
- Links with US projects (GriPhyN, PPDG, iVDGL,…)
Applications
EGSO
CROSSGRID
GRIA
GRIP
Middleware
& Tools
Underlying
Infrastructures
EUROGRID
GRIDLAB
DAMIEN
DATAGRID
DATATAG
Industry / business
Science
A. Baxevanidis
Marcel Kunze, DESY Seminar
The EU DataGrid Project
•
•
•
•
•
21 Partners
9.8 M Euros EU funding over 3 years
90% for middleware and applications
(Particle Physics, Earth Obs. and Biomedical)
Three year phased developments & demos
(2001-2003)
Spin-off:
– DataTAG (2002-2003)
– CrossGrid (2002-2004)
– GridStart (2002-2004)
Fabrizio Gagliardi
Marcel Kunze, DESY Seminar
DataGrid Partners
Main Partners
•CERN – International (Switzerland/France)
•CNRS - France
•ESA/ESRIN – International (Italy)
•INFN – Italy
•NIKHEF – The Netherlands
•PPARC - UK
Industrial Partners
•Datamat (Italy)
•IBM-UK (UK)
•CS-SI (France)
Research and Academic Institutes
Marcel Kunze, DESY Seminar
Fabrizio Gagliardi
•CESNET (Czech Republic)
•Commissariat à l'énergie atomique (CEA) – France
•Computer and Automation Research Institute,
Hungarian Academy of Sciences (MTA SZTAKI)
•Consiglio Nazionale delle Ricerche (Italy)
•Helsinki Institute of Physics – Finland
•Institut de Fisica d'Altes Energies (IFAE) - Spain
•Istituto Trentino di Cultura (IRST) – Italy
•Konrad-Zuse-Zentrum für Informationstechnik Berlin - Germany
•Royal Netherlands Meteorological Institute (KNMI)
•Ruprecht-Karls-Universität Heidelberg - Germany
•Stichting Academisch Rekencentrum Amsterdam (SARA) – Netherlands
•Swedish Research Council - Sweden
EU DataGrid Project Objectives
•
•
Use Grid technology to develop a sustainable
computing model for effective share of computing
resources and data for large scientific communities
Specific project objectives:
– Middleware for fabric & Grid management
– Large scale testbeds
– Production quality demonstrations
•
Key products:
– Resource broker (Compute elements, storage elements)
– Replica manager
– Virtual Organization manager
Marcel Kunze, DESY Seminar
EDG: Interaction with Sister Projects
• CrossGrid
• NorduGrid
– Using the same security certs.
– Testbed sites install EDG software
• Extending it for needs of intensive
interactive applications
– Using the same security certs.
– Involved in EDG architecture
work
• Good ideas for gatekeeper and
MDS configuration
• Helped develop GDMP and GSI
extensions for Replica Catalog
• Involved in GLUE schema work
• Security policy
– Participating in the EDG testing
activities
– Representatives in each projects
architecture & management groups
• DataTAG (EDT)
• iVDGL/GriPhyN/PPDG
– US members in EDG architecture
group
– Looking for common packaging
and toolkit usage solutions
Marcel Kunze, DESY Seminar
Fabrizio Gagliardi
– EDT is deploying EDG sw to
investigate inter-operability with
US projects
(iVDGL, GriPhyN, PPDG)
– Results feedback into EDG
software releases
(e.g. GLUE compatible
information providers/consumers)
– Middleware testing
– Working in WP8 (HEP
applications)
CrossGrid: An EDG Partner
•
New category of Grid enabled applications
–
–
–
–
•
•
•
•
Computing and data intensive
Distributed
Interactive, near real time response (a person in a loop)
Layered
New programming tools
Grid more user friendly, secure and efficient
Interoperability with other Grids
Implementation of standards
Marcel Kunze, DESY Seminar
CrossGrid Layered Architecture
Biomedical
Flood
Application Application
Applications
And Supporting
Tools
Applications
Development
Support
HEP Data
Mining on Grid
Application
MPI
Verification
Portal
Weather
Forecast
application
HEP High
LevelTrigger
Performance
Analysis
Metrics and
Benchmarks
MPICH-G
Globus
Replica
Manager
Grid
Common
Services
Datagrid
Job
Manager
GRAM
Local
Resources
HEP Interactive
Distributed Data
Access Application
Interactive
Distributed
Data Access
DataGrid
Replica
Manager
Replica
Catalog
Grid
Visualisation
Kernel
Data
Mining on
Grid
Grid
Resource
Management
User Interaction
Service
GSI
Globus-IO
MDS
Distributed
Data
Collection
Roaming
Access
Grid
Monitoring
GridFTP
GASS
Resource
Manager
Resource
Manager
Resource
Manager
Resource
Manager
Resource
Manager
Resource
Manager
Resource
Manager
Secondary
Storage
CPU
Optimization
of Data
Access
Scientific
Instruments
(Medical
Scaners,
Satelites,
Radars)
Detector
Local High
Level
Trigger
VR
systems
(Caves,
immerse
desks)
Visualization
tools
Tertiary
Storage
Marcel Kunze, DESY Seminar
Interactive Treatment Planning
The vascular geometry can be modified using a library of
models
Draw interactively
Computational geometry
Marcel Kunze, DESY Seminar
Biomedical Application
Sample pulse flow simulation
Marcel Kunze, DESY Seminar
Benefit of Standardization
Marcel Kunze, DESY Seminar
Fabrizio Gagliardi
Through links with sister projects, there is the
potential for a uniform global scientific applications grid
Common WorldGrid Demo at IST2002
VO centric Ganglia monitor
• WorldGrid Demo together with US and EU partners at IST2002
• Seamless interoperation of EDG with US Middleware (via GLUE)
Marcel Kunze, DESY Seminar
WorldGrid Lessons (+)
•
Advantages of the Grid:
– Possibility to execute tasks and move files over a
distributed computing infrastructure by using one single
personal certificate (no need to memorize dozens of
passwords)
– Possibility do distribute the workload adequately and
automatically, without logging in explicitly to each
remote system
– Possibility to do worldwide production in a perfectly
coordinated way, using identical software (RPMs),
scripts and databases
•
GLUE interfacing works to make the middleware
worlds talk to each other. Better solution: Agree on
open standards environment like OGSA !
Marcel Kunze, DESY Seminar
WorldGrid Lessons (-)
• EDG stability very much dependent on the quality of the
basic building blocks (Globus, Condor, etc)
• Globus support needs to be strengthened (formal contract,
European support center, internal Globus support team)
• Understand impact of OGSA and industrial involvement
– EDG Toolkit might need to be refactored
Marcel Kunze, DESY Seminar
EDG Future Plans
• Concentrate on production quality and real applications
• Educate new users and disseminate results
• Complete the program of work till end of 2003
• Port EDG to other platforms than Linux (Solaris)
• Port EDG to Globus ToolKit version 3 (OGSA)
• Make plans to conserve momentum and assets
in the EU FP6
Marcel Kunze, DESY Seminar
A Proposal to the EU
• EDG has already demonstrated the viability of Grid
technology
• EU had a fast start in supporting 18 Grid projects in
the last two years!
• RN Geant offers an excellent basis for a large
European Grid infrastructure
• The EU FP6 program should encourage and support
the deployment and production quality operation of a
large international Grid infrastructure open to
research and industry in Europe
Fabrizio Gagliardi
Marcel Kunze, DESY Seminar
Enabling Grids and e-Science in Europe
• EGEE: Integrated Infrastructure Initiative to support
European Research Area
• Vision: to create and deploy Grid technologies to enable
the widespread uptake of e-Science applications
throughout the European Research Area
• Four key objectives:
– integrating Grid technological developments from across Europe;
– establishing a Europe-wide Grid infrastructure for science and industry
with a focus on heterogeneity and interoperability;
– enabling the creation of e-Science applications from across the scientific
and industrial spectrum;
– ensuring the timely delivery of the project’s programme of work, guided
by the needs of academic and industrial partners.
Marcel Kunze, DESY Seminar
Fabrizio Gagliardi
Start by integration of the national Grid initiatives
GRID Computing Center GridKa
GRID Computing Center:
Infrastructure and Services
Competence Center:
Applications and e-Science
Marcel Kunze, DESY Seminar
Partners in the HGF
24.000 employees, 2.5 Billion Euro yearly budget
Grid Computing perfectly well matches the HGF vision of
“Konzertierte Forschung”
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
ALFRED-WEGENER-INSTITUT FÜR POLAR- UND MEERESFORSCHUNG AWI
DEUTSCHES ELEKTRONEN-SYNCHROTRON DESY
DEUTSCHES KREBSFORSCHUNGSZENTRUM DKFZ
DEUTSCHES ZENTRUM FÜR LUFT- UND RAUMFAHRT DLR
FORSCHUNGSZENTRUM JÜLICH FZJ
FORSCHUNGSZENTRUM KARLSRUHE FZK
GESELLSCHAFT FÜR BIOTECHNOLOGISCHE FORSCHUNG GBF
GEOFORSCHUNGSZENTRUM POTSDAM GFZ
GKSS-FORSCHUNGSZENTRUM GEESTHACHT GKSS
GSF-FORSCHUNGSZENTRUM FÜR UMWELT UND GESUNDHEIT GSF
GESELLSCHAFT FÜR SCHWERIONENFORSCHUNG GSI
HAHN-MEITNER-INSTITUT BERLIN HMI
MAX-PLANCK-INSTITUT FÜR PLASMAPHYSIK IPP
MAX-DELBRÜCK-CENTRUM FÜR MOLEKULARE MEDIZIN MDC
UFZ-UMWELTFORSCHUNGSZENTRUM LEIPZIG-HALLE UFZ
Marcel Kunze, DESY Seminar
A German Grid Initiative
D-GRID
•
•
•
•
•
Rostock
Kiel
Hamburg
Initially driven by the HGF
Global Upstream Oldenburg
centers and the DFN
Braunschweig
Hannover
Berlin
Magdeburg
Open to accept further partners
Bielefeld
Essen
Göttingen
in academia and industry
Leipzig
St. Augustin
Dresden
Marburg
Aim at a coordination of Grid
Ilmenau
Würzburg
Aachen
Frankfurt
activities
Erlangen
Heidelberg
Deployment of a new generation
Karlsruhe
Regensburg
Kaiserslautern
networking infrastructure
Stuttgart
Garching
Augsburg
(towards “TeraGrids”)
Promotion of open standards for
interfaces and protocols (GGF)
10 Gbit/s
2,4 Gbit/s
2,4 Gbit/s
622 Mbit/s
Marcel Kunze, DESY Seminar
Scientific Program
• Installation of a coordination office for Grid Computing
• Foster the enhanced application of IT in the field of
scientific and engineering disciplines
• Management of the relationship between the German
Grid competence centers and representation of their
common interests in the Global Grid Forum
• Training and education in the field of Grid Computing
and e-Science
• Deployment of national Grid resources and
development of a program for academic and industrial
projects
Marcel Kunze, DESY Seminar
Technical Program
• Goal: Integration of all kind of resources
e.g. sensors, computing infrastructure, people,…
• Deployment of a state-of-the-art production backbone
(2003: 10 GBPS, 2006: 100 GBPS)
• Discussion of safety and trust relationship
• Management of German Grid certificates
• Selection of middleware components and installation procedures
(Globus, UNICORE)
• Infrastructure: Find qualified manpower to offer and support the
additional services
• Definition of cost model (Grid-Accounting, credit points)
• Collaboration with the German HPC centers
=> D-GRID Kickoff Workshop in January 2003
(Wissenschaftszentrum Bonn)
Marcel Kunze, DESY Seminar
Where do we go from here??
Marcel Kunze, DESY Seminar
e-Science (enhanced Science)
Konzertierte Forschung
• e-Science is about more than networks, GRIDs, High
Performance Computing, cluster computing ...
• e-Science: „e-Science is about global collaboration in key
areas of science, and the next generation of infrastructure
that will enable it.“ (Dr. John Taylor, director of UK research
council)
• Foster the transition of data Grid to semantic Grid
– Federation of resources
– Federation of distinguished data sources
– Knowledge retrieval and collaboratories
Marcel Kunze, DESY Seminar
Semantic Grid:
Three Layer Grid Abstraction
e-Science
Information Grid
Computation/Data Grid
Automation
Marcel Kunze, DESY Seminar
Control
Knowledge Grid
The Future
• Where do we need to get to ?
– Applications to support an “e-society”
(“Cyber-Infrastructure”)
– An international Grid infrastructure which
hides the complexities from the users
(“Invisible Computing”)
– A powerful and flexible network
infrastructure
•
Grid-enabled
Applications
Where do we need to invest ?
– Applications targeted at realistic problems
in “e-science”
– Prototypes of Grid infrastructures
– Maintain and improve the GEANT network
Marcel Kunze, DESY Seminar
Prototype Grid
Infrastructures
Gèant: World Class
Networking