Transcript Marcel Kunze: The European CrossGrid Project

Forschungszentrum Karlsruhe
in der Helmholtz-Gemeinschaft
The European
CrossGrid Project
Marcel Kunze
Abteilung Grid-Computing und e-Science
Forschungszentrum Karlsruhe GmbH
On behalf of the CrossGrid Collaboration
Special thanks to Marian Bubak and Jesus Marco
www.eu-crossgrid.org
5th EU DataGrid Conference, Budapest, September 2002
Outline
The
CrossGrid Project
CrossGrid
Testbed
Aplications
Status
Deliverables
and Plans
5th EU DataGrid Conference, Budapest, September 2002
EU Funded Grid Project Space
(Kyriakos Baxevanidis)
- Links with European National efforts
- Links with US projects (GriPhyN, PPDG, iVDGL,…)
Applications
EGSO
CROSSGRID
GRIA
GRIP
Middleware
& Tools
EUROGRID
DAMIEN
DATAGRID
GRIDLAB
DATATAG
Underlying
Infrastructures
5 EU /DataGrid
Conference, Budapest, September 2002
Industry
business
th
Science
CrossGrid Collaboration
21 institutes
11 countries
Ireland:
Poland:
TCD Dublin
Netherlands:
UvA Amsterdam
Spain:
Portugal:
LIP Lisbon
CSIC Santander
Valencia &
RedIris
UAB Barcelona
USC Santiago
& CESGA
Germany:
FZK Karlsruhe
TUM Munich
USTU Stuttgart
Austria:
U.Linz
Italy:
DATAMAT
Cyfronet & INP Cracow
PSNC Poznan
ICM & IPJ Warsaw
Slovakia:
II SAS Bratislava
Greece:
Algosystems
Demo Athens
AuTh Thessaloniki
5th EU DataGrid Conference, Budapest, September 2002
Cyprus:
UCY Nikosia
Main Objectives

EU Vth Framework Programme (IST) Project, started March 2002

21 partners from 11 countries

New category of Grid enabled applications




Computing and data intensive
Distributed
Interactive, near real time response (a person in a loop)
Layered

New programming tools

Grid more user friendly, secure and efficient

Interoperability with other Grids

Implementation of standards
5th EU DataGrid Conference, Budapest, September 2002
Collaboration with other Grid Projects

Exchange of



Information
Software components
Partners




DATAGRID
DATATAG
GRIDLAB
EUROGRID and GRIP

GRIDSTART

Participation in GGF
5th EU DataGrid Conference, Budapest, September 2002
Workpackages

WP1 – CrossGrid Application Development








Weather forecast and air pollution modeling
Tools for parallel programming and debugging on the Grid (MPI)
Portals and roaming access
Resource management
Monitoring
Optimisation of data access
WP4 - International Testbed Organisation



Interactive distributed data analysis in HEP
WP3 – New Grid Services and Tools


Flooding crisis support
WP2 - Grid Application Programming Environments


Biomedical simulation and visualization
Integration team, certification authority, support and test procedures
Based on EDG 1.2 (see talk of Marcus Hardt in WP6 parallel session)
WP5 - Project Management

Coordination, architecture, dissemination
5th EU DataGrid Conference, Budapest, September 2002
Biomedical Application
CT / MRI scan
Segmentation
Medical
DB
LB flow
simulation
Medical
DB
10 simulations/day
60 GB/simulation
> 20 MB/s
5th EU DataGrid Conference, Budapest, September 2002
Visualization
VE
WD
PC
PDA
HDB
Interaction
VR-Interaction
5th EU DataGrid Conference, Budapest, September 2002
Cascade of Flood Simulations
Data sources
Meteorological
simulations
Users
Hydrological
simulations
Hydraulic simulations
Output visualization
5th EU DataGrid Conference, Budapest, September 2002
Example of the Flood Simulation - Flow and Water Depth
5th EU DataGrid Conference, Budapest, September 2002
Distributed Data Analysis in HEP

Objectives



Distributed data access
Distributed data mining techniques with neural networks
Issues





Typical interactive requests will run on o(TB) distributed data
Transfer/replication times for the whole data about one hour
Data transfers once and in advance of the interactive session
Allocation, installation and set-up of corresponding database servers
before the interactive session
Integration of user-friendly interactive access
(based on PROOF)
5th EU DataGrid Conference, Budapest, September 2002
Parallel ROOT Facility: PROOF
Selection
Parameters
TagD
B
CPU
Procedure
PROOF
RD
B
Proc.C
Proc.C
Proc.C
Proc.C
Proc.C
http://root.cern.ch
DB
1
DB
2
DB
3
DB
4
DB
5
DB
6
Local
CPU
Remote
CPU
CPU
CPU
CPU
5th EU DataGrid Conference, Budapest, September 2002
Weather Forecast and Air Pollution Modeling

Distributed/parallel codes on the Grid


Coupled Ocean/Atmosphere Mesoscale Prediction System
STEM-II Air Pollution Code

Integration of distributed databases

Data mining applied to downscaling weather forecast
5th EU DataGrid Conference, Budapest, September 2002
Key Features of CrossGrid Applications

Data



To be selected on demand
Processing



Data sources and data bases geographically distributed
Large processing capacity required; both HPC & HTC
Interactive
Presentation


Complex data requires versatile 3D visualisation
Support for interaction and feedback to other components
5th EU DataGrid Conference, Budapest, September 2002
Overview of the CrossGrid Architecture
Applications
Supporting Tools
1.1
BioMed
2.2 MPI
Verification
1.2
Flooding
2.3
Metrics and
Benchmarks
Applications
Development
Support
App. Spec
Services
Generic
Services
Fabric
MPICH-G
1.1
User Interaction
Services
3.2
Scheduling
Agents
DataGrid Job
Submission
Service
Resource
Manager
(CE)
CPU
1.3 Interactive
Distributed
Data Access
1.3 Data
Mining on
Grid (NN)
2.4
Performance
Analysis
3.1 Portal &
Migrating
Desktop
1.4
Meteo
Pollution
1.1, 1.2 HLA
and others
1.1 Grid
Visualisation
Kernel
3.4
Optimization of
Grid Data Access
GRAM
Resource
Manager
(SE)
Secondary
Storage
GridFTP
3.3
Grid
Monitoring
GIS / MDS
Resource
Manager
3.4
Optimization of
Local Data Access
3.1
Roaming
Access
GSI
Resource
Manager
Instruments
( Satelites,
Radars)
Tertiary Storage
5th EU DataGrid Conference, Budapest, September 2002
Globus-IO
DataGrid
Replica
Manager
Globus
Replica
Manager
Replica
Catalog
Replica
Catalog
Status after M6++

Software Requirements Specifications together with use cases

CrossGrid Architecture defined

Detailed Design documents for tools and the new Grid services
(OO approach, UML)

Analysis of security issues and the first proposal of solutions

Detailed description of the test and integration procedures

Testbed first experience

Sites: LIP, FZK, CSIC+USC, PSNC, AuTH+Demo

Basic: EDG release 1.2

Applications:


EDG HEP simulations (Atlas,CMS)
first distributed prototypes using MPI:
NN distributed training
Evolutionary Algorithms


5th EU DataGrid Conference, Budapest, September 2002
CrossGrid Testbed Map
Géant
TCD Dublin
USC Santiago
PSNC Poznan
UvA Amsterdam
ICM & IPJ Warsaw
FZK Karlsruhe
CYFRONET Cracow
CSIC-UC IFCA
Santander
II SAS Bratislava
LIP Lisbon
CSIC RedIris
Madrid
Auth Thessaloniki
UAB Barcelona
CSIC IFIC Valencia
DEMO Athens
5th EU DataGrid Conference, Budapest, September 2002
UCY Nikosia
Sample Testbed Sites
Valencia GoG farm, Santander (GridWall), FZK
5th EU DataGrid Conference, Budapest, September 2002
Plans for the Future

Participation in production testbed with DataGrid




Common DEMO at IST 2002, Copenhagen, November 4th-6th
Collaboration with DataGrid in specific points
(e.g. user support and helpdesk software)
Conference together with RI Forum and the “Across Grids”



All sites will be ready to join by end of September
Santiago de Compostella, Spain, February 9th-14th,2003
With Proceedings (reviewed papers)
CrossGrid workshop, Linz (w/ EuroPVM/MPI 2002), September 28th-29th
5th EU DataGrid Conference, Budapest, September 2002
Linz CrossGrid Workshop Sep.28th-29th


Evaluate the current status of all tasks
Contact partners who are developing software we are going to use:
discuss interfaces and functionality

Understand what we may expect as first prototypes

Coordinate the operation of testbeds

Agree about common rules for software development:
Standard Operational Procedures (SOP) document written

Start to organize the first CrossGrid EU review

Meet with EU DataGrid representatives

Discuss the technology for the future (OGSA)
Details at http://www.gup.uni-linz.ac.at/crossgrid/workshop/
5th EU DataGrid Conference, Budapest, September 2002
CrossGrid Schedule and Deliverables

M6 (now)




WP5 CrossGrid website
WP4 First testbed prototype release
M11 (Feb. 2003)




WP4 internal status report
M10 (Jan. 2003)


WP4 are working on launching the first testbed.
M9 (Dec.2002)


WPs 1-3 start to develop prototypes of the applications, tool environment, and the new grid services
WP5 Project leaflet/brochure
WP5 Dissemination and exploitation report
CrossGrid conference in Santiago di Compostella, February 2002
M12 (March 2003)





WP1 First software release
WP2 First prototypes
WP3 First prototypes
WP5 Report on requirements on integration and interoperability with DataGrid
WP5 Yearly report
5th EU DataGrid Conference, Budapest, September 2002
1980s: Internet 1990s: Web
2000s: Grid
 Where do we need to get to ?




A Grid infrastructure which hides the complexities from the users
(“Invisible Computing”)
A powerful and flexible network infrastructure
Where do we need to invest ?




Applications to support an “e-society”
(“Cyber-Infrastructure”)
Applications targeted at realistic problems in “e-science”
Prototypes of Grid infrastructures
Maintain and improve the GEANT network
Expression of Interest for EU FP6 program:

Grid-enabled
Applications
“Enabling Grids and e-Science in Europe (EGEE)”
5th EU DataGrid Conference, Budapest, September 2002
Prototype Grid
Infrastructures
Gèant: World Class
Networking