Transcript The CrossGrid Project

The CrossGrid Project
Marcel Kunze, FZK
representing the X#-Collaboration
Main Objectives
New category of Grid enabled applications




Computing and data intensive
Distributed
Interactive, near real time response (a person in a loop)
Layered
New programming tools
Grid more user friendly, secure and efficient
Interoperability with other Grids
Implementation of standards
ACAT 2002 Moscow
Marcel Kunze - FZK
CrossGrid Collaboration
21 institutes Ireland:
11 countries TCD Dublin
Poland:
Netherlands:
UvA Amsterdam
LIP Lisbon
U.Linz
CSIC Santander
Valencia &
RedIris
UAB Barcelona
USC Santiago
& CESGA
ACAT 2002 Moscow
FZK Karlsruhe
TUM Munich
USTU Stuttgart
Austria:
Spain:
Portugal:
Germany:
Italy:
DATAMAT
Marcel Kunze - FZK
Cyfronet & INP Cracow
PSNC Poznan
ICM & IPJ Warsaw
Slovakia:
II SAS Bratislava
Greece:
Algosystems
Demo Athens
AuTh Thessaloniki
Cyprus:
UCY Nikosia
IST Grid Project Space
- Links with European National efforts
- Links with US projects (GriPhyN, PPDG, iVDGL,…)
Applications
EGSO
CROSSGRID
GRIA
GRIP
Middleware
& Tools
EUROGRID
GRIDLAB
DAMIEN
Underlying
Infrastructures
ACAT 2002 Moscow
DATAGRID
DATATAG
Industry / business
Marcel Kunze - FZK
Science
Collaboration with other
# Projects
Objective – Exchange of


Information
Software components
Partners




DATAGRID
DATATAG
GRIDLAB
EUROGRID and GRIP
GRIDSTART
Participation in GGF
ACAT 2002 Moscow
Marcel Kunze - FZK
Project Phases
M 4 - 12: first development phase: design,
1st prototypes, refinement of requirements
M 25 -32: third development phase:
complete integration, final code versions
M 33 -36: final phase:
M 1 - 3: requirements
definition and merging
demonstration and documentation
M 13 -24: second development phase:
integration of components, 2nd prototypes
ACAT 2002 Moscow
Marcel Kunze - FZK
Structure Overview
APPLICATIONS
Interactive simulation and visualisation of a biomedical system
Flooding crisis team support
Distributed Data Analysis in High Energy Physics
Weather forecast and air pollution modeling
TOOLS
Remote Data
Access
Optimization
Schedulers
Monitoring
Grid
Visualization
Kernel
Grid
SERVICES
FABRIC
INFRASTRUCTURE
Roaming
Access
Benchmarks
DATAGRID
SET OF TOOLS
Protocols, Authentication,
Authorization,
Access policy, Resource
management, etc.
GLOBUS TOOLKIT , Condor-G, ...
Network infrastructure, archivers, HPC/HPV systems,
Labour instruments, etc.
ACAT 2002 Moscow
Portals
Marcel Kunze - FZK
Local domain services
CrossGrid Architecture
Biomedical
Flood
Application Application
Applications
And Supporting
Tools
Applications
Development
Support
HEP Data
Mining on Grid
Application
MPI
Verification
Portal
Weather
Forecast
application
HEP High
LevelTrigger
Performance
Analysis
Metrics and
Benchmarks
MPICH-G
Globus
Replica
Manager
Grid
Common
Services
Datagrid
Job
Manager
GRAM
Local
Resources
HEP Interactive
Distributed Data
Access Application
Interactive
Distributed
Data Access
DataGrid
Replica
Manager
Replica
Catalog
Grid
Visualisation
Kernel
Data
Mining on
Grid
Grid
Resource
Management
User Interaction
Service
GSI
Globus-IO
MDS
Distributed
Data
Collection
Roaming
Access
Grid
Monitoring
GridFTP
GASS
Resource
Manager
Resource
Manager
Resource
Manager
Resource
Manager
Resource
Manager
Resource
Manager
Resource
Manager
Secondary
Storage
CPU
Optimization
of Data
Access
Scientific
Instruments
(Medical
Scaners,
Satelites,
Radars)
Detector
Local High
Level
Trigger
VR
systems
(Caves,
immerse
desks)
Visualization
tools
Tertiary
Storage
ACAT 2002 Moscow
Marcel Kunze - FZK
Layered Structure
Interactive and Data Intensive Applications (WP1)
 Interactive simulation and visualization of
a biomedical system
 Flooding crisis team support
 Distributed data analysis in HEP
 Weather forecast and air pollution modeling
HLA
DataGrid
GriPhyN
...
Grid Visualization
Kernel
Grid Application Programming
Environment (WP2)
Data Mining
 MPI code debugging and
verification
 Metrics and benchmarks
 Interactive and semiautomatic
performance evaluation tools
New CrossGrid Services (WP3)
Services
 Portals and roaming access
 Grid resource management
 Grid monitoring
 Optimization of data access
Globus Middleware
Fabric Infrastructure (Testbed WP4)
ACAT 2002 Moscow
Marcel Kunze - FZK
Scope of Applications
Applications in health and environment


Data federation, processing and interpretation in geographically
distributed locations
Fast, interactive decision making
Interactive access to distributed





Databases
Super computers and High Performance Clusters
Visualisation engines
Medical scanners
Environmental data input devices
ACAT 2002 Moscow
Marcel Kunze - FZK
Application Requirements
High quality presentation
High frame rate
Intuitive interaction
Real-time response
Interactive algorithms
High performance computing and networking
Distributed resources and data
ACAT 2002 Moscow
Marcel Kunze - FZK
Role of Network Latency
Communication delay and rendering delay are negligible
ACAT 2002 Moscow
Marcel Kunze - FZK
CrossGrid Application
Development (WP1)
Interactive simulation and visualisation of a biomedical
system

Grid-based system for pre-treatment planning in vascular
interventional and surgical procedures through real-time interactive
simulation of vascular structure and flow.
Flooding crisis team support
Distributed interactive data analysis in HEP

Focus on LHC experiments (ALICE, ATLAS, CMS and LHCb)
Weather forecast and air pollution modelling



Porting distributed/parallel codes on Grid
Coupled Ocean/Atmosphere Mesoscale Prediction System
STEM-II Air Pollution Code
ACAT 2002 Moscow
Marcel Kunze - FZK
Interactive Simulation and
Visualisation
of a Biomedical System
Grid-based prototype system for treatment planning in vascular
interventional and surgical procedures through near real-time
interactive simulation of vascular structure and flow.
The system will consist of a distributed near real-time simulation
environment, in which a user interacts in Virtual Reality (VR) and
other interactive display environments.
A 3D model of the arteries, derived using medical imaging techniques,
will serve as input to a simulation environment for blood flow
calculations.
The user will be allowed to change the structure of the arteries, thus
mimicking an interventional or surgical procedure.
ACAT 2002 Moscow
Marcel Kunze - FZK
Current Situation
Diagnosis & Planning
Treatment
Observation
ACAT 2002 Moscow
Marcel Kunze - FZK
Experimental Setup
ACAT 2002 Moscow
Marcel Kunze - FZK
Simulation Based Planning
and Treatment
Alternatives
Preop
ACAT 2002 Moscow
AFB w/
AFB w/
E-S Prox.
E-E Prox.
Anast.- FZK Anast.
Marcel Kunze
Angio w/ Angio w/
Fem-Fem Fem-Fem &
Fem-Pop
VR-Interaction
ACAT 2002 Moscow
Marcel Kunze - FZK
Flood Crisis Prevention
Support system for establishment and operation of Virtual Organization for
Flood Forecasting associating a set of individuals and institutions involved in
flood prevention and protection.
The system will employ a Grid technology to seamlessly connect together the
experts, data and computing resources needed for quick and correct flood
management decisions.
The main component of the system will be a highly automated early warning
system based on hydro-meteorological (snowmelt) rainfall-runoff simulations.
System will integrate the advanced communication techniques allowing the
crisis management teams to consult the decisions with various experts. The
experts will be able to run the simulations with changed parameters and
analyze the impact.
ACAT 2002 Moscow
Marcel Kunze - FZK
Virtual Organization for
Flood Forecasting
Data sources
surface automatic
meteorological and
hydrological
stations
meteorologica
l radars
systems for
acquisition and
processing of
satellite
information
Storage systems
External sources of information
Global and regional centers GTS
EUMETSAT and NOAA
Hydrological services of other
countries
HPC, HTC
Grid
infrastructure
meteorological models
hydrological models
databases
hydraulic models
Users
Flood crisis
teams
ACAT 2002 Moscow
meteorologists
hydrologists
hydraulic engineers
river authorities
energy
insurance companies
navigation
Marcel Kunze - FZK
media
public
Flood Crisis Prevention
Váh River Pilot Site
Nosice
(Outflow point)
Water
stages/discharges in
the real time
operating hydrological
stations
Strečno
(Inflow point)
Mapping of the
flooded areas
Pilot Site Catchment Area: 2500km2
(above Strečno: 5500km2)
Váh River Catchment Area:
19700km2, 1/3 of Slovakia
ACAT 2002 Moscow
Marcel Kunze - FZK
Flood Simulations
Results
Flow +
water
depths
ACAT 2002 Moscow
Marcel Kunze - FZK
Distributed Analysis in
High Energy Physics
Challenging points




Access to large distributed databases in the Grid
Development of distributed data-mining techniques
Definition of a layered application structure
Integration of user-friendly interactive access
(based on PROOF)
Focus on LHC experiments
(ALICE, ATLAS, CMS and LHCb)
ACAT 2002 Moscow
Marcel Kunze - FZK
PROOF
Selection
Parameters
TagD
B
CPU
Procedure
PROOF
RD
B
Proc.C
Proc.C
Proc.C
Proc.C
Proc.C
ACAT 2002 Moscow
DB
1
DB
2
DB
3
DB
4
DB
5
DB
6
Marcel Kunze - FZK
Local
CPU
Remote
CPU
CPU
CPU
CPU
Weather Forecast and Air
Pollution Modeling
Integration of distributed databases into Grid
Migration of data mining algorithms to Grid
Porting distributed atmospheric & wave models to Grid
Porting parallel codes for air quality models to Grid
Integration, testing and demonstration of the application in the
testbed environment
ACAT 2002 Moscow
Marcel Kunze - FZK
COAMPS
Coupled Ocean/Atmosphere Mesoscale Prediction System:
Atmospheric Components
Complex Data Quality Control
Analysis:



Multivariate Optimum Interpolation Analysis of Winds and Heights
Univariate Analyses of Temperature and Moisture
Optimum Interpolation Analysis of Sea Surface Temperature
Initialization:


Variational Hydrostatic Constraint on Analysis Increments
Digital Filter
Atmospheric Model:


Numerics: Nonhydrostatic, Scheme C, Nested Grids, Sigma-z
Physics: Convection, Explicit Moist Physics, Radiation, Surface Layer
Features:






Globally Relocatable (5 Map Projections)
User-Defined Grid Resolutions, Dimensions, and Number of Nested Grids
6 or 12 Hour Incremental Data Assimilation Cycle
Can be Used for Idealized or Real-Time Applications
Single Configuration Managed System for All Applications
Operational at:



7 Areas, Twice Daily, using 81/27/9 km or 81/27 km grids
Forecasts to 72 hours
Operational at all Navy Regional Centers (w/GUI Interface)
ACAT 2002 Moscow
Marcel Kunze - FZK
Status Quo … Quo Vadis ?
Current state (briefly)
Simulation done on a single system or local clusters
Visualisation on a single system, locally
What we are going to achieve
HPC, HTC, HPV in geographically distributed environment
Improved interaction with the end user
Near real time simulations
Different visualisation equipments
(adaptive according to the end-user needs), like



PDA
Workstations
VR studio (e.g. CAVE)
ACAT 2002 Moscow
Marcel Kunze - FZK
Grid Application
Programming Environment
(WP2)
MPI code debugging and verification
Metrics and benchmarks
Interactive and semiautomatic performance
evaluation tools
Specify, develop, integrate, test tools for HPC
and HTC applications on the Grid
ACAT 2002 Moscow
Marcel Kunze - FZK
New Grid Services and
Tools (WP3)
Portals and roaming access
Grid resource management
Grid monitoring
Optimisation of data access
Objectives





To develop interactive compute- and data-intensive applications
To develop user-friendly Grid environments
To offer easy access to the applications and Grid
To have reasonable trade-off between resource usage efficiency
and application speedup
To support management issues while accessing resources
ACAT 2002 Moscow
Marcel Kunze - FZK
International Testbed
Organisation (WP4)
Testbed setup &
incremental
evolution
Integration with
DataGrid
Infrastructure
support
Verification &
quality control
TCD Dublin
PSNC Poznan
U v Amsterdam
FZK Karlsruhe
USC Santiago
CSIC Santander
ICM & IPJ Warsaw
CYFRONET Cracow
II SAS Bratislava
LIP Lisbon
15 sites
CSIC Madrid
Auth Thessaloniki
U A Barcelona
CSIC Valencia
DEMO Athens
ACAT 2002 Moscow
Marcel Kunze - FZK
UCY Nikosia
Summary
Layered structure of all X# applications
Reuse of SW from DataGrid and other # projects
Globus as the bottom layer of the middleware
Heterogeneous computer and storage systems
Distributed development and testing of SW



12 partners in applications
14 partners in middleware
15 partners in testbeds
ACAT 2002 Moscow
Marcel Kunze - FZK
1980s: Internet 1990s: Web
2000s: Grid
Where do we need to get to ?
Applications to support an “e-society”
(“Cyber-Infrastructure”)
 An international Grid infrastructure which
hides the complexities from the users
(“Invisible Computing”)
 A powerful and flexible network infrastructure

Grid-enabled Applications
Where do we need to invest ?
Applications targeted at realistic problems in
“e-science”
 Prototypes of Grid infrastructures
 Maintain and improve the GEANT network

Expression of Interest for EU FP6 program:
“Enabling Grids and e-Science in Europe
(EGEE)”

ACAT 2002 Moscow
Prototype Grid Infrastructures
Gèant: World Class Networking
Marcel Kunze - FZK
EGEE Project Space
Applications
EGSO
CROSSGRID
GRIA
GRIP
Middleware
& Tools
Enabling Grids and E-Science in Europe (EGEE)
EUROGRID
DATAGRID
GRIDLAB
DAMIEN
Underlying
Infrastructures
ACAT 2002 Moscow
DATATAG
Industry / business
Marcel Kunze - FZK
Science
First results of EGEE
Brainstorming
Creation and support
E-Science centres
European
Infrastructure
Modulable
Testbeds
R&D Agenda
Semantic GRID
Database
Security
Deployment with
IT Industry
S/W Hardening
GLOBUS
EuroGrid, Gridlab etc.
ACAT 2002 Moscow
National
eScience
Centres
Integrated Project
ENABLING GRIDS
ESCIENCE EUROPE
EGEE
Science
Outreach
Applications in
Other Sciences
EIROforum
Marcel Kunze - FZK
Consulting
Prototyping
Deployment
Industry
Applications
Industry Outreach
Consulting
Training Courses
Dissemination
Forum
SMEs developing
Grid-enabled Applications
Tools and
Service Development