Transcript SA2: “Networking Support”

Enabling Grids for E-sciencE
The EGEE Infrastructure and
Remote Instruments
Erwin Laure
EGEE-II Technical Director
[email protected]
www.eu-egee.org
EGEE-II INFSO-RI-031688
EGEE and gLite are registered trademarks
EGEE
Enabling Grids for E-sciencE
Flagship grid infrastructure project co-funded by the European Commission
Now in 2nd phase with 91 partners in 32 countries
Main Objectives
• Operate a large-scale,
production quality grid
infrastructure for e-Science
• Attract new resources and
users from industry as well
as sciences
EGEE-II INFSO-RI-031688
RISGE - OGF22
2
EGEE – What do we deliver?
Enabling Grids for E-sciencE
•
Infrastructure operation
– Sites distributed across many countries
 Large quantity of CPUs and storage
 Continuous monitoring of grid services & automated site
configuration/management
 Support multiple Virtual Organisations from diverse research
disciplines
•
Access
Middleware
CLI
– Production quality middleware distributed
under open source licence
 Implements a service-oriented
architecture that virtualises resources
 Adheres to recommendations on web
service inter-operability and evolving
towards emerging standards

•
Security
API
Information & Monitoring
Authorization
Information &
Monitoring
Auditing
Authentication
Data Management
Metadata
Catalog
File & Replica
Catalog
Storage
Element
Data
Movement
Application
Monitoring
Workload Management
Job
Provenance
Package
Manager
Computing
Element
Workload
Management
Accounting
User Support
Managed process from first contact through to production usage
–
–
–
–
Training
Expertise in grid-enabling applications
Online helpdesk
Networking events (User Forum, Conferences etc.)
EGEE-II INFSO-RI-031688
RISGE - OGF22
3
Enabling Grids for E-sciencE
Archeology
Astronomy
Astrophysics
Civil Protection
Comp. Chemistry
Earth Sciences
Finance
Fusion
Geophysics
High Energy Physics
Life Sciences
Multimedia
Material Sciences
…INFSO-RI-031688
EGEE-II
250 sites
48 countries
50,000 CPUs
13 PetaBytes
>5000 users
>200 VOs
>140,000 jobs/day
32%
RISGE - OGF22
4
Application Examples
Enabling Grids for E-sciencE
• EGEE is used to analyze data coming from remote
instruments
• LHC
• Medical Imaging
• Earth Observation
• and many others
EGEE-II INFSO-RI-031688
RISGE - OGF22
5
Accelerating and colliding particles
Enabling Grids for E-sciencE
Large Hadron Collider
•
•
•
27 km circumference tunnel
Due to start up in 2008
40 Million Particle collisions per
second
– Online filter reduces to a few 100
“good” events per second recorded
on disk and magnetic tape at 1001,000 MegaBytes/sec
– ~15 PetaBytes per year for all four
experiments
•
Data analyzed by 100s of research
groups world wide
Mont Blanc
(4810 m)
Downtown Geneva
EGEE-II INFSO-RI-031688
RISGE - OGF22
6
The Data Acquisition
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
RISGE - OGF22
7
Acquisition, First pass reconstruction, Storage
Distribution
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
RISGE - OGF22
8
Data Distribution on the Grid
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
RISGE - OGF22
9
Medical Data Manager
Enabling Grids for E-sciencE
• Objectives
DICOM
Interface
SRM
– Expose an standard grid interface (SRM) for medical image
servers (DICOM)
– Fulfil application security requirements without interfering with
clinical practice
DICOM server
DICOM clients
EGEE-II INFSO-RI-031688
Worker Node
User Interface
RISGE - OGF22
10
Medical Data Registration
Enabling Grids for E-sciencE
1. Image is acquired
2. Image is stored in DICOM server
4. image metadata
are registered
3. lcg-put
AMGA Metadata
gfal
DICOM server
3a. Image is registered (a GUID is associated)
3b. Image key
is produced and
registered
LFC
Hydra
Key store
EGEE-II INFSO-RI-031688
RISGE - OGF22
11
Earth Science Applications in EGEE
Enabling Grids for E-sciencE
Flood of a Danube riverCascade of models
(meteorology,hydraulic
,hydrodynamic….)
UISAV(SK)ESA, UTV(IT),
KNMI(NL), IPSL(FR)Production and
validation of 7 years of
Ozone profiles from
GOME
Rapid Earthquake
analysis
(mechanism and
epicenter)
50- 100CPUs
IPGP(FR)
DKRZ(DE)- Data access
studies, climate impacts on
agriculture
Mars atmosphere CETP(
FR):
EGEE-II INFSO-RI-031688
Specfem3D:
Seismic
application.
Benchmark for
MPI (2 to 2000
CPUs) (IPGP,FR)
Geocluster for
Academy and
industry CGG(FR)Data mining
Meteorology &
Space Weather
(GCRAS, RU)
Air Pollution
model- BAS(BG)
Modelling seawater
intrusion in costal
aquifer (SWIMED)
CRS4(IT),INAT(TU),
Univ.Neuchâtel(CH)-
RISGE - OGF22
12
GOME
Enabling Grids for E-sciencE
Raw satellite data
from the GOME instrument
(~75 GB - ~5000 orbits/y)
Level 1
ESA(IT) – KNMI(NL)
Processing of raw GOME
data to ozone profiles.
2 alternative algorithms
~28000 profiles/day
Level 2
Meta Database
server
PosgreSQL
geospatial
search
EGEE-II INFSO-RI-031688
(example of 1 day total O3)
IPSL(FR)
Validate some of the
GOME ozone profiles (~106/y)
Coincident in space and time
with
Ground-Based measurements
Visualization
& Analyze
EGEE
environment
RISGE - OGF22
13
Summary
Enabling Grids for E-sciencE
• EGEE provides a unique environment for storing, managing, and
analyzing data from remote instruments
• Data production, collection, and initial processing is typically out
of band
– Instruments are not (and will not) be integrated in EGEE
– Data is initially stored at domain specific data stores not connected
to the Grid
• Sensor Grids are being established
– E.g. LOFAR
– Potential to be more directly integrated
• EGEE provides mechanisms to connect data collections to the
infrastructure such that they can be used on the infrastructure
EGEE-II INFSO-RI-031688
RISGE - OGF22
14