Transcript Presentation

TAT
CrossGrid in Its Halfway:
Achievements and Challenges
Marian Bubak, Michal Turala and the CrossGrid Collaboration
www.eu-crossgrid.org
Cracow Grid Workshop, October 27 – 29, 2003
TAT
Outline
 Applications
 Overview of X# architecture
 Tool environment
 Grid middleware
 X# and OGSA
 Software development, standards, bodies, …
 Summary
Cracow Grid Workshop, October 27 – 29, 2003
TAT
Medical Application
LB flow
simulation
 Adding small modifications to the
proposed structure results in
immediate changes in the blood flow
Visualization
 Online presentation of simulation
results via a 3D environment
 The progress of the simulation and
the estimated time of convergence
should be available for inspection
Interaction
Cracow Grid Workshop, October 27 – 29, 2003
VE
WD
PC
PDA
Basic Features of Flood Simulation
 Meteorological
• Intensive simulation
(HPC), large input/output
data sets, high availability
of resources
TAT
Data sources
Meteorological
simulations
 Hydrological
• Parametric simulations
(HTC) may require
different models
(heterogeneous
simulations)
Hydrological
simulations
Users
Hydraulic
simulations
 Hydraulic
• Many 1-D simulations
HTC, 2-D hydraulic
simulations require HPC
Output visualization
Cracow Grid Workshop, October 27 – 29, 2003
TAT
Distributed Data Analysis in HEP
 Objectives
• Access to distributed
databases
• Data mining with neural
networks
 Issues
• Typical interactive requests will
run on o(TB) of distributed data
• Transfer/replication may take
hours
• Data are transferred before
interactive session
• Allocation, installation and
setup of the database servers
before the interactive session
Interactive
Session
Resource
Broker
Replica
Manager
Portal
XML in/out
On-line
output
Interactive
Session
Manager
Interactive
DISTRIBUTED
Interactive
Session
Interactive
Session
Interactive PROCESSING
Worker
Session
Interactive
Worker
Session
Worker
Session
Worker
Worker
Cracow Grid Workshop, October 27 – 29, 2003
DB Installation
Interactive Session
Database server
TAT
Weather Forecast and Air Pollution Modeling
 Distributed/parallel code on Grid
• Coupled Ocean/Atmosphere Mesoscale Prediction
System
• STEM-II Air Pollution Code
• Integration of distributed databases
 Data mining applied to downscaling weather forecasts
Cracow Grid Workshop, October 27 – 29, 2003
TAT
Interactions in the CrossGrid Applications
 Interactive steering
 Control of a cascade of simulations
 Distributed data access
 Parameter study with online results
 Interactive estimation of simulation time
Cracow Grid Workshop, October 27 – 29, 2003
TAT
Features of Interactive Grid Computing
 Run time application control
• Performance data on-line
 Possible effects of decisions
• Access to benchmark information
 Interpreting application’s behavior in heterogeneous
open system
• Access to infrastructure performance
 Information meaningful in the context of application field
• more application specific performance data
 Need in on-line standard and user-defined metrics
Cracow Grid Workshop, October 27 – 29, 2003
TAT
CrossGrid Architecture
Application
Applications
MPICH-G
Collective Layer
Libraries
Supporting
Tools
Portal and
HLA
Benchmarks
Tools
Migrating Desktop
Application
Specific Services
Infrastructure
Roaming Access
User Interaction
Grid Visualization
Services
Kernel
OCM-G
Monitoring
Scheduling
DataGrid Data
Agent
Management
Generic
Services
DataGrid Job
Globus
Management
Toolkit
Cracow Grid Workshop, October 27 – 29, 2003
Data Access
TAT
Migrating
Desktop
Application
Tool
Benchmark
OCM-G
Grid Visualization
Kernel
Portal and Roaming
Access
Simulation
Output
(Parallel)
Application
Running
User Interaction
Services
Infrastructure
Monitoring
Scheduling
Agent
Data
Access
DataGrid Data
Management
DataGrid Job
Management
Globus Toolkit
Multiple
Sites
Cracow Grid Workshop, October 27 – 29, 2003
TAT
8 rank 1 performs MPI_Recv
8 Rank 0 is pending!
8 Rank 1 is pending!
WARNING: deadlock detected
Tool
Application
Benchmark
MARMOT MPI
Verification Tool
OCM-G
Grid Visualization
Kernel
Portal and Roaming
Access
User Interaction
Services
(Parallel)
Application
Running
Infrastructure
Monitoring
Scheduling
Agent
Data
Access
DataGrid Job
Management
DataGrid Data
Management
Globus Toolkit
Multiple
Sites
Cracow Grid Workshop, October 27 – 29, 2003
TAT
Tool
Application
Benchmark
OCM-G
Portal and Roaming
Access
Performance
Prediction Tool
(Parallel)
Application
Running
Grid Visualization
Kernel
User Interaction
Services
Infrastructure
Monitoring
Scheduling
Agent
Data
Access
DataGrid Job
Management
DataGrid Data
Management
Globus Toolkit
Multiple
Sites
Cracow Grid Workshop, October 27 – 29, 2003
TAT
Tool
Application
Benchmark
OCM-G
Grid Visualization
Kernel
Portal and Roaming
Access
G-PM Performance
Analysis Tool
User Interaction
Services
(Parallel)
Application
Running
Infrastructure
Monitoring
Scheduling
Agent
Data
Access
DataGrid Job
Management
DataGrid Data
Management
Globus Toolkit
Multiple
Sites
Cracow Grid Workshop, October 27 – 29, 2003
Site Administrator
Tool
TAT
Application
Benchmark
OCM-G
Jiro-based
Infrastructure
Monitoring
(Parallel)
Application
Running
Grid Visualization
Kernel
Portal and Roaming
Access
User Interaction
Services
Infrastructure
Monitoring
Scheduling
Agent
Data
Access
DataGrid Job
Management
DataGrid Data
Management
Globus Toolkit
Multiple
Sites
Cracow Grid Workshop, October 27 – 29, 2003
Site Administrator
Tool
TAT
Application
Benchmark
OCM-G
SANTA-G Network
Packets Monitoring
Grid Visualization
Kernel
Portal and Roaming
Access
User Interaction
Services
(Parallel)
Application
Running
Infrastructure
Monitoring
Scheduling
Agent
Data
Access
DataGrid Job
Management
DataGrid Data
Management
Globus Toolkit
Multiple
Sites
Cracow Grid Workshop, October 27 – 29, 2003
TAT
Migrating
Desktop
Application
Tool
Benchmark
OCM-G
Grid Visualization
Kernel
Portal and Roaming
Access
Benchmark
Results
Benchmark
Running
User Interaction
Services
Infrastructure
Monitoring
Scheduling
Agent
Data
Access
DataGrid Data
Management
DataGrid Job
Management
Globus Toolkit
Multiple
Sites
Cracow Grid Workshop, October 27 – 29, 2003
TAT
Tool
Application
Benchmark
OCM-G
Estimation of Data
Access Time for
Replica
Management
Grid Visualization
Kernel
Portal and Roaming
Access
User Interaction
Services
Infrastructure
Monitoring
Scheduling
Agent
Data
Access
DataGrid Job
Management
Handling
Multiple
Mass
Storage
Systems
DataGrid Data
Management
Globus Toolkit
Cracow Grid Workshop, October 27 – 29, 2003
TAT
Sites of the CrossGrid Testbed
Géant
TCD Dublin
UvA Amsterdam
PSNC Poznan
ICM & IPJ Warsaw
FZK Karlsruhe
USC Santiago
CYFRONET Cracow
CSIC-UC IFCA
Santander
LIP Lisbon
CSIC RedIris Madrid
II SAS Bratislava
Auth Thessaloniki
UAB Barcelona
CSIC IFIC Valencia
DEMO Athens
Cracow Grid Workshop, October 27 – 29, 2003
UCY Nicosia
Resources in CG Testbed
Configuration
Machine
(LCFG)
TAT
 At each site
User
Interface
•
•
•
•
•
3Com
LCFG configuration server
User Interface
Gatekeeper (Computing Element)
Worker Nodes
Storage Element
 16 sites:
IDC
Storage
Element
Worker Nodes Gatekeeper
• 115 CPUs (Worker Nodes)
• 4 TB (Storage Elements)
 National Certification Authority machines
 Grid services (LIP)
Monitoring
CA machine
(offline)
•
Data General
Information Index
•
MyProxy
•
Resource Broker
•
•
3Com
•
Resource
Broker
Replica
Catalogue
IDC
Virtual
Organization
Short lived certificates for long lived processes, used by RB
Virtual Organization Server
•
•
Database for physical replica file location
Certificate Proxy Server
•
•
Matchmaking and load balancing scheduler
Replica Catalogue
•
IDC
Top MDS Information Server, points to site Information Servers
Database for user authentication (CROSSGRID VO)
Monitoring
•
Mapcenter: network monitoring system
Cracow Grid Workshop, October 27 – 29, 2003
And with Related Grids ...
Cracow Grid Workshop, October 27 – 29, 2003
TAT
TAT
X# and OGSA
 OGSA – Open Grid Services Architecture
• General vision of Grid made of services
 OGSI – Open Grid Services Infrastructure
• Defined conventions and basic interfaces of Grid
Services
• OGSI-compliant Grid Service
 Globus Toolkit 3.0
• Contains OGSI-implementation
• OGSI-based higher level Globus Toolkit services
Cracow Grid Workshop, October 27 – 29, 2003
TAT
Migration to GT3
 Replacing GT2 with GT3 on our testbed
 More technical than architectural issue
 CrossGrid uses Globus indirectly (through EDG)
 No stable relese of GT3 yet
 No direct requirements for using GT3
• However experimental installation for Task 1.1
Cracow Grid Workshop, October 27 – 29, 2003
TAT
Migration to OGSA
 Functionalities of CrossGrid from the beginning
designed as services
 Web Service protocols (SOAP) widely used
 Easy to be made OGSI-compliant
• when required
• when stable implementation available
Cracow Grid Workshop, October 27 – 29, 2003
TAT
Tools and OGSA
 MARMOT – MPI verification
• Independent of OGSA – can use any MPI
implementation
 Benchmarks
• Can publish benchmark results in Grid
information systems
 Performance evaluation and prediction tools
• Basically independent of OGSA
Cracow Grid Workshop, October 27 – 29, 2003
TAT
Services
 Portal and Roaming Access
• RAS uses SOAP protocol to communicate
• Little effort to migrate to OGSA
 Scheduling Agents
• Dependent on EDG and Condor
• OGSI-compliant interface can be added
 Monitoring
• Application monitoring is not going to migrate to OGSA – due to
performance issues, however feasibility studies are performed
• Infrastructure monitoring is compatible with EDG R-GMA that is
going to migrate to OGSA
 Optimization of Data Access
• Ready to migrate, SOAP protocol already used
Cracow Grid Workshop, October 27 – 29, 2003
TAT
Software Development and Testing
• Proceeds through all the development phases
• Requires gathering of quality indicators and
producing monthly reports
• Crucial for ensuring quality of code
• Currently in progress
Cracow Grid Workshop, October 27 – 29, 2003
TAT
Static Code Metrics
Measured separately for each task:
• amount of new lines of code since last report
• compliance with coding conventions
• count of packages, functions and inner classes
• complexity number (McCabe metric)
• count of formal Javadoc comments per class
and method
Cracow Grid Workshop, October 27 – 29, 2003
TAT
X# Management in Short
Main Bodies
• Project Coordinator
• CG Office
• Steering Group
• Technical Board
• Architecture Team
• Quality Assurance
• Central Dissemination
Better focus on technical
issues and quality control
Cracow Grid Workshop, October 27 – 29, 2003
TAT
Collaboration with DataGrid
 Architecture
• interoperability, reuse of MW components
• joint proposals to GGF
 Interactive applications
• methodology, generic structure, grid services
• security issues for medical applications
 Tools: benchmarks and application monitoring
 Interoperability of EDG testbeds
• collaboration of testbed support teams
• mutual recognition of Certification Authorities
• common access/usage policy and procedures
• common installation/configuration procedures
• joint infrastructure for HEP applications
 Software Testing
Cracow Grid Workshop, October 27 – 29, 2003
TAT
Collaboration with Other Grid Projects
 GRIDSTART
• Active participation in Technical Board meetings
• Conferences: Cracow Grid Workshop, Across Grids (Santiago) – to
stimulate exchange of information and personal contacts
 GLOBUS
• Regular contacts
• CrossGrid use cases for OGSA
 GridLab
• Development of grid applications
• Application monitoring
• Portals, data access, security
• Participation in joint testbeds (SC’2002)
 EuroGrid and GRIP
• Regular contacts and exchange of information
Cracow Grid Workshop, October 27 – 29, 2003
TAT
Research Areas in CG
(about 90 papers)
 New paradigms for grid application programming
 Grid PSE for complex problems
 Collaborative environments
 Grid monitoring and benchmarking
 Monitoring of grid applications
 Performance of grid applications (analysis and prediction)
 Grid services for interactive applications
 Building of Virtual Organizations (tools, rules)
 Methods and tools for grid middleware deployment
 In F. Berman, G. Fox, T. Hey (eds): Grid Computing, Wiley 2003
 Participation in 6FP projects (EGEE, Oct 15 – 5?)
 GGF – 2 x BOF, and plans for GGF10 in Frankfurt; ICCS’2004
Cracow Grid Workshop, October 27 – 29, 2003
TAT
www.eu-crossgrid.org
•21 partners
•5 Work Packages:
•WP1 – CrossGrid Applications
•WP2 – Grid Tool Environment
•WP3 – New Grid Services
•WP4 - International Testbed
•WP5 – Project Management
Cracow Grid Workshop, October 27 – 29, 2003