Transcript LEAD_OGCE_WorkflowSuite

OGCE Workflow and LEAD
Overview
Suresh Marru, Marlon Pierce
September 2009
The LEAD Gateway
Suresh Marru
LEAD Pioneering Technology
Suresh Marru
LEAD Dynamic Adaptive Infrastructure
Storms Forming
Forecast Model
Streaming
Observations
Data Mining
Instrument Steering
Refine forecast grid
Suresh Marru
LEAD Scientists and Educational Interactions
Developers
Researcherss
Lowering the barrier for using
complex end-to-end weather
technologies
End Users
Democratize
Empower
Facilitate
Suresh Marru
Analyze &
Predict
Discover
&Visualize
Research &
Reproducibility
Education &
Outreach
Suresh Marru
Open Grid Computing Environments
Gateways
(LEAD,
GridChem,
…)
TeraGrid
User
Portal
TG GIG
Generalize,
Harden, Build
Test
Gateways/E-Science Community
Suresh Marru
Flexible Layered Service Oriented Architecture
User Interactions
Web Portal
XBaya GUI
Other Clients
XBaya Core
Event Bus
Middleware Services
Workflow
Engine (ODE)
Computational
Cloud
XRegistry
GFac Services
Compute & Data Resources
Computational
Grids
XMCCat
Metadata
Catalog
Local Lab
Resources
Suresh Marru
Application Wrapper Framework
• Scientific Applications are wrapped
into web services by filling in a webform.
• The Application Factory generates a
web service for each application
with I/O interfaces.
• Registers WSDL for the service with
a registry
• Each service generates a stream of
notifications that log the service
actions back to the XMCCat
Metadata Catalog, user monitoring,
and provenance tracking tools
Application Factory
c
App
Service
Run program
& publish events
Suresh Marru
Application Wrapper Architecture
Suresh Marru
Service Monitoring via Events
• The service output is a stream of
events
•
•
•
•
•
•
I am running your request
I have started to move your input files
I have all the files
I am running your application
The application is finished
I am moving the output to you file
space
• I am done
• These are automatically generated
by the service using a
distributed event system
(WS-Eventing / WS-Notification)
• Topic based pub-sub system with
a well known “channel”
Application
Service
Instance
6
5
4
3
2
1
Notification
Channel
x
listener
x
publisher
Suresh Marru
Workflow Suite Architecture
Suresh Marru
Interoperable Xbaya Architecture
BPEL 1.1
BPEL 2.0
GPEL
Engine
Apache
ODE Engine
SCUFL
Composition
and
Monitoring
Abstract DAG
Model
Dynamic
Enactor/Interpreter
Python
Jython
Based Enactor
Taverna
Python Runtime
Message Bus
Suresh Marru
Simple Recovery Architecture
BPEL
Workflow
Engine
Portal
Application
Performance
Models
Resource
Reliability
Models
Fault Tolerance/
Recovery Service
NWS, MDS
BQP
OVP/
RST/
MIG
Application
Service
Notification
Service
Deadline &
Success
Probability
Suresh Marru
14
Workflow Composer
Suresh Marru
Example: Assimilation Workflow
Suresh Marru
Science Independence Example
• Domain webs of large genomes
– Input list of amino acid sequences
– Identify all known domains
– Construct webs
Ensemble-type processing
(minimal network reqs)
Capacity-type computing
Parallel processing
Capability-type computing
Suresh Marru
Biology Gateway
Suresh Marru
Polar Grid Example
Suresh Marru
Example Gateways using OGCE Workflow Suite
LEAD
Lower entry barrier to using weather analysis tools
Improve detection, analysis & prediction of mesoscale weather
Motif-Network
Transformation of sequenced genomes to “domain-space”
Cyber-Infrastructure Evaluation
Performance evaluation of future supercomputer architectures
ADAM
Algorithms for feature extraction, data normalization, classification
and normalization
GridChem
Molecular Chemistry Grid helping researchers run chemistry
applications on Grid Environment
Suresh Marru
Live Demo & Questions?
Suresh Marru