Transcript OGCE Build & Test

CloudCom 2010
Software for Science Gateways:
Open Grid Computing
Environments
Marlon Pierce, Suresh Marru
Pervasive Technology Institute
Indiana University
www.collab-ogce.org
Tutorial Agenda
• 1:00-2:15
– Introduction
– Computational chemistry workflow example
– Building the software
• 2:15-2:30: Break
• 2:30-5:00
– Hands on workflow: OREChem
– Hands on portal: Data mining
– Demo: gadget container
http://www.collabogce.org/ogce/index.php/Tutoria
ls
Link to demonstration movies.
TeraGrid is one of the largest investments in shared CI
from NSF’s Office of Cyberinfrastructure
TeraGrid resources today include:
•
Tightly Coupled Distributed Memory Systems, 2 systems in the top 10 at top500.org
–
–
•
•
•
Shared Memory•Data
Systems
Analysis and Vis systems
–
–
Cobalt (NCSA): Altix, 8 Tflop, 3 TB shared memory
•Longhorn (TACC): Dell/NVIDIA,
Pople (PSC): Altix, 5 Tflop, 1.5 TB shared memory
–
–
–
Abe (NCSA): 90 Tflops
•Data-Intensive Computing
Lonestar (TACC): 61 Tflops
•Dash (SDSC): Intel Nehalem, 544 processors, 4TB
QueenBee (LONI): 51 Tflops
CPU and GPU
•Nautilus (NICS): SGI UltraViolet, 1024 cores, 4TB
Clusters with Infiniband
global shared memory
•
•
flash memory
Condor Pool (Loosely Coupled)
–
•
Kraken (NICS): Cray XT5, 99,072 cores, 1.03 Pflop
Ranger (TACC): Sun Constellation,
62,976
cores, 579 Tflop,
123 systems:
TB RAM
But change
is constant
- new
•FutureGrid
Purdue- up to
22,000 cpus
–
•Experimental computing grid and cloud test-bed
tacklesupport
research challenges in computer science
Quarry (IU): virtualto
machine
–
–
high-performance computing
TeraDRE (Purdue):•Experimental,
48 node nVIDIA GPUs
Spur (TACC): 32 nVIDIA
GPUswith NVIDIA Tesla accelerators
system
Gateway hosting
•Keeneland
Visualization Resources
Storage Resources
–
–
–
GPFS-WAN (SDSC)
Lustre-WAN (IU)
Various archival resources
Source: Dan Katz, U Chicago
What Is a Science Gateway?
• Web and desktop user interfaces and user-centric Web
services for accessing Grid and Cloud resources.
– Clusters, supercomputers, mass storage
– Applications, databases
– Workflows
• Example Science Gateways from the NSF TeraGrid
–
–
–
–
GridChem: computational chemistry
UltraScan: biophysics computational analysis
LEAD: Atmospheric science
BioDrugScreen: drug docking, scoring, and discovery.
• Many others: see
https://www.teragrid.org/web/sciencegateways/gateway_list
• This tutorial is about software that powers gateways.
When is a gateway appropriate?
• Provide access to community applications
– WRF, Gaussian, CHARMM, Amber, BLAST, CCSM, UltraScan
– Create multi-scale workflows
• Provide access to community data sets
– National Virtual Observatory
– Earth System Grid
– Some groups have invested significant efforts here
• caBIG, extensive discussions to develop common terminology and
formats
• BIRN, extensive data sharing agreements
• Difficult to access data/advanced workflows
– Sensor/radar input
• LEAD, GEON
•
3 steps to connect a gateway to
Request an allocation TeraGrid
– Only a 1 paragraph abstract
required for up to 200k CPU hours
• Register your gateway
– Visibility on public TeraGrid
page
• Request a community account
– Run jobs for others via your portal
• Staff support is available!
• www.teragrid.org/gateways
SciDAC, Chattanooga, TN, July 16, 2010
OGCE Gateway Tool Adaption & Reuse
LEAD
LEAD
Experiment Builder, XRegistry Interface
GFac, XBaya,
XRegistry, FTR
Eventing System
GridChem
Xbaya, GC Middleware
GridChem
Resource
Discovery Service
TeraGrid
User Portal
GPIR, File
Browser
OGCE Team
Gadget Container,
GTLab, Javascript Cog,
XRegistry Interface,
Experiment Builder,
Axis2 Gfac, Axis2
Eventing System,
Resource Prediction
Service, Swarm
Ultrascan
OGCE
Re-engineer,
Generalize,
Build, Test and
Release
GFac, Eventing
System
OVP/
RST/
MIG
BioVLab
XBaya, GFac
ODI
Workflow Suite, Gadget Container
Bio Drug Screen
Swarm->GFac
EST Pipeline
Swarm->GFac
Future Grid
GFac, Xbaya, …
10
Software
OGCE Gadget
Container
XBaya
GFAC
XRegistry; Registry
Gadget
Experiment Builder
Description
Google Gadget/Open Social
compatible software for building Webbased user interfaces.
A visual user interface for composing,
launching and monitoring workflows
An application factory service for
wrapping command-line tools as Web
services
A service and workflow registry and its
user interface
User interface for creating online
experiments with registered
workflows
Science Gateways Layer Cake
User
Interfaces
Gateway
Services
Web/Gadg
et
Container
Web/Gadge
t Interfaces
Application
Abstractions
Fault
Tolerance
Workflow
System
Auditing &
Reporting
Resource
Middleware
Cloud Interfaces
Compute
Resources
Computational
Clouds
Color Coding
Web Enabled
Desktop
Applications
Application
Monitoring
Registry
Gateway
Abstraction
Interfaces
User
Management
Information
Services
Security
Provenance
& Metadata
Management
Grid Middleware
SSH & Resource
Managers
Computational
Grids
Local Resources
OGCE Gateway Components
Complimentary Gateway Components
Dependent resource provider components
XSUL/Apache Axis2
GFac Current & Future Features
Input
Handlers
Registry
Interface
Scheduling
Interface
Monitoring
Interface
Output
Handlers
Fault
Tolerance
Data Management
Abstraction
Auditing
Checkpoint
Support
Job Management
Abstraction
Globus
Campus
Resources
Amazon
Eucalyptus
Unicore
Condor
Color Coding
Existing Features
Planned/Requested Features
OGCE Layered Workflow Architecture:
Derived from LEAD Workflow System
Workflow
Interfaces (Design
& Definition)
XBaya GUI
(Composition,
Deploying, Steering
& Monitoring)
BPEL 2.0
Flex/Web
Composition
Gadget Interface for
Input Binding
Python
Scufl
Workflow
Specification
BPEL 1.0
Workflow
Execution & Control
Engines
Apache
ODE
GBPEL
Java Code
Pegasus DAG
Dynamic
Enactor
Condor
DAGMan
Jython Interpreter
Taverna
Putting It All Together
Software Strategy
• Focus on gadget container and tools for running
science applications on grids and clouds.
• Provide a tool set that can be used in whole or in
part.
– If you just want GFac, then you can use it without
buying an entire framework.
• Outsource security, information services, data
and metadata, etc to other providers.
– MyProxy, TG IIS, Globus, Condor, XMC Cat, iRods, etc.
More Information
• This is downloadable, packaged software.
– Apache Maven build system provides everything
you need to to build the gadget container,
gadgets, workflow composer, and backing
services.
– Get code by anonymous SVN checkout.
• Email: [email protected],
[email protected], [email protected]
• OGCE Web Site: www.collab-ogce.org
• Blog/News Feed: http://collabogce.blogspot.com/
Acknowledgements and People
• Funding by TeraGrid GIG, RP and by OCI SDCI
• IU: Marlon Pierce, Suresh Marru, Raminder
Singh, Archit Kulshrestha, Zhenhua Guo
• TACC: Maytal Dahan, Rion Dooley
• SDSC: Nancy Wilkins-Diehr, Jeff Sale
• SDSU: Mary Thomas
Demos Next
The OGCE Application Registry gadget allows users to
interactively register hosts and applications that are
dynamically wrapped as Web services.
The OGCE Gadget Container allows you to build portals out of
public and private Google Open Social gadgets. Supports HTTPS.
Downloadable, packaged software.
The OGCE Experiment Builder gadget allows users to
create projects and experiments out of previously
composed workflows.
OGCE Tools for Science Workflows
The XBaya workflow composer allows you to build
scientific workflows from services running across
the TeraGrid. This is part of our workflow suite.
What Is a Science Gateway?
• Web and desktop user interfaces and user-centric Web
services for accessing Grid and Cloud resources.
– Clusters, supercomputers, mass storage
– Applications, databases
– Workflows
• Example Science Gateways from the NSF TeraGrid
–
–
–
–
GridChem: computational chemistry
UltraScan: biophysics computational analysis
LEAD: Atmospheric science
BioDrugScreen: drug docking, scoring, and discovery.
• Many others: see
https://www.teragrid.org/web/sciencegateways/gateway_list
• This demo is about software that powers gateways.
Google Gadget-Based Science Gateways
PolarGrid
LEAD