Transcript No Slide Title

Grid Computing
Ian Foster
Mathematics and Computer Science Division
Argonne National Laboratory
and
Department of Computer Science
The University of Chicago
http://www.mcs.anl.gov/~foster
Partial Acknowledgements




2
Open Grid Services Architecture design

Carl Kesselman, Karl Czajkowski @ USC/ISI

Steve Tuecke @ANL

Jeff Nick, Steve Graham, Jeff Frey @ IBM
Grid services collaborators at ANL

Kate Keahey, Gregor von Laszewski

Thomas Sandholm, Jarek Gawor, John Bresnahan
Globus Toolkit R&D also involves many fine
scientists & engineers at ANL, USC/ISI, and
elsewhere (see www.globus.org)
Strong links with many EU, UK, US Grid projects
 Support from
[email protected]
DOE, NASA, NSF, IBM, Microsoft
ARGONNE  CHICAGO
3
Goals


Communicate the purpose, significance,
state, adoption, & future of Grid technology
Persuade you that Grid technology
represents an opportunity


Grids aren’t (particularly) about science or
servers—themes of virtualization, service
discovery, service management, and QoS
delivery are universal
Rapid uptake in industry & science represents
an exceptional opportunity for impact
[email protected]
ARGONNE  CHICAGO
4
Overview


Origins: Resource sharing within scientific
collaborations

Science drivers & science Grid projects

Globus Toolkit
Evolution: Resource virtualization

Commercial drivers

OGSA: Grid meets Web services
[email protected]
ARGONNE  CHICAGO
5
Overview


Origins: Resource sharing within
scientific collaborations

Science drivers & science Grid projects

Globus Toolkit
Evolution: Resource virtualization

Commercial drivers

OGSA: Grid meets Web services
[email protected]
ARGONNE  CHICAGO
6
E-Science: The Original Grid Driver

Pre-electronic science


Theorize &/or experiment, in small teams
Post-electronic science

Construct and mine very large databases

Develop computer simulations & analyses

Access specialized devices remotely


Exchange information within distributed
multidisciplinary teams
Need to manage dynamic, distributed
infrastructures, services, and applications
[email protected]
ARGONNE  CHICAGO
eScience Application:
Sloan Digital Sky Survey Analysis
7
Size distribution of
galaxy clusters?
Galaxy cluster
size distribution
100000
Chimera Virtual Data System
+ GriPhyN Virtual Data Toolkit
+ iVDGL Data Grid (many CPUs)
10000
1000
100
10
[email protected]
1
1
10
Number of Galaxies
100
www.griphyn.org/chimera
ARGONNE  CHICAGO
NASA’s Information Power Grid:
Aviation Safety
8
Wing Models
•Lift Capabilities
•Drag Capabilities
•Responsiveness
Airframe Models
Stabilizer Models
•Deflection capabilities
•Responsiveness
Crew Capabilities
- accuracy
- perception
- stamina
- re-action times
- SOPs
Engine Models
Human Models
•Braking performance
•Steering capabilities
•Traction
•Dampening capabilities
Landing Gear Models
[email protected]
•Thrust performance
•Reverse Thrust performance
•Responsiveness
•Fuel Consumption
ARGONNE  CHICAGO
Life Sciences: Telemicroscopy
DATA ACQUISITION
PROCESSING,
ANALYSIS
9
ADVANCED
VISUALIZATION
NETWORK
IMAGING
INSTRUMENTS
COMPUTATIONAL
RESOURCES
LARGE DATABASES
[email protected]
ARGONNE  CHICAGO
10
And Thus: The Grid
“Resource sharing & coordinated
problem solving in dynamic, multiinstitutional virtual organizations”
[email protected]
ARGONNE  CHICAGO
11
Underlying Technical Requirements




Dynamic formation and management of
virtual organizations
Online negotiation of access to services:
who, what, why, when, how
Configuration of applications and systems
able to deliver multiple qualities of service
Autonomic management of distributed
infrastructures, services, and applications
[email protected]
ARGONNE  CHICAGO
12
The Grid World: Current Status


Dozens of major Grid projects in scientific &
technical computing/research & education
Open source Globus Toolkit™ a de facto
standard for major protocols & services


Simple protocols & APIs for authentication,
discovery, access, etc.: infrastructure

Information-centric design

Large user and developer base

Multiple commercial support providers

Enabler of numerous tools and applications
Global Grid Forum: community & standards
[email protected]
ARGONNE  CHICAGO
13
Overview


Origins: Resource sharing within scientific
collaborations

Science drivers & science Grid projects

Globus Toolkit
Evolution: Resource virtualization

Commercial drivers

OGSA: Grid meets Web services
[email protected]
ARGONNE  CHICAGO
Resource Sharing within “VOs” is
Not Unique to Science!

Fragmentation of enterprise infrastructure



14
Driven by cheap servers, fast nets,
ubiquitous Internet, eBusiness workloads
Need to configure distributed collections of
services to deliver specified QoS
Virtualization


Emerging service infrastructure, utility
computing models, economies of scale
Services dynamically instantiated across
device spectrum
 B2B, B2C,
[email protected]
C2C interactions
ARGONNE  CHICAGO
Virtualization and
Distributed Service Management
Less capable, integrated
Less connected
User service locus
Device Continuum
Distributed service
management
Dynamic, secure
service discovery
& composition
[email protected]
15
Larger, more integrated
More connected
Dynamically provisioned
Resource &
service
aggregation
Delivery of virtualized
services with QoS
guarantees
ARGONNE  CHICAGO
Realizing the Promise
Requires Significant Innovation




16
Automation of infrastructure operation to
achieve economies of scale
Management and component models for
service discovery, composition, provisioning
New applications and tools powered by
distributed services and resources
Business and service models to support
specialization of function
[email protected]
ARGONNE  CHICAGO
Grid Evolution:
Open Grid Services Architecture




17
Refactor Globus protocol suite to enable
common base and expose key capabilities
Service orientation to virtualize resources
and unify resources/services/information
Embrace key Web services technologies:
WSDL as IDL, leverage commercial efforts
Result: standard interfaces & behaviors for
distributed system management
[email protected]
ARGONNE  CHICAGO
18
OGSA System Structure

A standard substrate: the Grid service



The “Grid Service Specification”
… supports standard service specifications



Standard interfaces and behaviors that
address key distributed system issues
Resource management, databases,
workflow, security, diagnostics, etc., etc.
Target of current & planned GGF efforts
… and arbitrary application-specific
services based on these & other definitions
[email protected]
ARGONNE  CHICAGO
19
Transient Service Instances

“Web services” address discovery & invocation
of persistent services


In Grids, must also support transient service
instances, created/destroyed dynamically



Interface to persistent state of entire enterprise
Interfaces to the states of distributed activities
E.g. workflow, video conferencing, distributed
data analysis, workload management
Significant implications for how services are
named, discovered, managed, and used
[email protected]
ARGONNE  CHICAGO
20
OGSI, OGSA, and Web Services

OGSI (I = Infrastructure)

Small extensions to WSDL


Conventions for naming service instances


Handles and references
portTypes for common behavior


Nested serviceType & serviceDataDescription
Instance creation, lifetime management, introspection and
monitoring, registration, notification, …
OGSA (A = Architecture) built on OGSI

A collection of Grid service interfaces

Resource description & provisioning
Higher-level services: messaging services, logging, etc.
[email protected]
ARGONNE  CHICAGO

The Grid Service =
Interfaces/Behaviors + Service Data
Service data access
Explicit destruction
Soft-state lifetime
Binding properties:
- Reliable invocation
- Authentication
GridService
(required)
Service
data
element
… other interfaces …
(optional)
Service
data
element
21
Standard:
- Notification
- Authorization
- Service creation
- Service registry
- Manageability
- Concurrency
Service
data
element
Implementation
+ applicationspecific interfaces
Hosting environment/runtime
(“C”, J2EE, .NET, …)
[email protected]
ARGONNE  CHICAGO
22
Service Data

A Grid service instance maintains a set of
service data elements




Described in WSDL extension
XML element encapsulated in standard
container: name, type, lifetime, etc.
Includes basic introspection information,
interface-specific data, and application state
Pull and push models for information query

GridService::FindServiceData operation


Pull: queries this information via extensible query language
NotificationSource::SubscribeServiceData

Push: Subscribe to notification of changes to information
[email protected]
ARGONNE  CHICAGO
23
Notification Interfaces

NotificationSource for client subscription


Subscription expression describes which service
data element changes are of interest
Creates a subscription manager service



Manages the lifetime and properties of subscription
NotificationSink for asynchronous delivery
of notification messages
Simple, flexible base with wide variety of uses

Dynamic discovery/registry services,
monitoring, application error notification, etc.

Intermediaries: filter, aggregate, archive, et.c

Can integrate commercial messaging services
[email protected]
ARGONNE  CHICAGO
Grid Service Example:
Database Service


A DBaccess Grid service will support at
least two portTypes
Grid

GridService

DBaccess
Service
Each has service data



24
DBaccess
Name, lifetime, etc.
DB info
GridService: basic introspection
information, lifetime, …
DBaccess: database type, query languages
supported, current load, …, …
Maybe other portTypes as well

E.g., NotificationSource
[email protected]
ARGONNE  CHICAGO
25
Lifetime Management

GS instances created by factory or manually;
destroyed explicitly or via soft state



Negotiation of initial lifetime with a factory
(=service supporting Factory interface)
GridService interface supports

Destroy operation for explicit destruction

SetTerminationTime operation for keepalive
Soft state lifetime management avoids

Explicit client teardown of complex state

Resource “leaks” in hosting environments
[email protected]
ARGONNE  CHICAGO
26
Factory

Factory interface’s CreateService operation
creates a new Grid service instance




Reliable creation (once-and-only-once)
CreateService operation can be extended to
accept service-specific creation parameters
Returns a Grid Service Handle (GSH)

A globally unique URL, resolves to GSR

Uniquely identifies the instance for all time

Based on name of a handle resolver
Or Grid Service Reference (GSR)
[email protected]
ARGONNE  CHICAGO
Example:
Transient Database Services
“What services
can you create?”
“Create a database
service”
Grid
Service
“What database
services exist?”
28
DBaccess
Factory
Grid
Service
DBaccess
Instance name, etc.
Name, lifetime, etc.
Factory info
DB info
Grid
Service
Registry
Grid
Service
DBaccess
Instance name, etc.
Name, lifetime, etc.
Registry info
DB info
[email protected]
ARGONNE  CHICAGO
29
OGSA Design & Implementation

OGSI (I=Infrastructure) WG in GGF
defining core Grid service specification


Globus Toolkit => GT3 (alpha end 2002)

GT3 Core: Grid service specification

GT3 Base: Globus Toolkit behaviors


(At least) three implementation efforts
CIM resource model, GRAM-2 SLA
negotiation, database services, …
Other GGF WGs address OGSA security,
OGSA-compliant database services, etc.
[email protected]
ARGONNE  CHICAGO
30
Recap: Goals


Communicate the purpose, significance,
state, adoption, & future of Grid technology
Persuade you that Grid technology
represents a significant opportunity


Grids aren’t only (or particularly) about
science and servers—themes of virtualization,
service discovery, service management, and
QoS delivery are universal
Rapid uptake in industry & science represents
an exceptional opportunity for impact
[email protected]
ARGONNE  CHICAGO
31
For More Information

The Globus Project™


Context & research articles


www.mcs.anl.gov/~foster
Open Grid Services
Architecture


www.globus.org
www.globus.org/ogsa
Global Grid Forum



www.gridforum.org
Edinburgh, July 22-24
Chicago, Oct 15-17
[email protected]
ARGONNE  CHICAGO
32
OGSA Implementation
1) OGSA builds on infrastructure


Plumbing: WSDL, WS-Security,
WS-Routing/Referral, reliable
messaging, transactions, etc.
OGSI/OGSA Interfaces
service description,
service provisioning, …
Hosting environments
2) to enable virtualization via

Service description

Service provisioning
3) Standard container avoids
implementing OGSI features
in every service instance
[email protected]
Standard
OGSI
container
Web
services
various
Hosting Environment
Resource virtualization
and QoS support
ARGONNE  CHICAGO
33
Building an OGSI Container

Service data mgmt, query, subscription

Container should provide simple interface for
interacting with an instance’s implementation
to get and manage dynamic service data


Container should handle query processing


Service instance = CLR object
.NET support for XPath & Xquery allows for rich functionality
Container manages notification subscriptions,
and drives asynchronous notification messages

Soft-state lifetime management

Soft-state registration
[email protected]
ARGONNE  CHICAGO