Transcript Overview of Grid Computing - TERENA Networking Conference 2002

Future Directions
in Grid Computing
Ian Foster
Mathematics and Computer Science Division
Argonne National Laboratory
and
Department of Computer Science
The University of Chicago
http://www.mcs.anl.gov/~foster
Invited Talk, Terena 2002 Conference, Limerick, June 5, 2002
2
The Grid Vision
“Resource sharing & coordinated problem
solving in dynamic, multi-institutional
virtual organizations”
– On-demand, ubiquitous access to
computing, data, and services
– New capabilities constructed dynamically
and transparently from distributed services
“When the network is as fast as the computer's
internal links, the machine disintegrates across
the net into a set of special purpose appliances”
(George Gilder)
[email protected]
ARGONNE  CHICAGO
Why the Grid?
(1) Evolution of the Scientific Process

3
Pre-electronic
– Theorize &/or experiment, alone or in small
teams; publish paper

Post-electronic
– Construct and mine very large databases of
observational or simulation data
– Develop computer simulations & analyses
– Access specialized devices remotely
– Exchange information quasi-instantaneously
within distributed multidisciplinary teams
 Need to manage dynamic, distributed
infrastructures, services, and applications
[email protected]
ARGONNE  CHICAGO
eScience Application:
Sloan Digital Sky Survey Analysis
[email protected]
4
ARGONNE  CHICAGO
eScience Application:
Sloan Digital Sky Survey Analysis
5
Size distribution of
galaxy clusters?
Galaxy cluster
size distribution
100000
Chimera Virtual Data System
+ iVDGL Data Grid (many CPUs)
10000
1000
100
10
[email protected]
1
1
10
Number of Galaxies
100
ARGONNE  CHICAGO
Challenging Technical
Requirements





6
Dynamic formation and management of
virtual organizations
Online negotiation of access to services:
who, what, why, when, how
Configuration of applications and systems
able to deliver multiple qualities of service
Autonomic management of distributed
infrastructures, services, and applications
Management of distributed state as a
fundamental issue
[email protected]
ARGONNE  CHICAGO
State of the Art:
Globus ToolkitTM (since 1996)

7
Small, standards-based set of protocols for
distributed system management
– Authentication, delegation; resource
discovery; reliable invocation; etc.

Information-centric design
– Data models; publication, discovery protocols

Open source implementation
– Large international user community

Successful enabler of higher-level services
and applications
[email protected]
ARGONNE  CHICAGO
8
Grid Projects in eScience
[email protected]
ARGONNE  CHICAGO
9
A Large Virtual Organization: CERN’s
Large Hadron Collider
1800 Physicists, 150 Institutes, 32 Countries
100 PB of data by 2010; 50,000 CPUs?
[email protected]
ARGONNE  CHICAGO
Data Grids for High Energy Physics
~PBytes/sec
Online System
~100 MBytes/sec
~20 TIPS
There are 100 “triggers” per second
Each triggered event is ~1 MByte in size
~622 Mbits/sec
or Air Freight (deprecated)
France Regional
Centre
1 TIPS is approximately 25,000
SpecInt95 equivalents
Offline Processor Farm
There is a “bunch crossing” every 25 nsecs.
Tier 1
10
Tier 0
Germany Regional
Centre
Italy Regional
Centre
~100 MBytes/sec
CERN Computer Centre
FermiLab ~4 TIPS
~622 Mbits/sec
Tier 2
~622 Mbits/sec
Institute
Institute Institute
~0.25TIPS
Physics data cache
Institute
Caltech
~1 TIPS
Tier2 Centre
Tier2 Centre
Tier2 Centre
Tier2 Centre
~1 TIPS ~1 TIPS ~1 TIPS ~1 TIPS
Physicists work on analysis “channels”.
Each institute will have ~10 physicists working on one or more
channels; data for these channels should be cached by the
institute server
~1 MBytes/sec
Tier 4
Physicist workstations
[email protected]
ARGONNE  CHICAGO
Grids at NASA: Aviation Safety
11
Wing Models
•Lift Capabilities
•Drag Capabilities
•Responsiveness
Airframe Models
Stabilizer Models
•Deflection capabilities
•Responsiveness
Crew Capabilities
- accuracy
- perception
- stamina
- re-action times
- SOPs
Engine Models
Human Models
•Braking performance
•Steering capabilities
•Traction
•Dampening capabilities
Landing Gear Models
[email protected]
•Thrust performance
•Reverse Thrust performance
•Responsiveness
•Fuel Consumption
ARGONNE  CHICAGO
12
Life Sciences: Telemicroscopy
DATA ACQUISITION
PROCESSING,
ANALYSIS
ADVANCED
VISUALIZATION
NETWORK
IMAGING
INSTRUMENTS
COMPUTATIONAL
RESOURCES
LARGE DATABASES
[email protected]
ARGONNE  CHICAGO
Why the Grid:
(2) And What About Business?

13
Fragmentation of enterprise infrastructure
– Specialized platforms -> commodity servers
– “Intelligence” embedded in networks

The rise of the “eUtility” (IBM, HP, …)
– Outsourcing, economies of scale

Business-to-business computing
– Especially complex virtual organizations

Ever more challenging QoS requirements
– “Green-screen” -> “ubiquitious web presence”
[email protected]
ARGONNE  CHICAGO
Today’s Enterprise
Computing Environment
[email protected]
14
ARGONNE  CHICAGO
15
Grids and Industry: Early Examples
Entropia: Distributed computing
(BMS, Novartis, …)
Butterfly.net: Grid for
multi-player games
[email protected]
ARGONNE  CHICAGO
16
The Business Opportunity

On-demand computing, storage, services
– Significant savings due to reduced build-out,
economies of scale, reduced admin costs
– Greater flexibility => greater productivity

Entirely new applications and services
– Based on high-speed resource integration

Solution to enterprise computing crisis
– Render distributed infrastructures manageable
[email protected]
ARGONNE  CHICAGO
Grid Computing
[email protected]
17
ARGONNE  CHICAGO
Realizing the Promise
Requires Significant Innovation




18
Automation of infrastructure operation to
achieve economies of scale
Management and component models for
distributed service provisioning
New applications and tools powered by
distributed services and resources
Business and service models to support
specialization of function
[email protected]
ARGONNE  CHICAGO
Grid Evolution:
Open Grid Services Architecture




19
Refactor Globus protocol suite to enable
common base and expose key capabilities
Service orientation to virtualize resources
and unify resources/services/information
Embrace key Web services technologies for
standard IDL, leverage commercial efforts
Result: standard interfaces & behaviors for
distributed system management: the Grid
service
[email protected]
ARGONNE  CHICAGO
Open Grid Services Architecture:
Transient Service Instances

20
“Web services” address discovery & invocation
of persistent services
– Interface to persistent state of entire enterprise

In Grids, must also support transient service
instances, created/destroyed dynamically
– Interfaces to the states of distributed activities
– E.g. workflow, video conf., dist. data analysis

Significant implications for how services are
managed, named, discovered, and used
– In fact, much of OGSA (and Grid) is concerned
with the management of service instances
[email protected]
ARGONNE  CHICAGO
21
Open Grid Services Architecture

Defines fundamental (WSDL) interfaces and
behaviors that define a Grid Service
– Required + optional interfaces = WS “profile”
– A unifying framework for interoperability &
establishment of total system properties

Defines WSDL extensibility elements
– E.g., serviceType (a group of portTypes)

Delivery via open source Globus Toolkit 3.0
– Leverage GT experience, code, community

And commercial implementations
[email protected]
ARGONNE  CHICAGO
The Grid Service =
Interfaces/Behaviors + Service Data
Service data access
Explicit destruction
Soft-state lifetime
Binding properties:
- Reliable invocation
- Authentication
GridService
(required)
Service
data
element
… other interfaces …
(optional)
Service
data
element
22
Standard:
- Notification
- Authorization
- Service creation
- Service registry
- Manageability
- Concurrency
Service
data
element
Implementation
+ applicationspecific interfaces
Hosting environment/runtime
(“C”, J2EE, .NET, …)
[email protected]
ARGONNE  CHICAGO
23
Service Data

A Grid service instance maintains a set of
service data elements
– XML fragments encapsulated in standard
<name, type, TTL-info> containers
– Includes basic introspection information,
interface-specific data, and application data

FindServiceData operation (GridService
interface) queries this information
– Extensible query language support

See also notification interfaces
– Allows notification of service existence and
changes in service data
[email protected]
ARGONNE  CHICAGO
Grid Service Example:
Database Service

A DBaccess Grid service will support at
least two portTypes
Grid
– GridService
Service
– DBaccess

24
Each has service data
DBaccess
Name, lifetime, etc.
DB info
– GridService: basic introspection
information, lifetime, …
– DBaccess: database type, query languages
supported, current load, …, …

Maybe other portTypes as well
– E.g., NotificationSource (SDE = subscribers)
[email protected]
ARGONNE  CHICAGO
25
Example:
Data Mining for Bioinformatics
Community
Registry
Mining
Factory
Database
Service
BioDB 1
User
Application
“I want to create
a personal database
containing data on
e.coli metabolism”
Compute Service Provider
.
.
.
.
.
.
Database
Service
Database
Factory
BioDB n
Storage Service Provider
[email protected]
ARGONNE  CHICAGO
26
Example:
Data Mining for Bioinformatics
“Find me a data Community
mining service, and Registry
somewhere to store
data”
Mining
Factory
Database
Service
BioDB 1
User
Application
Compute Service Provider
.
.
.
.
.
.
Database
Service
Database
Factory
BioDB n
Storage Service Provider
[email protected]
ARGONNE  CHICAGO
27
Example:
Data Mining for Bioinformatics
GSHs for Mining
and Database
factories
User
Application
Community
Registry
Mining
Factory
Database
Service
BioDB 1
Compute Service Provider
.
.
.
.
.
.
Database
Service
Database
Factory
BioDB n
Storage Service Provider
[email protected]
ARGONNE  CHICAGO
28
Example:
Data Mining for Bioinformatics
Community
Registry
“Create a data mining
service with initial
lifetime 10”
User
Application
“Create a
database with initial
lifetime 1000”
Mining
Factory
Database
Service
BioDB 1
Compute Service Provider
.
.
.
.
.
.
Database
Service
Database
Factory
BioDB n
Storage Service Provider
[email protected]
ARGONNE  CHICAGO
29
Example:
Data Mining for Bioinformatics
Community
Registry
“Create a data mining
service with initial
lifetime 10”
User
Application
“Create a
database with initial
lifetime 1000”
Mining
Factory
Database
Service
Miner
BioDB 1
Compute Service Provider
.
.
.
.
.
.
Database
Service
Database
Factory
BioDB n
Database
Storage Service Provider
[email protected]
ARGONNE  CHICAGO
30
Example:
Data Mining for Bioinformatics
Community
Registry
Mining
Factory
Query
Miner
User
Application
Database
Service
BioDB 1
Compute Service Provider
.
.
Query
.
.
.
.
Database
Service
Database
Factory
BioDB n
Database
Storage Service Provider
[email protected]
ARGONNE  CHICAGO
31
Example:
Data Mining for Bioinformatics
Community
Registry
Keepalive
Mining
Factory
Query
Miner
BioDB 1
Compute Service Provider
.
.
Query
.
User
Application
Keepalive
Database
Service
.
.
.
Database
Service
Database
Factory
BioDB n
Database
Storage Service Provider
[email protected]
ARGONNE  CHICAGO
32
Example:
Data Mining for Bioinformatics
Community
Registry
Keepalive
User
Application
Keepalive
Mining
Factory
Database
Service
Miner
BioDB 1
Compute Service Provider
.
.
.
.
.
.
Results
Database
Service
Database
Factory
Results
BioDB n
Database
Storage Service Provider
[email protected]
ARGONNE  CHICAGO
33
Example:
Data Mining for Bioinformatics
Community
Registry
User
Application
Keepalive
Mining
Factory
Database
Service
Miner
BioDB 1
Compute Service Provider
.
.
.
.
.
.
Database
Service
Database
Factory
BioDB n
Database
Storage Service Provider
[email protected]
ARGONNE  CHICAGO
34
Example:
Data Mining for Bioinformatics
Community
Registry
Mining
Factory
Database
Service
BioDB 1
Compute Service Provider
.
.
.
User
Application
Keepalive
.
.
.
Database
Service
Database
Factory
BioDB n
Database
Storage Service Provider
[email protected]
ARGONNE  CHICAGO
GT3: An Open Source OGSACompliant Globus Toolkit

35
GT3 Core
– Implements Grid service
interfaces & behaviors
– Reference impln of
evolving standard
– Multiple hosting envs:
Java/J2EE, C, C#/.NET?

GT3 Base Services
GT3
Data
Services
Other Grid
Services
GT3 Base Services
GT3 Core
– Evolution of current
Globus Toolkit capabilities

Many other Grid services
[email protected]
ARGONNE  CHICAGO
Summary:
Grids and Globus Toolkit




36
The Grid: Resource sharing & coordinated
problem solving in dynamic, multiinstitutional virtual organizations
Considerable impact within eScience,
growing interest & adoption within eBusiness
Globus Toolkit an open source, defacto
standard source of protocol and API
definitions—and reference implementations
A strong community organization: the Global
Grid Forum
[email protected]
ARGONNE  CHICAGO
Summary:
Open Grid Services Architecture





37
Open Grid Services Architecture represents
(we hope!) next step in Grid evolution
Service orientation enables unified treatment
of resources, data, and services
Standard interfaces and behaviors (the Grid
service) for managing distributed state
Deeply integrated information model for
representing and disseminating service data
Open source Globus Toolkit implementation
(and commercial value adds)
[email protected]
ARGONNE  CHICAGO
For More Information

38
Grid Book (somewhat old)
– www.mkp.com/grids

Survey + research articles
– www.mcs.anl.gov/~foster

The Globus Project™
– www.globus.org

GriPhyN project
– www.griphyn.org

Global Grid Forum
– www.gridforum.org
– www.gridforum.org/ogsi-wg
– Edinburgh, July 22-24
– Chicago, Oct 15-17
[email protected]
ARGONNE  CHICAGO