Transcript SKR5800_LectureIntroduction_to_Globus

Introduction to Globus
PHENIX experiment uses Grid to
transfer 270 TB of data to Japan

During the polarized proton-proton run that ended in
June at the Relativistic Heavy Ion Collider (RHIC) at
Brookhaven, Grid tools were used by the PHENIX
experiment to send recently acquired data to a
regional computing centre for the experiment in
Japan. Brookhaven National Laboratory, on Long
Island, New York, is home to the RHIC/ATLAS
Computing Facility (RCF/ACF), which is the main
computing centre for experiments at RHIC and a
Tier-1 computing centre for ATLAS. The PHENIX
regional computing centre in Japan (CCJ) is at the
RIKEN research centre on its Wako campus close to
Tokyo.

Daily rates of data transferred from the PHENIX experiment to the CCJ
computing centre in Japan (blue), and the integrated data volume (red). Overall,
270 TB of data were transferred.
Introduction

The Globus Project provides middleware services for grid
environments.

There are four main components of Globus:
 The Grid Security Infrastructure (GSI) provides


Authentication and authorization services using public key
certificates and Kerberos authentication.
The Globus Resource Management architecture
provides:



a language for specifying application requirements
mechanisms for immediate and advance reservations of one or
more computational components.
interfaces for submitting jobs to remote machines.
Introduction (cont.)

The Globus Information Management architecture
provides:



a distributed scheme for publishing and retrieving information
about resources in the wide area environment.
A distributed collection of information servers is accessed by
higher-level services that perform resource discovery,
configuration and scheduling.
The Globus Data Management architecture provides
two fundamental components:


A universal data transfer protocol for grid computing
environments called GridFTP
Replica Management infrastructure for managing multiple
copies of shared data sets.
The Globus Approach

The Globus toolkit provides a range of
basic Grid services


These services are simple and orthogonal




Security, information, fault detection,
communication, resource management, ...
Can be used independently, mix and match
Programming model independent
For each there are well-defined APIs
Standards are used extensively

E.g., LDAP, GSS-API, X.509, ...
Grid Security Infrastructure

Single-sign on, run anywhere [if authorized]





Standards based (GSS, SSL, X.509)
GSS-API Interface
Identity/credential mapping at each resource
Limited delegation of rights
Integrated into wide variety of tools



Globus Resource Management
Secure shell, FTP,
Storage Resource Broker
Grid Security Infrastructure (cont.)

Based on public key technology


Standard X.509 certificate, same as certificates
used for the Web
Each user has:

a Grid user id (called a Subject Name)




/C=US/O=Globus/O=University of Southern
California/OU=Information Sciences Institute/CN=Ann
Chervenak
a private key (like a password)
a certificate signed by a Certificate Authority (CA)
A “gridmap” file at each site specifies
grid-id to local-id mapping
Authentication Model

Authentication is done on a “user” basis



No communication of plaintext passwords
Most sites will use conventional account
mechanisms


Single authentication step allows access to all grid
resources
You must have an account on a resource to use
that resource
Sites may use “generic” Grid accounts

Not common, but Globus can deal with it
Certificate Based Authentication

User has a certificate, signed by a trusted
“certificate authority” (CA)




User’s private key is used to encode a challenge
string
Public key is used to decode the challenge


Certificate contains user’s name and public key
Globus project operates a CA
If you can decode it, you know the user
Treat your private key carefully!!

Private key is stored in encrypted form
User Proxies


Minimize exposure of user’s private key
A temporary credential for use by our
computations




We call this a user proxy certificate
Allows process to act on behalf of user
User-signed user proxy certificate stored in local file
Proxy’s private key is not encrypted

Rely on file system security, proxy certificate file
must be readable only by the owner
Delegation



Remote creation of a user proxy
Allows remote process to act on behalf
of the user
Avoids sending passwords or private
keys across the network
User
Assignment of
credentials to
“user proxies”
User Proxy
Globus
Credential
GRAM
Process
GSI
Process
Kerberos
Single sign-on
via “grid-id”
Site 2
Site 1
Ticket
CREDENTIAL
Authenticated
interprocess
communication
Mutual
user-resource
authentication
Process
GRAM
Process
GSI
GSSAPI:
multiple
Public Key low-level
mechanisms
Certificate
Process
Process
Mapping
to local ids
Resource Management

Globus Resource Allocation Manager (GRAM)


Globus Arch. for Reservation and Allocation





Uniform interface to resource management
Co-allocation of compute resources
Immediate and advance reservation of network and
computers in prototype form
Fault detection service
Network measurement tools
Code management and distribution
infrastructure
Resource Management



Resource Specification Language (RSL) is
used to communicate requirements
The Globus Resource Allocation Manager
(GRAM) API allows programs to be started on
remote resources, despite local heterogeneity
A layered architecture allows applicationspecific resource brokers and co-allocators to
be defined in terms of GRAM services
Resource Management Architecture
RSL
specialization
Broker
RSL
Queries
& Info
Application
Ground RSL
Information
Service
Co-allocator
Simple ground RSL
Local
resource
managers
GRAM
GRAM
GRAM
LSF
EASY-LL
NQE
Resource Specification Language


Common notation for exchange of
information between components
RSL provides two types of information:



Resource requirements: Machine type,
number of nodes, memory, etc.
Job configuration: Directory, executable,
args, environment
API provided for manipulating RSL
RSL Syntax

Elementary form: parenthesis clauses


Operators Supported:


<, <=, =, >=, > , !=
Some supported attributes:


(attribute op value [ value … ] )
executable, arguments, environment, stdin,
stdout, stderr, resourceManagerContact,
resourceManagerName
Unknown attributes are passed through

May be handled by subsequent tools
Constraints: “&”


For example:
& (count>=5) (count<=10)
(max_time=240) (memory>=64)
(executable=myprog)
“Create 5-10 instances of myprog, each on
a machine with at least 64 MB memory that
is available to me for 4 hours”
Multirequest: “+”

A multirequest allows us to specify multiple
resource needs, for example
+ (& (count=5)(memory>=64)
(executable=p1))
(&(network=atm) (executable=p2))



Execute 5 instances of p1 on a machine with at
least 64M of memory
Execute p2 on a machine with an ATM connection
Multirequests are central to co-allocation
Co-allocation

Simultaneous allocation of a resource
set


Handled via optimistic co-allocation based
on free nodes or queue prediction
In the future, advance reservations will
also be supported
A Co-allocation Multirequest
+( & (resourceManagerContact=
“flash.isi.edu:754:/C=US/…/CN=flash.isi.edu-fork”)
(count=1)
(label="subjob A")
Different resource
(executable= my_app1)
managers
)
Different ( & (resourceManagerContact=
counts
“sp139.sdsc.edu:8711:/C=US/…/CN=sp097.sdsc.edu-lsf")
(count=2)
(label="subjob B")
Different executables
(executable=my_app2)
)
Job Submission Interfaces

Globus Toolkit includes several command line
programs for job submission




globus-job-run: Interactive jobs
globus-job-submit: Batch/offline jobs
globusrun: Flexible scripting infrastructure
Others are building better interfaces

General purpose


Condor-G, PBS, GRD, Hotpage, etc
Application specific

ECCE’, Cactus, Web portals
Grid Information Services

Publish and retrieve information about
system elements


Used for discovery, configuration,
scheduling
Distributed collection of information
servers and index nodes

LDAP V3 as wire protocol and API
Examples of Useful Information

Characteristics of a compute resource


Characteristics of a network


IP address, software available, system
administrator, networks connected to, OS version,
load
Bandwidth and latency, protocols, logical topology
Characteristics of the Globus infrastructure

Hosts, resource managers
Grid Information Service



Provide access to static and dynamic
information regarding system components
A basis for configuration and adaptation in
heterogeneous, dynamic environments
Requirements and characteristics




Uniform, flexible access to information
Scalable, efficient access to dynamic data
Access to multiple information sources
Decentralized maintenance
The Globus Toolkit
Metacomputing Directory Service

Store information in a distributed directory


Directory can be updated by




Directory stored in collection of LDAP servers
Information providers and tools
Applications (i.e., users)
Backend tools which generate info on demand
Information dynamically available to


Tools
Applications
Directory Service Functions

White Pages


Yellow Pages


Look up the IP number, amount of memory, etc.,
associated with a particular machine
Find all the computers of a particular class or with a
particular property
Temporary inconsistencies are often considered
okay



In a distributed system, you often do not know the
state of a resource until you actually use it
Information is often used as “hints”
Information itself can contain ttl, etc.
MDS Approach
Application

Based on LDAP




Globus specific schema


Lightweight Directory Access
Protocol v3 (LDAPv3)
Standard data model
Standard query protocol

LDAP API
GRIS
…
GIIS
Host-centric representation
Globus specific tools

Middleware
GRIS, GIIS
Data discovery, publication,…
SNMP
…
NWS
NIS
LDAP
Grid Resource Information Service

Server which runs on each resource


Given the resource DNS name, you can find the GRIS
server (well known port = 2135)
Provides resource specific information

Much of this information may be dynamic



“White pages” lookup of resource information


Load, process information, storage information, etc.
GRIS gathers this information on demand
Ex: How much memory does machine have?
“Yellow pages” lookup of resource options

Ex: Which queues on machine allow large jobs?
Grid Index Information Service

GIIS describes a class of servers


Gathers information from multiple GRIS servers
Each GIIS is optimized for particular queries




Ex1: Which Alliance machines are >16 process SGIs?
Ex2: Which Alliance storage servers have >100Mbps
bandwidth to host X?
Akin to web search engines
Organization GIIS


The Globus Toolkit ships with one GIIS
Caches GRIS info with long update frequency

Useful for queries across an organization that rely on
relatively static information
Referral Service


Links together multiple GRIS and/or
GIIS servers into a single LDAP
namespace
Referral servers contain no actual
content
Data Grid Services

Access to remote data



Transport services



Uniform access to diverse, remote storage
management systems
Cache management
Standards based (GSI, FTP protocol)
Client API, Extensible server, support for third party
transfer
Replica Management
Data Intensive Issues Include
…

High-speed, reliable access to remote data

Automated discovery of “best” copy of data

Manage replication to improve performance

Co-schedule compute, storage, network

Enforce access control on data
The Globus Data Grid
Two major components:
1. Data Transport and Access

Common protocol


Secure, efficient, flexible, extensible data movement
Family of tools supporting this protocol
2. Replica Management Architecture

Simple scheme for managing:

multiple copies of files

collections of files
Motivation for a Common
Data Access Protocol

Existing distributed data storage systems




DPSS, HPSS: focus on high-performance access, utilize
parallel data transfer, striping
DFS: focus on high-volume usage, dataset replication,
local caching
SRB: connects heterogeneous data collections, uniform
client interface, metadata queries
Problems

Incompatible protocols



Each require custom client
Partitions available data sets and storage devices
Each protocol has subset of desired functionality
A Common, Secure, Efficient
Data Access Protocol



Common, extensible transfer protocol
Decouple low-level data transfer mechanisms
from the storage service
Advantages:



New, specialized storage systems are automatically
compatible with existing systems
Existing systems have richer data transfer functionality
Interface to many storage systems


HPSS, DPSS, file systems
Plan for SRB integration
Replica Management


Maintain a mapping between logical names for
files and collections and one or more physical
locations
Important for many applications

Example: CERN HLT data





Multiple petabytes of data per year
Copy of everything at CERN (Tier 0)
Subsets at national centers (Tier 1)
Smaller regional centers (Tier 2)
Individual researchers will have copies
Globus Approach to Replica
Management

Identify replica cataloging and reliable
replication as two fundamental services




Layer on other Grid services: GSI, transport,
information service
Use LDAP as catalog format and protocol, for
consistency
Use as a building block for other tools
Advantage

These services can be used in a wide variety of
situations
Replica Manager Components

Replica catalog definition


Low-level replica catalog API



LDAP object classes for representing logical-tophysical mappings in an LDAP catalog
globus_replica_catalog library
Manipulates replica catalog: add, delete, etc.
High-level reliable replication API


globus_replica_manager library
Combines calls to file transfer operations and calls to
low-level API functions: create, destroy, etc.
Replica Catalog Structure:
A Climate Modeling Example
Replica Catalog
Logical Collection
Logical Collection
C02 measurements 1998
C02 measurements 1999
Filename: Jan 1998
Filename: Feb 1998
…
Location
Location
jupiter.isi.edu
sprite.llnl.gov
Filename: Mar 1998
Filename: Jun 1998
Filename: Oct 1998
Protocol: gsiftp
UrlConstructor:
gsiftp://jupiter.isi.edu/
nfs/v6/climate
Filename: Jan 1998
…
Filename: Dec 1998
Protocol: ftp
UrlConstructor:
ftp://sprite.llnl.gov/
pub/pcmdi
Logical
File Parent
Logical File
Logical File
Jan 1998
Feb 1998
Size: 1468762
Replica Catalog Services
as Building Blocks: Examples

Combine with information service to build
replica selection services



E.g. “find best replica” using performance info
from NWS and MDS
Use of LDAP as common protocol for info and
replica services makes this easier
Combine with application managers to build
data distribution services

E.g., build new replicas in response to frequent
accesses
Relationship to Metadata Catalogs

Metadata services describe data contents


Must support a variety of metadata catalogs



Have defined a simple set of object classes
MCAT being one important example
Others include LDAP catalogs, HDF
Community metadata catalogs


Agree on set of attributes
Produce names needed by replica catalog:
 Logical collection name
 Logical file name
A Model Architecture for Data Grids
Metadata
Catalog
Attribute
Specification
Logical Collection and
Logical File Name
Application
Selected
Replica
gsiftp commands
Replica
Catalog
Multiple Locations
Replica
Selection
Tape Library
Replica Location 1
Disk Cache
Replica Location 2
MDS
Performance
Information and
Predictions
Disk Cache
Disk Array
NWS
Replica Location 3
Fault Detection:
Globus Heartbeat Monitor

Detect and report “failure” of component of a
computation



Limited by ability to distinguish between network
partition and system failure
Optionally used within Globus Toolkit to
monitor status of system processes
Can also be used to construct special fault
monitors for applications

Example: Netsolve
Fault Detection (cont.)


Goal: make low-level operations reliable
No libraries for checkpoint and restart



Can’t “checkpoint” a socket
Only application knows how to checkpoint
and restart
Likewise, storage system must do logging
Heartbeat Monitor
Application Level
Fault Handler
!
Application
Data
Collector
HBM
Data
Collector
System Monitoring Tools
Process and Host
Heartbeat
Process and Host
Heartbeat
Host 1
Process Status
Inquiry
HBM
Monitor
Register/
Unregister
Host 2
Process Status
Inquiry
Monitored
Process
Monitored
Process
Monitored
Process
HBM
Monitor
Register/
Unregister
Monitored
Process
Monitored
Process
Monitored
Process