Transcript DISCWorld – A Java-Based Metacomputing Environment

PAGIS: An Architecture for
Programming on the Grid
Andrew Wendelborn
Distributed & High Performance Computing Group
Department of Computer Science, University of Adelaide
Adelaide SA 5005, Australia
Email: [email protected]
December 2002
Background
• PAGIS
–
–
–
–
DHPC propject
Processing of satellite imagery
Started as a compositional process-oriented model for GIS applications
Now an architecture for building grid applications
• Outline
– Base-level API and GUI
– Reflective (metalevel) programming for grid applications
• Framework for defining grid behaviours
– Adaptive software: safe reconfiguration
– An adaptive resource broker
Principles
• High-level computational model
– Programmers shouldn’t need to be grid experts
– Reason about behaviour at an abstract level
• Simple, intuitive interface
– Following compositional component-based paradigm
• Access to system
– Programmer abstracts over low-level details … BUT
– Wants to tune aspects of system critical to grid application
– We provide this via a separate interface
Process Networks
• General, clearly defined semantics
• Separates meaning from implementation
– Different strategies produce equivalent results
• Compositional
• Reconfigurable
• Highly applicable to grid programming
Process2
Process1
Process3
Process4
Client
PAGIS overview
GUI
PN structure
Evaluator
Client
Scheduler
Evaluator
Client
Grid enabled
Globus toolkit
Distributed JVMs
Scheduler
GIIS
GRAM - GRIS
Evaluator
GRAM - GRIS
Tuning and Behaviour
• We have seen how to build base-level application
– How to fine-tune its implementation and behaviour
• Types of tuning
– Replacing code for PN process
– Allocation and re-allocation
– Monitoring, and other non-application (meta) concerns
• “Actors” observe and influence behaviour
– Human observation and intervention
– Other software systems
– Change how the computation occurs, not the result of the computation
A Reflective Approach
• Separate concerns into two programs
– Functional concerns: baselevel program
– Non-functional concerns: metalevel program (meta-behaviour)
– Reflects separation in process network model
• Reify certain aspects of baselevel
• Metalevel programmer reflects upon selected aspects
to control or customize aspects critical to application
• Software engineering for the grid
Method Reification
• Constructs an explicit representation of a method
invocation at the metalevel
• Provides hooks to control and customize method
invocation process
• We have designed a customization framework, Enigma
MetaA
ObjectA
MI
MetaB
ObjectB
Grid Behaviours
• Enigma gives us a model for describing and
implementing interesting grid behaviours
• Locality
– Migrating the computation and its parts
• Allocation and Adaptation
– Computation meta-attributes to drive allocator
– Periodically reassess allocation
• Performance Monitoring
– Intercept invocations to monitor progress
PAGIS as a Grid Programming Model
• API provides baselevel interface to system
• PN provides semantics and safety of adaptation
• Enigma: hooks to observe / influence behaviour
– Software engineering of adaptive applications
Process Network API
Enigma
Metalevel
Interface
PN System
Implementation
The System
Client
• Java implementation
• Globus toolkit
– Java CoG
Manager
Allocation
Globus
Plugin
Distribution Tools
Channel
Service
Broker
RSL
LDAP
query
MDS
Gatekeeper
Job Manager
Worker
JVM
COG
GRAM
Worker
Process
Service
Client
Plugin
fork
GRAM
data
• Coglets for peer-to-peer
Java / Globus
Current Implementation
• Building a PN
– Java; CoG; Globus toolkit
– Operations for simple image processing
• Inspection of meta-level attributes
• Changing meta-level attributes
– Influences allocation
– May lead to migration
• Underway: integration with DARC
PAGIS and DARC
Host A
Host B
DR
TCP
DR
Node
Node
DR
DR
Node
DR
DR
Host C
• DARC provides data abstractions
• PAGIS provides process abstractions
DARC provides data sources
Powerful abstraction over
Globus toolkit
Client
Scheduler
GIIS
DARC
Evaluator
GRAM - GRIS
Adaptive Resource Broker Project
• Testbed for different RB strategies
• Emulator (NISTNet) and simulators
• NWS (Network Weather Service)
– Performance database
– Forecasts
• Detection of anomalous conditions
– Generate threshold events
– RB adapts its behavior
Possible emulation framework
Grid Simulator
• Simple discrete event simulation package
– Builds on GridSim and SimGrid
– Java, multi-threaded
•
•
•
•
Can specify a full network topology
Heterogeneous resources of any kind modelled easily
Arbitrary information can be published about resources
Allows study of RB techniques on an arbitrary grid
– monitored by NWS
– threshold events feedback to RB
Next Steps
• Java peer-to-peer environment
– Coglets for peer-to-peer Java / Globus
•
•
•
•
•
•
DARC integration
Adaptive middleware for mobile computing
Experimental testbed for Enigma-based behaviours
Adaptive RB framework
XML intermediate form for PN structure
… applications …