Transcript Lecture_13a_Using_the_GAT

Using the GAT
(GAT the semester project?) :-P
Hartmut Kaiser
[email protected]
http://www.cct.lsu.edu/~gallen/Teaching
Semester Project
• Problem
– You have an application which has to be run 1000
times with different input data
– You have the 1000 different input files
– You want to get back 1000 result files
• You want to distribute this to all available
machines
• The data transfer should be as effective as
possible
October, 17th 2005
Using the GAT
2
Semester Project - Roadmap
•
•
•
•
•
•
Requirements analysis
Problem decomposition
Interface analysis
System architecture design
Tool selection
Goal setting
• We will do that for the whole system today
• You‘ll be expected to do that for your subproject in
the coursework
October, 17th 2005
Using the GAT
3
Requirements analysis
• Overall application needs to be able to
–
–
–
–
–
–
–
–
–
Discover and select resources and services
Copy files reliable using existing services
Submit jobs to selected resources
Decide about parameters for file transfer
Decide about how to distribute jobs to available
resources
Figure out when to bring back result files
Correct things when something fails
Keep track what‘s going on
Be independent from the Grid architecture
October, 17th 2005
Using the GAT
4
Problem decomposition
• Service/resource discovery
– Using information repositories -> MDS, IGRID, GPIR, local
configuration files?
• Service/resource selection
– Using which criteria?
– Match with application requirements
• File transfer
– Protocol/service selection, parameter selection, optimisation
• Job submission
– Selection of method to use
• Information gathering
– Notifications, polling?
October, 17th 2005
Using the GAT
5
Interface analysis
• Service/resource description
– What information available/required?
• Job description
– What needs to be specified?
• Notification description (do we need them at
all?)
– What notifications need to be defined?
October, 17th 2005
Using the GAT
6
Tool selection
• Libraries to use, depends on
– Availability of libraries
– Availability of services
• Programming language
– C, C++ or Python
• Target operating system(s)
– Portability
• Compilation and debug environments
(compiler, debugger, tracing/logging libraries)
October, 17th 2005
Using the GAT
7
Libraries/Services to use
• Libraries
– Database module
– Web-service access
– Logging of application activity
• Services
– GT4 RFT
– GPIR information service
– Globus GRAM
October, 17th 2005
Using the GAT
8
Programming Language
•
Since you want to use the GAT you have 3 alternatives
C
C++
(+) Easy to learn and use
(+) Lots of GAT examples
(+) Complete documentation of C GAT API available
(-) Low level, Error prone
(-) Large code overhead for adaptor administration
(-) Bad support for webservice access
(+) Very easy adaptor writing
(+) Lots of GAT examples
(+) Good library support (data structures as list, map etc.)
(+/-) More complex to learn and use
(-) Incomplete documentation of the C++ GAT API
(-) Bad support for webservice access
Python
(+) Easy to learn and use
(+) Very easy adaptor writing
(+) Very good support for webservice access
(-) Less GAT examples
(-) Incomplete documentation of the Python GAT API
October, 17th 2005
Using the GAT
9
Target operating system(s)
• Linux (*nix)
• Mac OS (Darwin)
• Windows (unfortunately currently not fully
supported)
October, 17th 2005
Using the GAT
10
Goal setting
• Create a Task farming manager
– Usable for a class of applications:
• Single executable, non-MPI
• Reading all input from one or more files
• Producing one or more files in the output
– Distribute tasks evenly on available resources
– Stage in/out input and output files as effectively
as possible (by selecting the appropriate transfer
protcoll and parameters)
– Keep track on the current status of all tasks, so
the TFM may be restarted (no need to run it all
the time)
October, 17th 2005
Using the GAT
11
System architecture design
Task farming manager (Python/C++/C)
Supporting libraries
GAT
Python/C++/C adaptor
(file transfer)
Python/C++/C adaptor
(resource and service
discovery)
C adaptor
(job submission)
GT4 RFT Service
GPIR Service
GRAM Service
October, 17th 2005
Using the GAT
12
Example: Prequisites (Python)
• Python V2.4.3
– http://www.python.org/
• GAT (V1.7.1), GAT Python wrapper (V1.5.1)
– http://tinyurl.com/4l288
• SOAPpy (V0.11.6) – webserice bindings
– http://tinyurl.com/8mctn
• GPIR and GT4 RFT running somewhere
– http://tinyurl.com/bbrkp (GPIR)
– http://tinyurl.com/9udvb (GT4 RFT)
October, 17th 2005
Using the GAT
13
Submodules, project groups
• Group 1: Task farming manager
– Application itself
– Strategies of job distribution
-> Cornelius, Krishnendu, Chirag
• Group 2: Resource discovery
– Adaptor to connect to GPIR
– Defining metadata for service/resource information
-> Srikanth, Ibrahim, Rakesh
• Group 3: File transfer
– Adaptor to connect to GT4 RFT
– Optimisation of transfer parameters
-> Memet, Santiago, Teresa
October, 17th 2005
Using the GAT
14
Semester Project
-> Write a Grid based task farming manager using the GAT
– Discover available resources to run jobs on
• Grid Information Service (GPIR)
– Discover available file transfer and job submission services
• Grid Information Service (GPIR)
– Stage input files, get back result files
• File Transfer Services (GT4 RFT)
– Submit jobs
• Job Submission Service (GRAM)
-> Write a GAT adaptor to connect to GPIR
-> Write a GAT adaptor to use GT4 RFT
-> Create your own webservice to receive notifications about job
status?
October, 17th 2005
Using the GAT
15
Coursework
Every of the project groups is expected to write a
paper about their submodule containing:
• Elevator pitch
• Defining requirements, setting goals etc.
• Architecture diagram and description of the
subprojects to deliver, containing submodules, their
interfaces etc.
(Due Wednesday October 26th)
October, 17th 2005
Using the GAT
16
The Elevator Pitch
Write a short statement describing what your
software will do, and why
– Short is better than long
– Functionality trumps technology (what and not
how)
– Solve a problem (explain the initial problem)
– Pitch benefits not yourself
– Figure out, what‘s important to your audience,
address excatly that (cost, innovation, features,
compatibility, or something else entirely).
October, 17th 2005
Using the GAT
17
Defining requirements
• Business requirements
– High level objectives of the organisation or
customer who requests the system
• User requirements
– User goals or tasks that the users must be able to
perform
• Functional requirements
– Software functionality that must be built into the
product to accomplish their tasks
• System requirements
– Top-level requirements for the product
October, 17th 2005
Using the GAT
18