Transcript jsaga
Grid interoperability
developments at CC-IN2P3
Lyon, 7 September 2010
Sylvain Reynaud
Plan
JSAGA
– the SAGA specification
– the JSAGA implementation
– example applications
other grid interoperability developments at CC-IN2P3
– JJS
– JUX
– JPySAGA
perspectives
JSAGA
2
Simple API for Grid Applications
SAGA is a specification
– standard
• recommendation of
– object oriented
– simple
• 80/20 rule: 20% of the features
cover 80% of the needs
– integrated ("consistent")
• common "look & feel" for all the
functional packages (see slide)
– high-level
• hide low-level details about
middlewares
– middleware-independent
• use the same API to access all
middlewares
– language-independent
• Java
• C++
• python (wrappers)
– stable
• initial recommendation SAGA
1.0 (15 January 2008)
• a 1st update soon (backward
compatible)
– extensible
• Service Discovery, Message…
JSAGA
3
Simple API for Grid Applications
JSAGA
Source: S Jha, T Kielmann, A Merzky 4
The JSAGA implementation
JSAGA is an implementation of SAGA
– written in/for Java
focuses on
–
–
–
–
–
uniform usage of middleware….
ease of extension……………….
efficiency and scalability……….
control of the API behavior........
operating-system independency
(beyond what they natively support)
(thanks to the design of adaptor interfaces)
(configuration rather than attempts)
(tested on
)
Under LGPL license
JSAGA
5
Uniform usage of existing grids
JSAGA
6
Example – uniform usage
SAGA
job
user applications
JSAGA
Globus
Gatekeep.
gsiftp
WMS
VOMS
GridFTP
JSAGA
7
Example – what happens behind
SAGA
job
user applications
wrapper
script
Globus
Gatekeep.
gsiftp
job
WMS
modified
JSAGA
VOMS
EGEE
proxy + attributes
translate
submit job
poll status
delegate transfer
JDL
RSL
DGrid
proxy
adapt + translate
submit job
receive status
transfer
– to any protocol
VOMS WMS GridFTP
LCG-CE
JSAGA
CREAM-CE
Globus GK
Unicore
8
interface
Layered software architecture
implementation
JSAGA
JSAGA
9
interface
Layered software architecture
user applications
SAGA
Java binding
JSAGA
core engine
JSAGA
JSAGA
adaptors interfaces
JSAGA
Used by end users
– web portals, GUI, CLI…
Used by application developers
– a single uniform interface
– object oriented, high-level
Implemented by adaptors
– each "way" to implement a
given feature has 1 interface
– service oriented, low-level
adaptors
middleware APIs
implementation
Implemented by middleware
– each technology has its own
interfaces
10
Layered software architecture
user applications
Used by end users
– web portals, GUI, CLI…
Used by application developers
– a single uniform interface
– object oriented, high-level
JSAGA
core engine
JSAGA
adaptor interfaces
adaptor
Globus GK
upload
data
get
output
get
status
Implemented by adaptors
– each "way" to implement a
given feature has 1 interface
– service oriented, low-level
adaptor
GridFTP
JSAGA
Implemented by middleware
– each technology has its own
interfaces
11
Layered software architecture
user applications
Used by end users
– web portals, GUI, CLI…
Used by application developers
– a single uniform interface
– object oriented, high-level
JSAGA
core engine
JSAGA
adaptor interfaces
adaptor
Globus GK
upload
data
get
output
get
status
Implemented by adaptors
– each "way" to implement a
given feature has 1 interface
– service oriented, low-level
adaptor
GridFTP
JSAGA
Implemented by middleware
– each technology has its own
interfaces
12
Layered software architecture
user applications
JSAGA
core engine
JSAGA
adaptor interfaces
adaptor
Globus GK
upload
data
Middleware API can be used
more efficiently
– engine selects best interface for
each user request
adaptor
GridFTP
JSAGA
get
output
get
status
Developing new adaptors is
easier
– just wrap functionalities
supported by middleware API
13
Supported technologies
local
ssh
naregi
cream
unicore6
wsgram
zip
file
ftp
sftp
https
http
irods
srb
cache
Physical
srm
gsiftp
rbyteio
catalog
irods
srb
lfn
X509
Login / pwd
SSH
JKS
VOMS
MyProxy
G. RFC820
G. Legacy
Globus
done
construction
JSAGA
planned
InMemCred
Security
Logical
rns
Data
gLite-WMS
gatekeeper
Execution
Minimal dependencies on
external libraries
external tools
– e.g. no gLite-UI
operating system
– tested on
14
Selection of security context
Automatic selection is attractive to beginners, but it is…
– inefficient
• waste of time while trying with wrong contexts
• jobs may fail at the end because they didn't run with expected VO/role
– unreliable
• may create files that will not be accessible to partners
• may lock an account (e.g. by trying with 3 wrong passwords)
– insecure
• sending all our passwords to unexpected services is not safe
JSAGA's solution : configure selection of security context
– bind job/data services to context types (+ URL patterns)
– dynamic, per session, robust (all conflicts are detected at config-time)
JSAGA
15
Example applications
Pandora Gateway
– the grid platform of the
company
• a leading Grid/Cloud Computing
SimExplorer
• software for simulation
experiment management
• includes a workflow engine that
uses JSAGA to run simulations
on grids
Business Services Provider
– used in several grid projects,
including
JSAGA
• used by experiences in
–
–
–
–
biology
computer science
physics
food industry
16
Example applications
jCAE (Java CAE)
• Computer Aided Engineering
with support for distributed
computing facilities
Calcul Intensif pour les LOgiciels
de CAO Electronique et les
applications embarquées
BioInformatique
(protein sequence alignement...)
Elis@
• web portal for industrial
and academic production
grids
/
JSAGA
• Elis@ is used in several
grid projects, including
Plasturgy (simulation of injection…)
17
Other interoperability tools – JJS
Java Job Submission
uses JSAGA in order to hide
middleware evolutions for
– security
• globus proxy VOMS proxy
– data management
tool to submit jobs on
EGEE/EGI grid infrastructure
– efficient and scalable
– resource selection based on
the QoS observed while
submitting jobs
• gridFTP SRM
http://cc.in2p3.fr/docenligne/269
JSAGA
– execution
• LCG-CE CREAM-CE
• WMS
used by D0 experiment to
submit 15 000 jobs/day on
– EGEE/EGI (mainly Europe)
– Open Science Grid (USA)
18
Other interoperability tools – JUX
Java Universal eXplorer
multi-protocols file explorer
extensible
http://cc.in2p3.fr/docenligne/821
– for protocols
• just put the JSAGA
plug-ins into the
directory "lib/".
– for viewers
JSAGA
19
Other interoperability tools – JPySAGA
C Python
Jython
a user application
Python Bindings for SAGA
SAGA-C++Py
legacy python bind.
Boost wrapper
C++ Bindings for SAGA
SAGA-C++
?
JPySAGA
JySAGA
Java Bindings for SAGA
JSAGA
Java SAGA
Java GAT
JSAGA
20
Upcoming…
Develop new plug-ins
– NorduGrid middleware
• in progress…
– batch systems (DRMAA-based)
• Grid Engine
– batch systems (CLI-based/ssh)
• torque
• Grid Engine
Develop API extensions
– Grid-RPC (from SAGA core)
– Service Discovery API (a SAGA
extension)
• with plug-ins for technologies
– BDII
• with plug-ins for schemas
– GLUE
– CIM
Interoperability demo at next OGF (Brussels, 25-29 October)
– objective is not to show interoperability between middlewares, but to show
interoperability between SAGA implementations
JSAGA
21
Upcoming…
job
Job submission
– principle
• describe user's job
• submit it to the grid
– middleware select the
execution host
– OGF standards
• BES, DRMAA, SAGA
– examples
JSAGA
Grid-RPC
Remote Procedure Call (RPC)
– principle
• deploy user's service on grid
• call a procedure
– middleware select the service
instance
– OGF standards
• Grid-RPC, SAGA
– examples
22
JSAGA
23