Transcript jsaga
EGI Technical Forum Amsterdam, 16 September 2010 Sylvain Reynaud Plan SAGA quick overview JSAGA focuses – – – – – uniform usage of middleware ease of extension efficiency and scalability control of the API behavior operating-system independency JPySAGA Example applications Perspectives JSAGA 2 SAGA quick overview SAGA : the specification – a recommendation of Open Grid Forum – aS Simple A API – for using G Grid middleware • execution management – job submission, monitoring... • data management independent of middlewares => use middlewares in a uniform way consistent "look-and-feel" => use functionalities in a uniform way – transfer, list files, search… • service discovery • … – in user A Applications • implementations (or wrappers) exist for C++, Java and Python JSAGA See next talk ! 3 The JSAGA implementation a Java implementation of the SAGA specification focuses on – – – – – uniform usage of middleware…. ease of extension………………. efficiency and scalability………. control of the API behavior........ operating-system independency (beyond what they natively support) (thanks to the design of adaptor interfaces) (configure binding rather than trying) (tested on ) Under LGPL license JSAGA 4 Uniform usage of existing grids JSAGA 5 Example – uniform usage SAGA job user applications JSAGA Globus Gatekeep. gsiftp WMS VOMS GridFTP JSAGA 6 Example – what happens behind SAGA job user applications wrapper script Globus Gatekeep. gsiftp job WMS modified JSAGA VOMS EGEE proxy + attributes translate submit job poll status delegate transfer JDL RSL DGrid proxy adapt + translate submit job receive status transfer – to any protocol VOMS WMS GridFTP LCG-CE JSAGA CREAM-CE Globus GK Unicore 7 interface Layered software architecture implementation JSAGA JSAGA 8 interface Layered software architecture user applications SAGA Java binding JSAGA core engine JSAGA JSAGA adaptors interfaces JSAGA Used by end users – web portals, GUI, CLI… Used by application developers – a single uniform interface – object oriented, high-level Implemented by adaptors – each "way" to implement a given feature has 1 interface – service oriented, low-level adaptors middleware APIs implementation Implemented by middleware – each technology has its own interfaces 9 Layered software architecture user applications Used by end users – web portals, GUI, CLI… Used by application developers – a single uniform interface – object oriented, high-level JSAGA core engine JSAGA adaptor interfaces adaptor Globus GK upload data get output get status Implemented by adaptors – each "way" to implement a given feature has 1 interface – service oriented, low-level adaptor GridFTP JSAGA Implemented by middleware – each technology has its own interfaces 10 Layered software architecture user applications Used by end users – web portals, GUI, CLI… Used by application developers – a single uniform interface – object oriented, high-level JSAGA core engine JSAGA adaptor interfaces adaptor Globus GK upload data get output get status Implemented by adaptors – each "way" to implement a given feature has 1 interface – service oriented, low-level adaptor GridFTP JSAGA Implemented by middleware – each technology has its own interfaces 11 Layered software architecture user applications JSAGA core engine JSAGA adaptor interfaces adaptor Globus GK upload data Middleware API can be used more efficiently – engine selects best interface for each user request adaptor GridFTP JSAGA get output get status Developing new adaptors is easier – just wrap functionalities supported by middleware API 12 Supported technologies local ssh naregi cream unicore6 wsgram zip file ftp sftp https http irods srb cache Physical srm gsiftp rbyteio catalog irods srb lfn X509 Login / pwd SSH JKS VOMS MyProxy G. RFC820 G. Legacy Globus done construction JSAGA planned InMemCred Security Logical rns Data gLite-WMS gatekeeper Execution Minimal dependencies on external libraries external tools – e.g. no gLite-UI operating system – tested on 13 Java-based Python SAGA wrapper C Python Jython a user application Python Bindings for SAGA SAGA-C++Py legacy python bind. Boost wrapper C++ Bindings for SAGA SAGA-C++ ? JPySAGA JySAGA Java Bindings for SAGA JSAGA Java SAGA Java GAT JSAGA 14 Example applications DIRAC (Distributed Infrastructure with Remote Agent Control) • the pilot jobs system of LHCb experiment JUX (Java Universal eXplorer) • a multi-protocols extensible file explorer Pandora Gateway • the grid platform of the company • a Grid/Cloud Computing Business Services Provider JSAGA 15 Example applications Elis@ • web portal for industrial and academic production grids • software for simulation experiment management • includes a workflow engine that uses JSAGA to run simulations on grids / JJS (Java Job Submission) • tools for submitting jobs to EGI • resource selection based on QoS observed while submitting jobs JSAGA SimExplorer jCAE (Java CAE) • Computer Aided Engineering with support for distributed computing facilities 16 Perspectives Develop new plug-ins – NorduGrid middleware • ARC plug-in in progress… – LRMS (DRMAA-based) • Grid Engine – LRMS (CLI-based/ssh) • torque • Grid Engine JSAGA Develop API extensions – Grid-RPC (from SAGA core) – Service Discovery API (a SAGA extension) • with plug-ins for technologies – BDII • with plug-ins for schemas – GLUE – CIM 17 Perspectives job Job submission – principle • describe user's job • submit it to the grid – middleware select the execution host – OGF standards • BES, DRMAA, SAGA – examples JSAGA Grid-RPC Remote Procedure Call (RPC) – principle • deploy user's service on grid • call a procedure – middleware select the service instance – OGF standards • Grid-RPC, SAGA – examples 18 Future direction of interop. Client and server sides interoperability are complementary – Server-side interoperability efforts (in particular standardization) are essential to slow down middleware APIs diversity – But can we imagine server-side interoperability for all kind of distributed resources ? • HPC/HTC • academic/industrial grids • international/regional grids • clusters, SSH server, local host • RPC grids, desktop grids, clouds • next technologies… – Server-side interoperability depends on middleware and resource providers only • with client-side interoperability, (advanced) users can integrate themselves resources they need to access • …and share their contribution with others ! JSAGA 19 JSAGA 20