Transcript ENM Service - Indico

The EUBrazilOpenBio-BioVeL Use Case in EGI
Daniele Lezzi,
Barcelona Supercomputing Center
EGI-TF2013- 16 September 2013
Open Modeller Web Service
• One WS instance and one oM server
• ~50min for a single species (until the final model is
generated)
• How to scale?
request
(Brazilian Virtual Herbarium)
response
openModeller
Web Service
(single machine)
Complex executions
•
•
•
•
•
31 718 angiosperms (flowering plants)
Assuming that 30% will have enough points to generate models (~9 000
species):
– 495k models, 540k tests, 90k projections
10 months to generate all models!
But what if we want to generate models for
– All ~43 thousand plant species from Brazil?
– Using more than one spatial resolution?
– Projecting into different environmental climatic scenarios?
– With global coverage?
Note: models may be regenerated every time new data is available for each
species...
Open Modeller Web Service 2 (OMWS2)
• Open Modeller Web Service1 is a specification to expose Open
Modeller Ecological Niche Modelling (ENM) features as a Web Service.
• It enables decoupling interface from processing and it is used to
connect OM Desktop2 applications with Web servers.
• Describes the complete life cycle of a single operation of ENM
– Complete specification of ENM experiments, including occurrences,
parameters for the algorithms and location of layers in a single document.
– One set of occurrence points, one algorithm, one operation, one set of
parameters.
• ENM processing is embarrassingly parallel though
– OMWS2 addresses this need.
1
http://openmodeller.cria.org.br/ws/2.0/openModeller.wsdl
2 http://openmodeller.sf.net
ENM Service (OMWS2)
•
Shared requirements between EUBrazilOpenBio
BioVeL
•
The EUBrazilOpenBio ENM service is exposed through an
extended openModeller Web Service interface (OMWS2 in
the picture).
•
Such interface in EUBrazilOpenBio supports multi-staging
and multiparametric experiments implemented through
COMPSs and the openModeller software and managed
through a Virtual Research Environment (VRE) portal.
•
The OMWS extensions are backwards compatible with the
original specification, allowing existing clients, as the
Taverna Workflow Management System in BioVeL, to be
fully supported in the new implementation.
•
In the case of the EGI Federated Cloud, the VENUS-C
middleware is used to instantiate openModeller workflows
on cloud resources from different providers dynamically
deployed by COMPSs.
•
An OCCI connector is used for the VMs management while
data management supports CDMI endpoints.
VENUS-C Cloud Middleware
COMPSs Workflow
Orchestrator
OCCI
CDMI
EGI Federated Cloud
and
Biodiversity VRE
Result
Visualisation
Service
Species
Discovery
Service
ENM architecture
Information
System
Workspace and
Storage Service
Use Case
Enactment
Service
Biodiversity
Use Case GUI
Open Modeller Web Service Extended
Experiment
Orchestrator
Service
OMWS2
VENUS-C
Legacy Clients
OMWS
OMWS Cluster
OMWS
Server
Condor
OMWS2
VENUS-C Cloud
PMES
Monitoring
Monitoring
Workflow
Orchestrator
Internal
Storage
Internal
Storage
Public clouds
(EC2, Azure)
Private clouds
(OpenNebula, EMOTIVE)
Execution Scenario 1: Static VMs
• ENM Service: the OpenBio ENM service receives from the BioVeL portal
simple requests for the generation of models, balancing them between different
RPs.
• COMPSs Execution Service: deployed at each site, executes the requests to
pre-deployed VMs. Additional VMs are dynamically created to serve additional
requests.
• openModeller application: the application first checks if the layer is available,
otherwise downloads it from a layers repository to the storage local to the
provider.
ENM Service (OMWS2)
COMPSs Execution Service
RP2
COMPSs Execution Service
RP1
VM1
RP1
VMn
VM1
RP2
VMn
THREDDS Data Server
Execution Scenario 2
• COMPSs Execution Service: deployed at each site, starts the
execution of complex COMPSs workflows using resources of a RP.
• COMPSs Orchestrator: executes (in parallel) the different parts of
the complex wf using additional VMs dynamically requested from
other RPs.
ENM Service (OMWS2)
• Shared Storage Issue: the layers should be accessible via a shared
storage or synchronized at each site.
COMPSs Execution Service
RP1
COMPSs Workflow
Orchestrator
VM1
RP1
VMn
VM1
RP2
Shared Storage
VMn
THREDDS Data Server
The demo testbed
• Input Data: for simplicity the layers are already stored at each site.
Cmd Line
(Complex WFs)
Taverna
Portal/Desktop
• VMs: Model operations are executed on Extra-Large instances at
CESNET, Test and Project on Large instances at CESGA. This is
achieved through the mapping of tasks requirements of the
COMPSs workflow and RP templates.
ENM Service (OMWS2)
Orchestrator
UPV
COMPSs Execution Service
• Providers: OpenNebula providers have been selected but any
rOCCI compliant RP is transparently supported.
COMPSs Execution Service
COMPSs Workflow
Orchestrator
COMPSs Workflow
Orchestrator
OpenNebula
OpenNebula
CESNET
Extra-Large instances:
4 cores, 15GB RAM. 10GB Disk
CESGA
Large instances:
2 cores, 8GB RAM. 10GB Disk