Transcript Workflow Optimisation

Workflow Optimisation Services
for e-Science Applications
David W. Walker
Cardiff University
WOSE Overview
• Draws together JISGA and Triana work
at CU, with ICENI at IC, and portal
expertise at DL.
• Topics addressed
– Service aggregation and deployment
– Runtime discovery and late binding of
services
– Service discovery and selection from
multiple semantically equivalent services
Workflow Optimisation
• Types of workflow optimisation
– Through service selection
– Through workflow re-ordering
– Through exploitation of parallelism
• When is optimisation performed?
– At design time (early binding)
– Upon submission (intermediate binding)
– At runtime (late binding)
Service Binding Models
• Late binding of abstract service to
concrete service instance means:
– We use up-to-date information to decide
which service to use when there are.
multiple semantically equivalent services
– We are less likely to try to use a service
that is unavailable.
Late Binding Case
• Search registry for all services that are
consistent with abstract service description.
• Select optimal service based on current
information, e.g, host load, etc.
• Execute this service.
• Doesn’t take into account time to transfer
inputs to the service.
• In early and late binding cases we can
optimise overall workflow.
WOSE Architecture
Configuration
script
User
Web service
instance
Workflow
script
Converter
ActiveBPEL
workflow engine
Discovery
Service
Registry services
(such as UDDI)
Proxy
Optimization
Service
Work at Cardiff has
focused on implementing
a late binding model for
dynamic service
discovery, based on a
generic service proxy,
and service discovery
and optimisation
services.
History database
Performance
Monitor Service
Service Discovery Issues
• Service discovery and optimisation is
based on service metadata.
• Could store in a database.
• Could obtain by interrogating service.
Optimisation by Re-Ordering
• Work at Imperial has looked at static optimisation
– Optimise the runtime execution of workflow before it is
executed
– Achieves the goal through:
•
•
•
•
Re-ordering of components
Addition of components
Substitution of components
Pruning of the workflow
• Performance and workflow aware Scheduling
• Runtime Optimisation
– through monitoring, check-pointing and migration
Component Manipulation
• Re-ordering: Workflows (often composed
from composite workflows) may contain
non-optimal ordering of components
– Use re-ordering to improve performance
Matrix
Gen
Matrix
Gen
Vector
Gen
Matrix
Gen
multiply
multiply
Matrix
Gen
Vector
Gen
multiply
multiply
Component Addition
• Addition: For a component requiring a
specific format of data as input, a
transformer component could be added
to achieve the desired format.
– Allows more optimal components to be
used together
C1
Output in LP
format
Input required
in MPS format
LP to MPS
C2
Component Substitution
• Substitution:
– A Jacobi Iteration linear solver replaced by
Conjugate Gradient linear solver according to
the output of the Discretizer (FEM)
• Based on observing the meta-data associated with
previous components
A (sparse and diagonally
dominant)
JI linear solver
FEM
b
Pruning
• Workflow Pruning:
– Workflows may contain unused
components. Especially when
composed from other sub-workflows
• Remove redundant components
a
f
b
e
d
c

g
Not needed

h
Not needed
Work Done at Daresbury
• Daresbury started working with evaluating different BPEL engines to
have proper understanding of BPEL specification i.e. ActiveBPEL,
Oracle BPEL engine initially Collaxa BPEL engine.
• Most projects at CCLRC are portal based, so we studied the relevance
of Portal based workflow monitoring and controlling solution.
• Web Services for Remote Portlets (WSRP) is an emerging specification,
with limited support - seems to bridge Portal and Web Services and a lot
of work has been done, with demo prototype.
• WSRF is extension of existing Web Services and provides extra layer of
functionality without changing the Web Service implementation.
• WSRF extension for Web Service starts from modification of existing
WSDL file of specific Web Service.
• WSRF extension of WSDL is not straight forward and needs a thorough
understanding of WSDL and WSRF.
• Daresbury has built working prototype for testing Web Services based
on WSDL.
Future Work
• Complete WOSE prototype with
dynamic scheduling.
• Compare dynamic and static
approaches at Cardiff and Imperial.
• Improve Discovery Service
mechanisms.
• Investigate different approaches to
optimisation.