Transcript Paul Watson - National e

Dynasoar
Dynamic Deployment of Web Services on a
Grid or the Internet
or
Why it’s good to be Jobless
Paul Watson
School of Computing Science
University of Newcastle
The Dynasoar team: Chris Fowler, Paul Watson,
Charles Kubicek, Arijit Mukherjee, John
Colquhoun, Savas Parastatidis, Mark Hewitt
The GridShed team: Isi Mitrani, Jennie Palmer,
Paul McKee (BT) & Mike Fisher (BT)
www.neresc.ac.uk
Why Jobs & Services?
• Grid applications are being built from Web Services
• If the computational requirements can’t be met by the
service hosting environment then a job must be created
• Do we need both jobs and services?
• Dynasoar
• a service-only approach to building grid applications
• an infrastructure for the dynamic deployment of web services
www.neresc.ac.uk
2
Dynasoar Components
req
C
Consumer
req
WSP
res
HP
res
• Web Service Provider (WSP)
•
•
•
•
exposes service endpoints
accepts the incoming SOAP message sent to the endpoint
chooses a Host Provider and passes the message to it
holds a copy of service code
• Host Provider (HP)
•
•
•
•
manages computational resources (e.g. a cluster or a grid)
accepts the message from the WSP
dynamically deploys the service if necessary
processes the message and returns any response
www.neresc.ac.uk
3
Routing to an Existing Service Deployment
node 1
s2, s5
req
node 2
C
WSP
Consumer
…
res
Web Service
Provider
node n
s2
Host Provider
A request for s2 is routed to an existing
deployment of the service
www.neresc.ac.uk
4
Dynamic service deployment
A request to s4 cannot be met by an existing deployment of the service
2: service fetch &
deploy
R
req
1
C
WSP
Web Service
Provider
The deployed service remains in place and
can be re-used
- unlike job scheduling
www.neresc.ac.uk
3
node 2
…
res
Consumer
node 1
s2, s5
node n
s2
Host Provider
5
Dynasoar Advantages
• Simplicity: just services
• Efficiency: a deployed service can process many
messages
• Support a range of new e-science/ e-business models:
• defining the interactions between the major components allows
them to be distributed in a variety of ways
www.neresc.ac.uk
6
Dynamic Outsourcing
C
…..
C
WSP
HP
BioCorp
Hosting Inc
BioCorp
Customers
• Biocorp are experts in writing bioinformatics services
• They don’t want to manage their own compute resources
• Therefore, they use Hosting Inc to process messages sent to their
services
• In e-science, BioCorp could be a research group writing specialist escience services, and Hosting Inc the NGS
www.neresc.ac.uk
7
The National Grid Service as a Host Provider
node
C
…...
node
WSP
..
Researcher’s
Local Resources
node
node
Quarantine Nodes
National Grid
• A researcher writes their own services but does not have sufficient
local compute resources
• They deploy a local WSP, and configure it so that it sends messages
to the National Grid Service
• their services are then transparently deployed on the NGS as required
www.neresc.ac.uk
8
A Marketplace for Matching Web Service
Providers to Host Providers
WSP1
HP1
......
......
Marketplace
WSPn
HPn
Web Service
Providers
Host Providers
www.neresc.ac.uk
9
A Marketplace for e-Science
WSP1
HP1
......
......
Marketplace
Local
Campus
Grid
National
Grid
Service
WSPn
HPn
Web Service
Providers
Host Providers
www.neresc.ac.uk
10
Moving Computation to Data
• In many e-science applications analysis services operate on data
extracted from a data store (e.g. OGSA-DAI, SRB…)
• often large amounts of data are transferred
• this may severely limit the performance
req
C
Analysis Service
res
www.neresc.ac.uk
req
Database
Service
res
11
Moving Computation to Data
2: service fetch &
deploy
node 1
req
1
C
WSP
Consumer
node 2
Database
Service
…
res
3
High Performance
Network
Web Service
Provider
offering the
Analysis Service
node n
Host Provider
• The data owner provides compute resources close to a database
• Researchers can write services and deploy them on their own WSP
• The service is dynamically deployed close to the database when
requests are sent to the WSP
www.neresc.ac.uk
12
Results for Deploying a Service Close to a Database
www.neresc.ac.uk
13
Current Implementation
SOAP Endpoints
Dynasoar
Host Provider
Interface
Message Handler
Security Policies
Deployed Service
Registry
Host Provider
Computational resources
Hosting deployed services
GridShed Cluster
Management
Host Provider
Registry
QoS Policies
Web Service
Code Store
Host Provider
Endpoint
Code Store SOAP
interface
Service Provider
Endpoint
Service Provider
www.neresc.ac.uk
14
New Host Provider Architecture
• Layer as high-level infrastructure over lower level grid fabric
• Use OMII Job Submission and Monitoring Service to provide stable
interface to different underlying fabrics
• Newcastle Grid (Condor), National Grid Service, local clusters,….
Dynasoar
Host Provider
Interface
Host Provider
www.neresc.ac.uk
OMII Job Submission and
Monitoring Service
(WS-JDML)
Computational resources
Hosting deployed services
15
Current Work
• Exploring Virtual Machines as a general service
deployment mechanism
• Freeze services and their environments in a VM
• Store in Service Store
• Dynamically Deploy as required
• Use of QoS to enhance decisions on where to deploy a
service
• Exploring tripartite security model
• Consumer, Web Service Provider and Host Provider express
policies that are enforced at run-time
• A HP may only accept messages from WSPs that it trusts to not send
malicious code
• A WSP may only deploy services on HPs it trusts won’t use the service without
paying
• Dynamic database deployment
• ogsa-dai, ogsa-dqp
www.neresc.ac.uk
16
Conclusions
• It is possible to build grid applications entirely from
services
• jobless grid computing
• simpler conceptual model
• performance improvements due to sharing the cost of service
deployment over multiple requests
• Separating the Web Service Provider from the Host
Provider opens a range of deployment options
• Dynasoar can be built as a high-level infrastructure on
top of existing grid fabrics
• Ongoing work on VMs, QoS, Security, dynamic db
deployment
• Technical Report on-line: Newcastle CS-TR-890…
• http://www.cs.ncl.ac.uk/research/pubs/trs/papers/890.pdf
www.neresc.ac.uk
17
Thanks
• The Dynasoar team
• Chris Fowler, Charles Kubicek, Arijit Mukherjee, John Colquhoun,
Savas Parastatidis, Mark Hewitt
• The GridShed team
• Isi Mitrani, Jennie Palmer
• BT
• Paul McKee & Mike Fisher
• This work is supported by the DTI, EPSRC, Core e-Science
Programme & CodeWorks
www.neresc.ac.uk
18