Grid Computation

Download Report

Transcript Grid Computation

grid computation and web services
1
Agenda
• Grid computation.
– Motivation, a very simple example.
– Architecture and implementation problems.
• Web Services
– What are they?, why do we need them?
– Some technologies involved XML, SOAP, WSDL,
UDDI.
• Putting it all together
grid computation and web services
2
Motivation (some history)
• 95% of CPU power in the world is idle.
• Multiple scientific tasks like require
supercomputing power (weather
forecast, weapons simulations, etc).
• The cost of this supercomputers is
millions of dollars.
• Why not using existing resources to
tackle this tasks.
grid computation and web services
3
The Idea
• Use the CPU power available with the
help of volunteers ready to contribute
to your project.
• Connect the computers throw existing
networks (internet).
• Get a de facto supercomputer.
• Save millions.
grid computation and web services
4
[email protected]
• SETI (Search for Extra-Terrestrial Intelligence).
• [email protected] connects more than 4.000.000
computers throughout the world having a workpower bigger than any existing super-computer.
• SETI Statistics.
–
–
–
–
Users: 4.763.661 (the original target was 300.000).
Total CPU time: 1706429.454 years.
FPO: 3.979678e+21.
24 hours FPO 61.86 TFLOPS/sec.
• The earth simulator (350 million US$) has 40
TFLOPS/sec.
grid computation and web services
5
CRASS (1)
• The problem.
– More than 7000 observed orbiting
objects larger than 10 cm.
– Between 70.000 to 120.000 objects
larger than 1 cm.
– High speeds can make collisions fatal,
making the debris problem worse.
– 1 on 10 shuttle missions have performed
collisions avoidance maneuvers.
grid computation and web services
6
CRASS (2)
• USS space com DB tracks the
position of the debris.
• CRASS uses the DB resources and a
model of the forces acting to predict
possible collisions.
• CRASS master assigns to each node
the operational spacecraft and some
debris.
grid computation and web services
7
CRASS (3)
• The nodes calculate the debris orbit
and issue collisions warnings in a
specific time scope.
• Master receives the results and assign
another period of time till the
simulation is completed.
grid computation and web services
8
CRASS (4): Architecture
MASTER
Scheduler
node
node
grid computation and web services
node
DB
- Test speed of the nodes
- Distribute the debris
accordingly
node
9
Conclusions for CRASS
• CRASS works because the debris does not
interact between them.
• No need of communication between nodes.
• Overhead of data is small (the number of
operational spacecraft is small in relation to
the debris).
• The problem is easily distributed.
• But there are other not so simple cases.
• Grids are more than CRASS.
grid computation and web services
10
A Grid is more than that
• A grid is a seamless computing
information environment composed of:
– Storage systems.
– Networks.
– Data servers.
– On-line instruments (telescopes, particle
accelerators, etc).
– Computing processors.
grid computation and web services
11
Implementing a Grid
• Putting together so different elements
is a complex task.
– Need of a common language between
them.
– Need of communications protocols.
– Security issues.
• Grid applications are usually
distributed and thus hard to program.
grid computation and web services
12
Implementing a Grid (2)
• Component (object) oriented
programming has helped.
– Reusability of code.
– OO design subdivides problems in small
ones. Suitable approach to deploy big
applications.
• Making objects work together is still a hard
task involving many issues.
• CORBA was supposed to be the solution
grid computation and web services
13
More difficulties
• Slow communications between computers.
• Wide geographical area.
• Different users platforms (WIN, UNIX,
MAC).
• Some problems require strong interaction
between the different CPUs.
– For these problems supercomputers cannot
be replaced (for example: chain reactions,
weather simulations...).
– So we still need supercomputers.
grid computation and web services
14
Dividing the problems
• It’s very important to divide developers
and users.
– Scientific programmers (no knowledge of
grid technologies).
– Grid developers (no knowledge of the
scientific problem).
– End users (no knowledge at all).
• Each group needs appropriate tools to
solve their problems.
• We need tools to connect the elements.
grid computation and web services
15
Dividing the problems (2)
• Throughout the presentation we will
focus in the second group problems
(grid architecture).
• Some new arising tools called hosting
environments help also to deploy
highly distributed applications.
– The programs (elements, objects) do not
run over the OS but over a hosting
environment.
grid computation and web services
16
Grid Portals
• Grid portals are a solution for end users.
• They provide an easy way to use then
resources of a grid with no special
knowledge of computing.
• The user will be able to use the portal for:
– Locate grid resources.
– Set in the portal the task he wants to perform.
– Define execution parameters (number of
processors, memory needs, etc).
grid computation and web services
17
Nimrod Portal Applet
grid computation and web services
18
Grid Architecture elements
• Security Protocols
– Authentication and privacy.
• Information Services
– users need to see which resources are
available for use.
• Schedulers
– multiple resources can be scheduled
concurrently.
grid computation and web services
19
An Example of a Grid
Portal Architecture
Web Browser
(2)
(1)
Certificates
Server
Web Server
(4)
(3)
Messages DB
Application
Manager
grid computation and web services
(4) The Grid
(1)
(2)
(3)
user browser
Server
checks
Application
connects
if the
toManager
the
user is
Portal
a
is
connects
registered
notified
Server.
to
of the
grid
the user
user presence
through
a messages
channel
20
Towards Grid Services
• The grid model is developing.
– Not only scientific cooperation.
– Business want to link their different departments
maybe in a very wide geographical area using
grid technologies.
– Outsourcing: companies are using specialized
companies and reducing their IT departments.
• Economies of scale.
• Price/performance ratio.
grid computation and web services
21
Towards Grid Services (2)
• We would like to make grids more open.
Other people to use the grid capabilities.
• Redefine the grid elements as services.
• service: network-enabled entity that
provides some capability.
• The key issue is interoperability between
services.
grid computation and web services
22
Towards Grid Services (3)
• Redefine the architecture elements of
a grid as services:
– Security services: authentication and privacy.
– Information services: what resources do we
have in the grid (data, processors, instruments).
– Job-submission resources.
– Co-scheduling service: multiple resources.
– Caching: in large grids caching data can
improve performance dramatically.
grid computation and web services
23
Towards Grid Services (4)
• The interoperability problem basically
consists in:
– Defining what the service does.
– Giving instructions for it’s use (invocation
APIs)
• We would also like a mechanism to
find appropriate services.
• We need semantic information about
the services.
grid computation and web services
24
It’s Not Easy
• We need specific protocols (hopefully
universally accepted) to define our
services.
• XML helps with it’s semantic approach
but it’s not enough.
• If we use the Internet as a transport
channel how do we avoid firewalls?
– RPCs are not firewall-friendly.
grid computation and web services
25
Agenda
• Grid computation.
– Basic ideas.
– Implementation problems.
• Web Services
– What are they?, why do we need them?
– Some technologies involved XML, SOAP, WSDL,
UDDI.
• Putting it all together
grid computation and web services
26
What the Web Can Offer
• Web sites offer a myriad of services. Buy
books, check stocks, bank transactions…
• We would like to be able to do that
automatically.
• Amazon could buy book from providers
when it’s DB shows stock is low.
• Business would like to talk to each other.
• We need a ways to define and provide
services.
grid computation and web services
27
Some History
• Earlier attempts to B2B failed because
HTML was not able to give enough
semantic information about the services
provided.
• In 2001 several companies got together
with W3C and came up a precise set of
XML based standards in order to help B2B.
grid computation and web services
28
Some Standards
• WSDL Web Services Description Language.
– An XML Schema that defined the service
capabilities and how to invoke it.
• UDDI Universal Description, Discovery and
Integration.
– Registry specification on how to publish WSDL
documents.
• WSIL Web Services Inspection Language.
– How to find WSDL documents on a site.
• The idea is that these tools will help a user to find a
WS and use it (automatically).
grid computation and web services
29
A Possible Scenario
USER
User
User
chooses
checks
prices
the
best
offered
price
Registry
sends
to
user
the by
USER
wants
to
buy
books
and
by
the
buys
services
the
book
matching
services
ISBN,
searches
the registry
using WSIL
Buy books by isbn WS
Amazon.com
WS Registry
BN.com
HarperCollins.com
Registry (written in UDDI)
contains WSDL documents
describing the services
grid computation and web services
30
Using a WS
• There are two types of web services.
– RPC (remote procedure call): the user
calls a function provided by the service
and gets a value.
– DOC: the user invokes the service and
gets a documents as a response.
• In order to use the service we need to know
how to call it, i.e under which protocol is it
working (TCP, SMTP, HTTP or other).
grid computation and web services
31
Understanding a WS
• A WS is a collection of endpoints.
• An endpoint is a combination of a
binding and an address (URI).
• A binding is a concrete protocol and
data format for a port type
• The set of message exchanges is
called an operation in WSDL terms:
– error messages, encoding, parameters
type, etc.
grid computation and web services
32
Understanding a WS (2)
• Port types are interfaces (in wsdl 1.2
they are changing the term).
• Related concrete end points can be
grouped in interfaces and thus provide
abstract endpoints, also called
services
grid computation and web services
33
Messages
TCP
HTTP
SMTP
endpoints
interface
interface
operation
operation
operation
operation
SERVICE
Application (resource)
grid computation and web services
34
SOAP
• Communication with the WS is done
through messages, however we do
not want to write the XML messages
ourselves.
• SOAP simple object access protocol
solves that.
• SOAP is simply a standard to send
messages. There are others, but Microsoft,
W3C and Sun use SOAP so it’s a de facto
standard.
grid computation and web services
35
SOAP (2)
• We send a message in a SOAP envelope.
Using whatever protocol we like (FTP,
HTTP, JMS,...)
• The SOAP processor is the one in charge
of understanding the SOAP message and
call the WS.
• There are Apache packages for java and
Microsoft packages designed to use
SOAP… we’ll see some examples.
grid computation and web services
36
SOAP (3)
SOAP request
SOAP response
Client
XML Parser
Application
XML Parser
HTTP
(SOAP
translator)
Server
Application
Firewalls
grid computation and web services
37
Using SOAP to Invoke a WS
• Barnes and Noble provides a service
to get a books price with it’s ISBN
– method name: getPrice(String isbn).
– The service is located in the Web Server
of Xmethods.
• Send though HTTP a SOAP message
requesting the price of isbn =
0439139597 (Harry Potter 5).
grid computation and web services
38
The SOAP envelope header:
information about the SOAP
specification
SOAP request
<SOAP-ENV:Envelope
xmlns:SOAP-ENV=
"http://schemas.xmlsoap.org/soap/envelope/"
The information about the method
xmlns:xsi="http://www.w3.org/1999/XMLSchemainstance"
invoked and the encoding type
xmlns:xsd="http://www.w3.org/1999/XMLSchema">
The parameter needed (isbn)
<SOAP-ENV:Body>
<ns1:getPrice xmlns:ns1="urn:xmethodsBNPriceCheck"
SOAP-ENV:encodingStyle=
"http://schemas.xmlsoap.org/soap/encoding/">
<isbn xsi:type="xsd:string">0439139597</isbn>
</ns1:getPrice>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
grid computation and web services
39
SOAP answer
<SOAP-ENV:Envelope
xmlns:SOAP-ENV=
"http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsi="http://www.w3.org/1999/XMLSchemainstance"
xmlns:xsd="http://www.w3.org/1999/XMLSchema">
The return value of the method
<SOAP-ENV:Body>
<ns1:getPriceResponse xmlns:ns1="urn:xmethodsBNPriceCheck"
SOAP-ENV:encodingStyle=
"http://schemas.xmlsoap.org/soap/encoding/">
<return xsi:type="xsd:float">15.57</isbn>
</ns1:getPriceResponse>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
grid computation and web services
40
Using JAVA with SOAP
• SOAP packages for JAVA give the
possibility to:
– Create SOAP envelopes.
– Set the envelope data (URL, URI,
method name, parameters,…
– SEND the envelope.
– GET the answer (also a SOAP
envelope).
– PARSE the envelope and get the
relevant data.
grid computation and web services
41
Java SOAP
import java.net.URL;
import java.util.Vector;
import org.apache.soap.*;
import org.apache.soap.rpc.*;
public class Client {
public static void main(String[] args) throws
Exception {
The URN of the method and the
URL url = new
invoked method name
URL("http://xmethods.vet/sd/2001/"+
”BNQuoteService");
// Build the call.
Call call = new Call();
call.setTargetObjectURI("urn:xmethodsBNPriceCheck");
call.setMethodName(”getPrice");
grid computation and web services
42
Java SOAP (2)
call.setEncodingStyleURI(Constants.NS_URI_SOAP_ENC);
Vector params = new Vector();
params.addElement(new Parameter("name", String.class,
args[0], null));
call.setParams(params);
// Invoke the call.
Response Setting
resp =args[0]
null;as the isbn to look
try {
resp = call.invoke(url, "");
}
catch( SOAPException e ) {
System.err.println("Caught SOAPException (" +
e.getFaultCode() + "): "+ e.getMessage());
System.exit(-1);
}
grid computation and web services
43
Java SOAP (3)
// Check the response.
if( !resp.generatedFault() ) {
Parameter ret = resp.getReturnValue();
Object value = ret.getValue();
System.out.println(value);
}
else {
Fault fault = resp.getFault();
Getting the fault:
returned");
value
System.err.println("Generated
System.out.println (" Fault Code
= " +
fault.getFaultCode());
System.out.println (" Fault String = " +
fault.getFaultString());
grid computation and web services
44
Agenda
• Grid computation.
– Basic ideas.
– Implementation problems.
• Web Services
– What are they?, why do we need them?
– Some technologies involved XML, SOAP, WSDL,
UDDI.
• Putting it all together
grid computation and web services
45
The Idea
• Use the WS standards to solve the
implementation problems of a grid.
• WS are platform-independent and
programming-language independent.
• Communication protocols are widely
accepted (firewalls-friendly).
• WSDL offers a way to publish the
services.
• UDDI offers a way to find the wanted
service.
grid computation and web services
46
Grid Services As WS (2)
• However there are grid needs that are
not provided by the WS specifications
and tools. We need new interfaces.
– Grids need dynamic creation of
application instances. (The application
instance is created when the user needs
it).
– Grids need lifetime management.
operations.
– WS are usually stateless.
grid computation and web services
47
Grid Factories
• The factory creates an instance of a new WS in
some resource.
– Dynamic creation of WS is supported is hosting
environments as J2EE Server, .NET and AXIS.
• An application manager instance is created and
publishes a WSDL document.
• The client can contact the new instance
through the WSDL directly (we do not need the
factory any more).
– A messages channel is opened.
• Keep alive messages, destroy...
grid computation and web services
48
Grid Factories (2)
Grid Factory
Service
User
Resource Broker
Service
Application Instance
Service
grid computation and web services
Authentication
Service
Grid users
DB
49
Lifetime Management
• Normally a transient service will be created
and will run till it’s task termination.
However errors can (and do) happen.
• Need a ways to kill idle WS that consume
resources. A possible solution will be:
– The service is created with a specific
lifetime.
– The user can request to extend the lifetime.
– The user needs to send keep-alive
messages, or explicitly kill it.
grid computation and web services
50
User
Storage Factory
Service
user
DB
Mining Factory
Service
Storage Service
Instance
Storage
Mining Service
Instance
- As
The
user
invokes
“create
service”
from
the 2arethe
Keep
no
Alive
more
messages
keep-alive
and
messages
status messages
are
received
sent
factories.
Storage
room is
allocated.
services kill
themselves
freeing
all their resources.
grid computation and web services
51
The role of hosting
environments
• A grid service architecture should be
independent of the implementations issues.
• Nowadays most grid application rely in
native OS (for example the creation of a
new service means the creation of a
process).
• This fact means for example that the factory
service has to be platform-aware.
grid computation and web services
52
The role of hosting
environments (2)
• WS can be created with more sophisticated
tools called hosting environments or
containers as J2EE (Sun), WebSphere
(IBM) or .NET (Microsoft)
– These containers work in a componentbased basis.
• Hosting environments build complex
applications by using components that
adhere to specific interfaces (i.e. EJB).
grid computation and web services
53
The role of hosting
environments (3)
– Building applications which can have
components distributed in different
machines and places.
– Communications are handled by the hosting
environment.
– Easy security.
– Is not the component that handles
communication or creation of instances but
the hosting environment.
– Messaging between components is also
handled by the host no matter where the are.
grid computation and web services
54
The role of hosting
environments (4)
An accountancyThe
bean askslooks
container
beans
the
bean
host
in
communicate
thefor
wants
directory
a handle
to as
getifto
the
sales-bean.
some
where
they
where
data
the bean
from
running
is
the
and
returns
handle
sales
in
the
department
same a
machine.
DB.
Communications
and security issues
are managed from
the host
Container
Clients
DB
Sales bean
Accountancy
bean
grid computation and web services
55
The role of hosting
environments (5)
• This approach affects also the third
level of users (scientific
programmers).
• The programmer needs to follow
specific interfaces for his application.
• In other words if he is a java
programmer he needs to deliver
beans.
grid computation and web services
56
Conclusion
• WS provide a powerful tool to
implement grid services.
• New interfaces (hopefully widely
accepted) should be created to
support specific grid needs:
– Factories.
– Lifetime management.
– Security...
grid computation and web services
57
References
• Articles.
– The anatomy of the grid.
www.globus.org/research/papers/anatomy.pdf.
– The physiology of the grid.
www.globus.org/research/papers/physiology.pdf.
– Programming the grid: distributed software
components, P2P and grid web services for
scientific applications.
http://citeseer.nj.nec.com/gannon02programming.html.
• Grid services for distributed system
integration.
www.globus.org/research/papers/ieee-cs-2.pdf.
grid computation and web services
58
References (2)
• Some links
–
–
–
–
SOAP: www.w3.org/TR/SOAP/
UDDI: www.uddi.org
WDSL: www.w3.org/TR/wsdl
To see some WS in action:
www.xmethods.net
– Microsoft’s WS page:
http://msdn.microsoft.com/webservices/defa
ult.aspx
grid computation and web services
59