Transcript on web service

ANABAS
Use of Grids in DoD
Applications
Geoffrey Fox, Alex Ho
SAB Briefing November 16, 2005
1
The Ten areas covered by the 60 core WS-* Specifications
WS-* Specification Area
Examples
1: Core Service Model
XML, WSDL, SOAP
2: Service Internet
WS-Addressing, WS-MessageDelivery; Reliable
Messaging WSRM; Efficient Messaging MOTM
3: Notification
WS-Notification, WS-Eventing (Publish-Subscribe)
4: Workflow and Transactions
BPEL, WS-Choreography, WS-Coordination
5: Security
WS-Security, WS-Trust, WS-Federation, SAML,
WS-SecureConversation
6: Service Discovery
UDDI, WS-Discovery
7: System Metadata and State
WSRF, WS-MetadataExchange, WS-Context
8: Management
WSDM, WS-Management, WS-Transfer
9: Policy and Agreements
WS-Policy, WS-Agreement
10: Portals and User Interfaces
WSRP (Remote Portlets)
\ NCOW (and RTI) needs all of these?
2
Activities in Global Grid Forum Working Groups
GGF Area
GS-* and OGSA Standards Activities
1: Architecture
High Level Resource/Service Naming (level 2 of fig. 1),
Integrated Grid Architecture
2: Applications
Software Interfaces to Grid, Grid Remote Procedure Call,
Checkpointing and Recovery, Interoperability to Job Submittal services,
Information Retrieval,
3: Compute
Job Submission, Basic Execution Services, Service Level Agreements
for Resource use and reservation, Distributed Scheduling
4: Data
Database and File Grid access, Grid FTP, Storage Management, Data
replication, Binary data specification
and interface, High-level
publish/subscribe, Transaction management
5: Infrastructure
Network measurements, Role of IPv6 and high performance
networking, Data transport
6: Management
Resource/Service configuration, deployment and lifetime, Usage
records and access, Grid economy model
7: Security
Authorization, P2P and Firewall Issues, Trusted Computing
NCOW needs all of these?
3
The Global Information Grid Core Enterprise Services
Core Enterprise Services Service Functionality
CES1: Enterprise Services
Management (ESM)
including life-cycle management
CES2: Information
Assurance (IA)/Security
Supports confidentiality, integrity and availability. Implies
reliability and autonomic features
CES3: Messaging
Synchronous or asynchronous cases
CES4: Discovery
Searching data and services
CES5: Mediation
Includes translation, aggregation, integration, correlation,
fusion, brokering publication, and other transformations
for services and data. Possibly agents
CES6: Collaboration
Provision and control of sharing with emphasis on
synchronous real-time services
CES7: User Assistance
Includes automated and manual methods of optimizing
the user GiG experience (user agent)
CES8: Storage
Retention, organization and disposition of all forms of
data
CES9: Application
Provisioning, operations and maintenance of applications.
4
Major Conclusions I





One can map 7.5 out of 9 NCOW and GiG core
capabilities into Web Service (WS-*) and Grid (GS-*)
architecture and core services
• Analysis of Grids in NCOW document inaccurate
(confuse Grids and Globus and only consider early
activities)
Some “mismatches” on both NCOW and Grid sides
GS-*/WS-* do not have collaboration and miss some
messaging
NCOW does not have at core level system metadata
and resource/service scheduling and matching
Higher level services of importance include GIS
(Geographical Information Systems), Sensors and
data-mining
5
Major Conclusions II



Criticisms of Web services in a recent paper by Birman
seem to be addressed by Grids or reflect immaturity of
initial technology implementations
NCOW does not seem to have any analysis of how to
build their systems on WS-*/GS-* technologies in a
layered fashion; they do have a layered service
architecture so this can be done
• They agree with service oriented architecture
• They seem to have no process for agreeing to WS-*
GS-* or setting other standards for CES
Grid of Grids allows modular architectures and
natural treatment of legacy systems
6
DoD Core Services and WS-* plus GS-* I
NCOW Service or Feature
WS-* Service area
GGF
Others
A: General Principles
Use Service Oriented Architecture
WS-1: Core Service
Model
Build Grids
Services
on
Web
Grid of Grids Composition
Industry Best Practice
(IBM, Microsoft …)
Legacy subsystems
modular architecture
and
B: NCOW Core Services (to be continued)
CES 1: Enterprise Services
Management
WS-8 Management
GS-6: Management
CES 2: Information
Assurance(IA)/Security
WS-5
WS-Security
GS-7
(Authorization)
CES 3: Messaging
WS-2, WS-3
Service Internet
Notification
CES 4: Discovery
WS-6 UDDI
CES 5: Mediation
WS-4 Workflow
CES 6: Collaboration
Shared Web Resources
Asynchronous
Organizations
CES 7: User assistance
WS-10 Portlets
GridSphere
CIM
Security
Grid-Shib, Permis Liberty
Alliance etc.
NaradaBrokering,
Streaming/Sensor
Technologies
Extended UDDI
Treatment
of
systems.
Transformations
Virtual
Legacy
Data
XGSP,
Shared
Web
Service ports, Anabas
NCOW
Capability7
Interfaces, JSR168
DoD Core Services and WS-* and GS-* II
NCOW Service or Feature
WS-* Service area
GGF
Others
B: NCOW Core Services Continued
CES 8: Storage (not real-time
streams)
GS-4 Data
NCOW Data Strategy
CES 9: Application
GS-2; invoke GS-3
Best Practice in building
Grid/Web services (proxy
or direct)
Environmental
Services ECS
Control WS-9 Policy
C: Key NCOW Capabilities not directly in CES
System Meta-data
WS-7
Semantic Grid
Globus MDS
C2IEDM,
DDMS, WFS
XBML,
Resource/Service
Matching/Scheduling
Distributed Scheduling Extend
computer
and SLA’s (GS-3)
scheduling to networks
and data flow
Sensors (real-time data)
Work starting
Geographical
Systems GIS
Information
OGC Sensor standards
OGC GIS standards
8
Information Exploitation | Information Fusion & Understanding | Information Management | Cyber Operations | Command & Control | Connectivity | Advanced Computing Architectures
Grid of Grids Service Architecture for Net-Centric
Operations and Warfare
Problem: Unable to satisfy interoperability, scalability, and security information management requirements for
Net-Centric Operations without an advanced grid-based scalable service oriented framework
Objective: Integrate Global Grid Forum (GGF)
information management research to provide a Grid
of Grids Middleware for Net-Centric Operations
consistent with the Global Information Grid (GIG).
Approach:
• Analyze Net-Centric Operations and Warfare
(NCOW) service specifications and relate core
enterprise services in Net-Centric Enterprise
Services (NCES) to core GGF and Web Services
(WS-*) standards
• Develop the Grid of Grids architecture and
information management middleware to address
federation of legacy and new DoD enterprise systems
with service oriented mediation between component
Grids
• Develop prototype for NCES capabilities
(Collaboration, Messaging, Management,
Security/Information Assurance, Discovery,
Mediation, User Assistance, Storage, Applications)
with advanced Grid and Web Service standards
support
• Demonstrate for Geographical Information System
• Tech transfer Anabas & Ball Aerospace, Global Grid
Forum
Challenges:
•
Combining scalability and interoperability of
Grid and Web Services with DoD real time and
security requirements
• Supporting legacy systems and standards while
maximizing value of new commercial service
oriented architectures
Technologies:
•
•
•
•
•
Business Process Execution Language
Global Multimedia Collaboration System
Narada Brokering
Web Service Standards
Open Grid Computing Environment
Results:
•
Phase I middleware addresses 6 of 9
NCOW Core Enterprise Services
Demonstrated GIS application in
collaboration grid
Middleware demonstrated with 500 users
and 20,000 messages/second
•
•
Impact:
•
•
•
A scalable managed interoperable secure
architecture and middleware for NCOW
Rapid implementation strategy leveraging
commercially pervasive technology
Brought GIG and GGF communities
together
Team: Indiana University/Anabas (Dr. Geoffrey Fox), Ball Aerospace, AFRL/IFSD (Michelle Cheatham)
NAME | SAB2003
McQuay | SAB2005
General Message I



Our proof of concept demonstrates many of the NCOW
core enterprise services (CES) implemented using Grid
services built on top of the WS-* Web service industry
specifications.
We will illustrate the use of the Grid of Grids
architecture to integrate heterogeneous systems. The
papers describe how all CES can be implemented using
Grid technology and this is proposed in phase II SBIR.
Note the adherence to standards with a common line
protocol SOAP implies that all service implementations
are interoperable and one takes services from multiple
sources. Anabas/Indiana University only has to
implement some of the key Grid services.
10
General Message II: Why Grids





Web services gives us interoperability but Grids are
essential as we aim at Information Management
Grids are the key idea to manage complexity but
applying uniform policies and building managed
systems
Grids of Grids allows one to build out the management
in a modular fashion
Uniform Grid messaging handles complex networks
with managed QoS such as real-time constraints
Managed Services and Messaging gives scalability and
performance (later slide)
11
Scenario







Laptop 8 Windows Client (small Sony)
Laptop 2 Old Windows Client and PPT machine
Laptop 3 Windows Server
Laptop 4 Linux
Assume no Internet access
List of CES and GS-* WS-* mapping on Laptop 2
followed by complete stream of PPT screen shots if
failure
On mapping slide highlight services
12
Script I: Data Mining and GIS Grid





This will show a set of Open Geospatial Consortium (OGC)
compatible services implementing a GIS (Geographical
Information System) grid supporting streaming of feature and
map data.
Intrinsic features of a region are supplemented here by
features coming from a data-mining code that is filtering data
to predict likely earthquake positions.
This uses discovery, metadata, database, workflow, messaging,
data transformation, simulation (data-mining) services.
Note the OGC compatible WFS (Web Feature Service) plays
role as a domain specific service interface to a database
This used by Los Alamos for DHS simulations replacing data
mining by critical infrastructure simulations
13
Script I: Google Map Grid Service



This first demo also illustrates how the Google map
system can be wrapped as a Grid itself front-ended by a
OGC Web Map Service.
This is used in a Grid of Grids fashion with Google
linked with traditional (NASA) Web Map services.
Illustrates how linking NCOW to commodity Grid
technology allows access to major IT resources
• Google’s 100,000 computers
• DoD DoE NSF Supercomputers
14
Script II: Collaborative Grid Service



This demonstrates how streams can be formed from
messages and managed in a uniform way whether maps
or video. Collaboration is achieved by multicasting of
the input or output streams to Grid services.
Our messaging infrastructure handles all multicasting
(using software) transparently to services
First we demonstrate collaborative maps using “shared
input ports” on web service
15
Script III: Collaboration Grid






Collaborative Audio-video conferencing on Laptop 3 and 8
with services running on 3
Collaboration uses basic Grid services – metadata, discovery,
workflow, security plus the XGSP stream management
services.
Complex collaboration scenarios correspond to additional
services for particular shared applications and to gateways
in Grid of Grids fashion to H323, SIP and other protocols.
Annotation, record, replay, whiteboards, codec conversion,
audio and video mixing become services.
We demonstrate MPEG4 transcoding and video mixing
services
Only Grid Web service based collaboration environment
Use of Grids ensures scalability and performance (next slide)
16
Performance





Reduction of message delay jitter to a millisecond.
Dynamic meta-data access latency reduced from seconds to
milliseconds using web service context service.
The messaging is distributed with each low end Linux node
capable of supporting 500 users at a total bandwidth of 140
Mbits/sec with over 20,000 messages per second.
Systematic use of redundant fault tolerance services supports
strict user QoS requirements and fault tolerant Grid
enterprise bus supports collaboration and information
sharing at a cost that scales logarithmically with number of
simultaneous users and resources.
Supporting N users at the 0.5 Mbits/sec level each would
require roughly (N/500)log(N/500) messaging servers to
achieve full capability.
17
ANABAS
I: Data Mining and
GIS Grid
SAB Briefing November 16, 2005
18
I: Data Mining and GIS Grid
Databases with
NASA, USGS features
SERVOGrid Faults
WFS1
UDDI
Data Mining Grid
WFS3
WFS2
NASA WMS
WMS handling
Client requests
SOAP
WMS
WMS Client
Client
HTTP
19
California fault data from Quake Tables fault database via Web
Feature Service.
20
Standard Open Geospatial Consortium WMS Clients
Standard Open Geospatial
Consortium WMS Clients
Get Feature Info allows
users to get map
information. This can also
be used to read feature info
off the map when creating
input data for applications
21
I: Data Mining Grid
Databases with
NASA,USGS features
SERVOGrid Faults
UDDI
WFS4
SOAP
Pipeline
Filter
PI Data Mining
HPSearch
Workflow
Filter
WS-Context
Narada
Brokering
System Services
WFS3
GIS Grid
22
PI demo
combines WFS,
WMS, and
HPSearch for
service
orchestration.
Tool bar items
allows you to
adjust maps
Users set up problems by adding
filtered seismic archives from
WFS as map layers.
23
Hot spots
calculations-areas of
increased
earthquake
probability in
the forecast
time-calculations
are re-plotted
on the map as
features.
24
Electric Power and Natural Gas data from LANL
Interdependent Critical Infrastructure Simulations
Zoom-in
Zoom-out
FeatureInfo mode
Measure distance mode
Clear Distance
Drag and Drop mode
Refresh to initial map
ANABAS
I(contd): Google Grid
Service
SAB Briefing November 16, 2005
26
Google Maps
as Service
accessed from
our WMS
Client
27
ANABAS
II: Sensor Grid
(available on movie)
SAB Briefing November 16, 2005
28
Typical use of Grid Messaging in NASA
Sensor Grid
Grid Eventing
Datamining Grid
GIS Grid
29
Typical use of Grid Messaging
Filter or
Datamining
Sensor Grid
Post after
Processing
Post before
Processing
Narada
Brokering
Grid Database
Archives
Web Feature Service
Notify
Subscribe
HPSearch
Manages
WS-Context
Stores dynamic data
WFS
(GIS data)
GIS Grid
Geographical
Information System
30
Real Time GPS
and Google Maps
Subscribe to live GPS
station. Position data
from SOPAC is
combined with Google
map clients.
Select and zoom to
GPS station location,
click icons for more
information.
31
Google maps
can be
integrated with
Web Feature
Service
Archives to
filter and
browse seismic
records.
Integrating
Archived Web
Feature Services
and Google Maps
32
ANABAS
III: Collaborative
Google Grid Service
SAB Briefing November 16, 2005
33
Collaborative Google Maps
with faults from WFS
34
ANABAS
IV: Collaboration Grid
SAB Briefing November 16, 2005
35
GlobalMMCS Web Service Architecture
Use Multiple Media servers to scale to many codecs and many
versions of audio/video mixing
Session Server
XGSP-based Control
NaradaBrokering
All Messaging
NB Scales as
distributed
Admire
Web
Services
SIP
H323
Media Servers
Filters
High Performance (RTP)
and XML/SOAP and ..
Access Grid
Native XGSP
Gateways convert to uniform XGSP Messaging
NaradaBrokering
36
Average Video Delays for one broker –
Performance scales proportional to number of brokers
Latency ms
Multiple
sessions
One session
30 frames/sec
# Receivers
37
Collaboration Grid
WS-Context
HPSearch
UDDI
Narada
Broker
Audio Mixer
Video Mixer
Narada
Broker
WS-Security
Gateway
XGSP Media
Service
Narada
Broker
Gateway
SharedWS
Transcoder
Thumbnail
Replay
Record
Annotate
SharedDisplay
WhiteBoard
38
GIS
TV
Chat
Video Mixer
Webcam
GlobalMMCS SWT Client
39
ANABAS
V: Annotation Service
SAB Briefing November 16, 2005
40
e-Annotation
e - Annotation
Player
Player
Archived
Archieved
stream list
Stream List
Archived Stream
Archived stream
Player
player
Real time
Real
Time
stream list
Stream List
Annotated
Annotation
/WB
Stream
Player
player
Real time stream
player
e-Annotation
e -Annotation
Whiteboard
Whiteboard
Real Time
Player
41
ANABAS
Messaging
SAB Briefing November 16, 2005
42
NaradaBrokering 2003-2006











Messaging infrastructure for collaboration, peer-to-peer and Grids
Implements JMS and native high-performance protocols (message
transit time of 1 to 2 ms per hop)
Order-preserving message transport with QoS and security profiles
Support for different underlying transport such as TCP, UDP,
Multicast, RTP
SOAP message support and WS-Eventing, WS-RM and WS-Reliability.
• WS-Notification when specification agreed
Active replay support: Pause and Replay live streams.
Stream Linkage: can link permanently multiple streams – using in
annotation of real-time video streams
Replicated storage support for fault tolerance and resiliency to storage
failures.
Management: HPSearch Scripting Interface to streams and brokers
(uses WS-Management)
Broker Topics and Message Discovery: Locate appropriate
Integration with Axis2 Web Service Container (?)
43
High Performance Transport supporting SOAP Infoset