Transcript Worldwide Messaging Support for High Performance Real

Worldwide Messaging
Support for High
Performance
Real-time Collaboration
Pete Burnap, Hasan Bulut, Shrideep
Pallickara, Geoffrey Fox, David Walker, Ali
Kaplan, Beytullah Yildiz, and Mehmet A. Nacar
Community Grids Lab, Indiana University
School of Computer Science, Cardiff University, UK
What is NaradaBrokering?





Distributed messaging infrastructure
Provides support for P2P, Centralised and Distributed
interaction
Efficiently routes messages between originator and
registered consumers
OBJECTIVE - To provide a recognised open-source
standard for a messaging infrastructure to build SOAs
and Grids upon
It is hoped that these results will influence others to use
NaradaBrokering as a messaging protocol for their own
applications
Motivation




Build distributed systems from “interoperable” services linked by messages
(SOAP) – architect capabilities as services
Grids are “just” large scale sets of such services
Need to support real time streams and NOT just files (collections of messages)
consistent with WS standards (P2P and “central”)
Manage messaging for








Open Source http://www.naradabrokering.org (4 downloads/day) is a scalable
distributed pub-sub system supporting multiple standards (JMS, WS) and
subscription methods


Optimize communication for bad links, firewalls etc
Collaboration (multi-cast streams)
Fault tolerance with re-transmitted messages and Replicated Services
Replay – access any message at any time
Virtual addressing with pub-sub metaphor
Performance from protocol (UDP v Parallel TCP ..) and representation
Heterogeneous Clients – filter to and from PDA’s
Implements “Service Internet” and Notification areas of WS-*
Infrastructure
Candidate for Axis2-MOM (Message Oriented Middleware) infrastructure
Architecture
NaradaBrokering Features
Multiple protocol transport
support
In publish-subscribe
Paradigm with different
Protocols on each link
Transport protocols supported include TCP, Parallel TCP streams, UDP,
Multicast, SSL, HTTP and HTTPS.
Communications through authenticating proxies/firewalls & NATs.
Network QoS based Routing
Allows Highest performance transport
Subscription Formats
Subscription can be Strings, Integers, XPath queries, Regular
Expressions, SQL and tag=value pairs.
Reliable delivery
Robust and exactly-once delivery in presence of failures
Ordered delivery
Producer Order and Total Order over a message type. Time
Ordered delivery using Grid-wide NTP based absolute time
Recovery and Replay
Recovery from failures and disconnects.
Replay of events/messages at any time. Buffering services.
Security
Message-level WS-Security compatible security
Message Payload options
Compression and Decompression of payloads
Fragmentation and Coalescing of payloads
Messaging Related
Compliance
Java Message Service (JMS) 1.0.2b compliant
Support for routing P2P JXTA interactions.
Grid Feature Support
NaradaBrokering enhanced Grid-FTP. Bridge to Globus GT3.
Web Services supported
Implementations of WS-ReliableMessaging, WS-Reliability and
WS-Eventing.
NaradaBrokering Services
Buffering Service


This service is incorporated into the system to
facilitate the buffering of events prior to releasing
them.
Buffering service time orders events and releases
event based on three metrics
 Number of events in the buffer
 Size of the buffer
 Time spent by event in a buffer.
Time Differential Service



This service is essential to reduce jitters in large
distributed environments.
 Networks introduce unpredictable delays that
increase jitter.
Service takes events released by buffering
service, and ensures that it preserves time
spacing between events.
TDS can provide time spacing resolution of up to
1 millisecond between events.
Reliable Delivery Service






Guaranteed delivery in multiple producer/ consumer
settings. Guarantees hold true in the presence of
 Node/Link Failures
 Links can lose messages and garble message order.
 Storage failures: Stores need to recover after
failure.
 Prolonged entity disconnects
Exactly-Once and Ordered delivery of events
Uses both positive & negative acknowledgements
Supports Replay and Fast Recovery from failures
Independent of underlying archival system.
Was used to enhance fault tolerance in Grid-FTP.
Dealing with large payload sizes



To cope with large payloads, the substrate incorporate
2 sets of services.
Compression/Decompression service: The substrate
incorporate support for zlib based compression and
decompression of payloads.
Fragmentation/Coalescing Service: These service can
break up a large payload into smaller fragments. The
coalescing service can take these smaller fragments
and coalesce them into the original large payload.
 This was used to deal with transfer of large
payloads (up to 1 GB) in the NB enhanced Grid-FTP
application.
Broker Discovery Service

Locates the nearest available broker that a client can
connect to
 Incorporates specialized nodes – broker discovery
nodes – to maintain broker info.
 Depending on load or security issues, brokers may
decide to respond/ignore discovery requests.
 If available the scheme can exploit IP multicast for
discovery.
 Nearest broker determined by ping times, loss rates
and available bandwidth.
Topic Discovery Service




Allows publishers and subscribers to advertise
topics.
Creator of topic possesses credentials to
indicate ownership of the topic.
Discovery of topics takes into account
credentials of client trying to discover topic.
 Topic owner may restrict discovery to a
limited authorized set of clients.
Discovery requests can be made using simple
strings or regular expression queries.
Security Service
[ Based on Message Level Security ]
1
Request permission to publish
2
Respond back with topic
key if authorized to publish
3
Encrypt message with topic key
Compute Message Digest(MD)
Sign MD and message ID
Publish Message
4
Verify Signature & Permissions
Check integrity by verifying MD
Check ID for replay attacks
5
Request permission to subscribe
6
Respond back with topic key if
authorized to subscribe
6
5
7
8
Key
Management
Center (KMC)
NaradaBrokering Broker
Cloud
4
1
2
3
Broker Node
7
Create subscription request
Compute Message Digest
Sign MD and message ID
Issue Subscription request Message
8
Verify Signature
Verify Permissions for Subscribing
Check integrity by verifying MD
Check ID for replay attacks
Entity (Publisher or Subscriber)
SSL encrypted
communications
• Messages organized into topics
• Each topic has a separate key; Topics can be organized into sessions
Web Services Support


Currently we have incorporated support for the following Web
Service specifications
 WS-Eventing (WSE): This is a publish/subscribe based
notification framework from Microsoft and IBM.
 WS-ReliableMessaging (WSRM): This is a protocol for
ensuring the guaranteed delivery of SOAP messages
between 2 Web Service endpoints. This specification is
from IBM and Microsoft.
 WS-Reliability (WSR): This is a competing specification
from Oracle and Sun in the area of reliable messaging
between Web Services.
These handlers are available for use in Axis1.2 or exploiting
NB SOAP Intermediary support without a container
 Axis1.2 version can be used inside container or as a Proxy
Web Services Support – cont..

We are also working on implementing support for the
WS-Notification (WSN) suite of specification that is part
of the Web Services Resource Framework (WSRF).

WS-Notification explicitly adds brokers to Eventing

Note that almost all these specifications leverage the
WS-Addressing (WSA) specification.
 We have incorporated support for all the rules
associated with WSA.
Parallel TCP Streaming



Data transfer across high-performance networks suffers
within current TCP implementation due to tendency to
hold back maximum bandwidth usage
Can be solved either by using TCP multiple streams in
parallel
TCP streams consists of three basic steps:



splitting of data into sub packets at sender side
sending these sub packets over the network by using multiple
Java socket streams in parallel
merging of received sub packets at the receiver side.
TCP Test Observations
• This illustrates the capability of NaradaBrokering to maintain a high speed
message delivery performance despite the scaling of message content
payload.
• Mean transmission delay times start to rise as the payload increase becomes
increasingly steep between 1000 and 10000 bytes
TCP Streaming Results
Continental WAN Test
Bandwidth (mbps)
Bandwidth (mbps)
LAN Test
TCP Streaming Results cont…
Inter-Continental WAN Test
Future of NaradaBrokering








Support for replicated storages within the system.
 In a system with N replicas the scheme can sustain the loss of N1 replicas.
Clarification and expansion of NB Broker to act as a WS container
Integration with Axis 2.0 as Message Oriented Middleware
infrastructure
Support for High Performance transport and representation for Web
Services
 Needs Context catalog under development
Performance based routing
 The broker network will dynamically respond to changes in the
network based on metrics gathered at individual broker nodes.
Replicated publishers for fault tolerance
Pure client P2P implementation (originally we linked to JXTA)
Security Enhancements for fine-grain topic authorization, multi-cast
keys, Broker attacks
Further Information



Web - http://www.naradabrokering.org/
Geoffrey Fox - [email protected]
Shrideep Pallickara - [email protected]