Transcript Services - About DSC - Indiana University Bloomington

Global Multimedia
Collaboration System
Wenjun Wu
Indiana University Bloomington IN 47401
[email protected]
http://www.globalmmcs.org
Outline


Service-Oriented Collaboration
Current main stream real-time collaboration
technologies
• Videoconferencing: H.323, SIP, Access Grid
• Instant messaging & VoIP: MSN/Aol/Yahoo, Jabber, Skype

Global-MMCS introduction
Service Oriented Collaboration

Collaboration has
a) Mechanism to set up members (people, devices) of a
“collaborative sessions”
b) Shared generic tools such as text chat, white boards, audiovideo conferencing
c) Shared applications such as Web Pages, PowerPoint,
Visualization, maps, (medical) instruments ….



b) and c) are “just shared objects” where objects
could be regarded as Web Services
We can port objects to Web Services and build a
general approach for making Web services
collaborative
a) is a “Session Service” which is set up in many
different ways
Shared Event in Collaboration

All collaboration is about sharing events defining state changes
• Audio/Video conferencing shares events specifying in
compressed form audio or video
• Shared display shares events corresponding to change in
pixels of a frame buffer
• Instant Messengers share updates to text message streams
• Microsoft events for shared PowerPoint (file replicated
between clients) as in Access Grid

Using Web services allows one to expose update events of all
kinds as message streams
Need publish/subscribe approach to share messages

Session Service in Collaboration


Membership: Participant list
Role & floor assignment
• Management policy based on shared objects ( audio/video,
text, whiteboard, game


Ad-hoc or formally schedule
Session is also a “shared meta object” associated with
shared objects
• It needs eventing to keep consistent state as well
Multimedia Streaming Service



Streaming-In  Stream-Out filter
process media “events” between a stream source
and stream sink; and can be shared
composite media service is a DAG ( directed
acyclic graph)
common case ~ a filter chain
QoS(in)  QoS(out) ( bandwidth, delay,
jitter,loss)
for example: video/audio mixing, transcoding,
video-audio synchronization
VoIP & Videoconferencing
technology
H.323 Introduction
Major audio-video standard but broader
 “Binary” format for both “data” and “control”
 Supported by many commercial vendors and used
throughout the world in commercial and educational
markets
 Supports small-scale multipoint conferences
 Has conference management functionality and the call
signaling functionality
H.225 ~ call set-up
H.245 ~ call control
H.243 ~ Audio/Video multipoint control
T.120 ~ Data Collaboration

H.323 Protocols




H.323 is a “framework” document that describes
how the various pieces fit together
H.225.0 defines the call signaling and
communication between endpoints (Call Signaling)
and the Gatekeeper (RAS)
Annex G/H.225.0 defines communication between
Border Elements
H.245/H.243 is the conference control protocol
T.120 is the data conference protocol
Typical H.323 Stack
H.323
Multimedia Applications, User Interface
Data
Applications
V.150
T.120
Media Control
Audio
Video
Codecs
Codecs
G.711
H.261
G.723.1
H.263
G.729
H.264
..
..
T.38
Terminal Control and Management
H.225.0
RTCP
Call
H.245
Signaling
H.225.0
RAS
RTP
UDP
TCP
TCP/UDP
UDP
IP
http://www.packetizer.com
TCP/UDP
TCP
UDP
H.323 Architecture
Gatekeeper
(security, QoS,
routing etc.)
MC MP
MCU
Packet Switch
Network
H.323
Terminal 1
H.323
Terminal 2
....
H.323
Terminal N
SIP


Initially SIP was designed to solve problems for IP
telephony.
SIP basic functions
• user location resolution,
• capability negotiation
• call management.
equivalent to the service H.225 and point to point part of H.245

The major difference from H.323
• SIP was designed in a text format and took request-response
protocol style like HTTP
• SIP doesn’t define the conference control procedures like
multipoint parts of H.245 and T.120.
A Integrated SIP
Service System: CINEMA
From Columbia University
Desktop
SIP clients
H.323
SIPUA
sip323
sipd
Gatekeeper
SIP-H.323
signaling gateway
SIP
Programmable
SIP servers
Conferencing
Hardware SIP phone
sipconf
Unified messaging
sipum
sipgw
SIP-MGCP gateway
rtspd
MGCP
Streaming media
SIP-PSTN gateway
PSTN
Quick-time
Regular telephones
RTSP
Sipconf : SIP based Centralized
conferencing


SIP323
sipc





SIP based conferencing server
SIP/SDP and RTP/RTCP
Audio mixing
Play-out delay algorithm
Web based conference setup
G.711 A and Mu law, G.721, DVI
ADPCM
Multiple simultaneous conferences
SIP/PSTN
http://www.cs.columbia.edu/~kns10/software/sipconf
Summary of H.323/SIP
Conferencing Systems





Most products are Centralized conferencing system
MCU integrates the service of media processing service and
session management
Call-based
A conference call represents control connections between clients
and MCUs.
MCU is just a endpoint connected to VoIP softswitch cloud
Most vendors offer hardware solutions
Thought as services and controllers but specialized protocols and
implementations; NOT Service-Oriented Architectures!
Only support small size or medium size meetings ( < 20 sites )
Access Grid
Access Grid : a large scale audio/videoconference based on
a multicast network
 provides the group-to-group collaborations among 150
nodes connected to Internet 2 world wide.
 Use improved MBONE audiovisual tools VIC and RAT
 Depends upon high-speed network ( each node needs
20Mbps )
 Peer to peer architecture for distribution with centralized
non standard session control (venue server)
 Did not develop many new capabilities but made existing
public domain software better packaged and easier to use
Access Grid II

Supports multiple screens and dominates some
research communities
Instant Messaging & VoIP
Proprietary IM

MSN/Yahoo/AOL
• Ah-hoc small-group collaboration
• Text, audio-video, gaming and others….
• Remote Presence Service
( typical publish/subscribe messaging application )
• Massive “chat servers” running behind to support millions of
users across the world
• Limited size of buddy list and multi-party meeting
• Poor/fair quality for audio/video communication
• Close protocol which is unfriendly to third-party developers
Skype

Skype: p2p IM&VoIP solution
gained a big success.
• improving sound quality ( use new iLBC audio codec ) from
Global Sound
• Uses a variant of IETF Stun to identify NAT and firewall
• using p2p overlay (Kazaa) rather than expensive, centralized
infrastructure.
• provided supplemental features like instant messaging service.
• Free on-net VoIP service and a fee-based off-net SkypeOut
service that allows calling to PSTN and cellular phones
• Millions of download and on-line users in the world
Why is Skype so successful?




Better voice quality
excellent audio codec, fancy echo cancellation algorithm
Global IP Sound ( iLBC audio codec )
Ability to work behind firewalls and NAT
Ease of use ( quite simple UI ) based on IM metaphor
P2P style without centralized MCU
any peer that has enough resource can be selected to host
the mixing service
limited the number of participants in a conference ( at most
4 which is common for private social meetings )
use p2p overlay to discover resources and route packets
But they are simply not good enough!
Although all of these systems have advantages, they are not
sufficient for building more advanced and integrated
collaboration systems:

SIP :
had a huge development recently, especially in wireless world
very limited supported for conference control

H.323 :
AV collaboration and T.120 are not well integrated.
the AV communication services and T.120 overlay networks don’t have very
good scalability.
H.323 and T.120 are designed in a relative complicated OSI model. It is not easy
to understand and develop in their APIs
Most H.323 and SIP conferencing products are based on centralized MCU
And no way to take full use of private MCU resources
( Imagine how to use these private MCUs to create a meeting that have
thousands of participants )
But they are simply not good enough!
• Access Grid
heavily depends on multicast service and limited number of
uni-cast bridge servers in the Internet 2
No way to be deployed in current Internet
• Skype : Most promising
use its own propriety protocols and can’t interoperate with
other legacy VoIP clients such as H.323 and SIP
only support small-scale audio conferencing ( at most 5party ) and have no video service
Skype-2 is said to be able to support 10-party in dual-core
Intel machines
Above all, no system can support medium / large size
meetings in current Internet and adapt different client
What’s the ideal videoconferencing system I

A unified, scalable, robust “overlay” network is needed to
support AV and data group communication over heterogeneous
networking environments
• go through firewall and NAT
• provide group communication service in whatever unicast and multicast
networks
• offer reliable data delivery in whatever loss network
• to be configured as P2P or distributed server-based overlay to provide
differential services for VIP and regular users

A service-oriented architecture for hosting media processing
service and session control service
• More scalable than centralized MCU
• Support various style of conferencing ( massive scale of broadcasting as well
as medium size of private social meetings )
• Service providers can be highly distributed and p2p
~ Skype p2p audio mixing
• Scalable service discovery based on p2p search
• Customized media filters for different clients ( PC, PDA, … )
What’s the ideal videoconferencing system II

A core conference control mechanism is required for
establishing and managing the multi-point conference
• Complete conference control service like T.124 (Generic
Conference Control) in T.120 framework
• more flexible facilities to describe application sessions and
entities ( role-based, XML ) for all kinds of collaboration:
audio/video, game, whiteboard
• Session border management
Integrate different AV sessions ( H.323 , SIP, Access Grid,
RealStreaming … )
Simply regard these bridging gateways as “add-on services”
Global-MMCS Service Architecture
Audiovisual
Collaboration
Media Processing Service
( Adaptation, Mixing,
Transcoding ... )
Shared Data
Application
Instant
Messaging
Session Management Service
( Membership, Role
Management, Floor control)
Media Delivery, Storage Services
( QoS : Reliable and Secure Delivery,Transport Mechanism,
Massive Dependable Storage )
XGSP Web Service MCU Architecture
Use Multiple Media servers to scale to many codecs and many
versions of audio/video mixing
Session Server
XGSP-based Control
NaradaBrokering
All Messaging
NB Scales as
distributed
Admire
Web
Services
SIP
H323
Media Servers
Filters
High Performance (RTP)
and XML/SOAP and ..
Access Grid
Gateways convert to uniform XGSP Messaging
NaradaBrokering
Native XGSP
Break up into “Services”

Monolithic MCU becomes many different “Simple Services”
•
•
•
•
•
•
•
•



Session Control
Thumbnail “image” grabber
Audio Mixer
Video Mixer
Codec Conversion
Helix Real Streaming
PDA Conversion
H323/SIP Session/Signaling Gateways
As independent can replicate particular services as needed
• Codec conversion might require 20 services for 20 streams
spread over 5 machines
1000 simultaneous users could require:
• 1 session controller, 1 audio mixer, 10 video mixers, 20 codec
converters, 2 PDA converters and 20 NaradaBrokers
Support with a stream optimized Grid Farm in the sky
• Future billion way “Video over IP” serving 3G Phones and home media
centers/TV’s could require a lot of computing
GlobalMMCS and NaradaBrokering








All communication – both control and “binary” codecs are
handled by NaradaBrokering
Control uses SOAP and codecs use RTP transport
Each stream is regarded as a “topic” for NB
Each RTP packet from this stream is regarded as an “event” for
this topic
Can use replay and persistency support in NB to support
archiving and late clients
Can build customized stream management to administer replay,
and who gets what stream in what codec
NaradaBrokering supports unicast and multicast
Use firewall penetration and network monitoring services in NB
to improve Q0S
NaradaBrokering
Queues
Stream
NB role for Grid is
Similar to
NB supports messages
and streams
Incorporating Support for
Audio/Video Delivery into
NaradaBrokering





Added support for an unreliable transport protocol,
UDP
Implemented a fixed size (fast) topic (8 bytes).
Designed a new compact event with minimum headers.
Added support for legacy RTP clients (both unicast
clients and multicast groups)
Improved the routing algorithm to handle real-time
audio and video stream delivery.
XGSP Conference Control Architecture
Conference
Calendar
Conference Manager
Application
Instance 0
Application
Registry
User
Accounts
App Sessions
Narada
Brokering
User 5
Node
Manager
Application
Instance 1
User 1
User 2
User 3
XML based General Session Protocol
The XGSP conference control includes three services:

Conference management
supports user sign-in, user create/terminate/join/leave/inviteinto XGSP conferences
conference calendar service

Application session management
provides users with the service for creating/terminating
application sessions, managing session related services such as
audio/video mixing

Floor control
manages the access to shared collaboration resources in
different application sessions
for example, in a large scale of meetings having thousands of people, only
limited people are allowed to become presenters so that they can send
audio/video
Global-MMCS Community Grid




This includes an open source protocol independent Web Service
“MCU” which will scale to an arbitrary number of users and
provides support for thousands of simultaneous users of
collaboration services.
The function of A/V media server is distributed using
NaradaBrokering architecture.
• Media Servers mix and convert A/V streams
Open XGSP MCU based on the following open source projects
• openh323 is basis of H323 Gateway
• NIST SIP stack is basis of SIP Gateway
• NaradaBrokering is open source messaging
• Java Media Framework basis of Media Servers
• Helix Community http://www.helixcommunity.org for Real
Media
http://www.globalmmcs.org open source release
Audio/Video Meeting Tests
for single broker
first user
last user
Average
first
last
average
40
120
35
Average Latency in ms
Average latency in ms
100
30
25
20
15
10
80
60
40
20
5
0
0
0
500
1000
1500
number of participants
Audio Meeting Tests
2000
0
500
1000
number of participant
Video Meeting Tests
1500
Distributed Brokers Tests
Linux Cluster 2
• Going through multiple
brokers does not introduce
considerable overhead.
• Scalability of the system can
be increased almost linearly by
adding new brokers
Linux Cluster 1
Machine 1
Broker 4
Video
Receivers
Broker 3
Video
Receivers
Broker 2
Video
Receivers
Video
Transmitter
Broker 1
Video
Receivers
180
Avrg. Latency in ms
Measuring
Receivers
150
120
broker1
broker2
90
broker3
broker4
60
30
0
0
200
400
600
Number of receivers
800
1000
Average Latencies for Video Conferencing Clients at different locations.
Sites in Indiana, Florida, New York and Cardiff
100
10
1
Indiana
New York
Florida
Cardiff UK
0
20
40
60
80
100
120
Number of Users per Site
140
160
Analysis of the broker network’s performance





Test results showed that the broker network can scale well for
both single large size meetings and multiple smaller size meetings.
In large size meetings, the capacity of the broker network is
increased with respect to the capacity of the added brokers.
In multiple smaller size meetings, the distribution of users
among brokers are important. Inter-broker stream exchange can
reduce the scalability. Few users should not be scattered around
the broker network.
In wide area networks, this videoconferencing system provides
many benefits with distributed broker architecture: bandwidth
savings, latency savings, and better quality services.
In summary, thousands of concurrent users can easily be
supported in distributed broker settings.
media services computation overhead
Media Services
Computation Overhead
Audio Mixing
46% while 20 audio mixers ( six active speakers )
are running
Video Mixing
94% while 4 video mixers ( 4-way mixing ) are
running
Image Grabber
70% while 50 image grabbers are running
Real Streaming
Producer
90% while 4, 23fps stream producers are running
Improved JMF Performance
VIC
Old JMF Client
Fast JMF Client
1
8% - 9%
15% - 16 %
6% - 7%
2
13% - 14%
24% - 25 %
9% - 10%
3
17% - 18%
33% - 34 %
15% - 16%
4
23% - 24%
40% - 41%
17% - 18%
5
26% - 27%
46% - 47%
23% -24%
6
32% - 33%
51% - 52%
27% - 28%
7
35% - 36%
58% - 59%
31% - 32%
8
40% - 41%
62% - 63%
34% - 35%
Fraction of CPU used versus number of received streams
The CIF-size video sequence from a 30-second movie with a lot
of motions is streamed to the clients. Each stream is encoded in
H.261, and has average bandwidth of 400Kbps and 20 fps.
Global-mmcs user interface
Polycom, Access Grid
and RealVideo views of
video-mixed streams
using GlobalMMCS
MPEG-4 vs. H.261


We added MPEG4 video to Java Media Framework
Higher quality and flexible video sizes including
distributed pixels
Coupled Diverse Streams



GlobalMMCS supports many diverse streams managed by “video
system”
• Different audio and video codecs
• Shared display using video codecs (MPEG4 or H261)
• Motion JPEG – stream of images to and from PDA
NaradaBrokering represents these and other collaborative
streams just a “topics”; collaboration from multiple clients
subscribing to a topic
• Text Chat
• Traditional lossless codec based shared display
• White boards
• Control streams
Streams can be linked to provide composite topics
• eSports project linking video streams and real time annotation
of any frame
• Can rewind and choose any frame of a real-time stream
eSports Snapshot
Master
Video
Annotation
Whiteboard
Collaborative
Video
Annotation
Whiteboard
Synchronized Replay of archived
video and annotation
Esports with sport movies
Integration of PDA, Cell phone and Desktop Grid Access
NB Support for optimized
PDA Communication
PDA Download video as images
GlobalMMCS Status/Futures I


1. New Collaboration tools
• Shared IDL (Visualization), PowerPoint, OpenOffice (Applications
need a month or so more)
• SVG game ( stable )
• Whiteboard ( stable )
• e-Sport ( prototype)
• Jabber IM client ( prototype)
• XGSP needs extension to support
2. JMF Audio/Video client ( stable)
• performance enhancement finished
• new codec ( MPEG4-DivX finished; try MPEG4-Xvid and H.264)
• support different platform ( Linux, Mac – Mac well developed but
need to chase bug(s) )
• support NAT/firewall transparently like Skype
• Google Desktop PlugIn (under development )
GlobalMMCS Status/Futures II

3. Replay & Archive (prototype)
• Replay Engine based on NaradaBroker Storage Service
• XGSP-RTSP gateway
• Extend RTSP and NaradaBrokering for Instant Replay

4. Web Server Portal ( stable)
• Standard calendar service ( iCalendar, vCalendar)
• Flexible conference management
• Need to package UI’s as portlets

5. Conferencing Media Processing Service ( Stable)
• Support new codec (H.264 )
• Add DCT domain MPEG4-H.261 transcoder

6. H.323 Gateway ( Stable)
• Import it to Linux platform
GlobalMMCS Status/Futures III





7. RealStreaming Gateway ( Stable )
• Import it to Linux
• Support Mobile clients
8. Global-MMCS deployment & test
• Core performance measurements complete
• Test under the setting of multiple NaradaBroker and
NAT/Firewall
• support deployment for AFRL, NASA, DOE portals
• test with remote sites
9. SmartPhone Clients (prototype)
10. Improved video codec-based shared display
11. Scheduler of dynamic services sensitive to streaming
bandwidth requirement as well as CPU use of codec
conversion