Transcript PPT

CS 414 – Multimedia Systems Design
Lecture 25 –
Synchronization Issues
Klara Nahrstedt
Spring 2012
CS 414 - Spring 2012
Administrative


MP2 posted
MP2 Deadline – April 7, Saturday, 5pm.
CS 414 - Spring 2012
Internet Multimedia Protocol Stack
APPLICATION
DASH
Media encaps
(H.264, MPEG-4)
Synchronization Service
SIP
RTSP
RSVP
RTP
HTTP
KERNEL
TCP
DCCP
UDP
IP Version 4, IP Version 6
AAL3/4
Layer 5
(Session)
RTCP
AAL5
MPLS
CS 414 - Spring 2012
Layer 3
(Network)
Layer 2
(Link/MAC)
Ethernet/WiFi
ATM/Fiber Optics
Layer 4
(Transport)
Outline
Synchronization in Distributed
Environments
 Clock Synchronization
 Synchronization Reference Model

 Media,
Stream, Object, Specification Levels
CS 414 - Spring 2012
Synchronization in Distributed Environments


Information of synchronization must be
transmitted with audio and video streams, so
that receiver(s) can synchronize streams
Sync information can be delivered before start of
presentation (used by synthetic synchronization)



Advantage: simple implementation
Disadvantage: presentation delay
Sync information can be delivered using
separate sync channel - out-band (used by live
synchronization)


Advantage: no additional presentation delay
Disadvantage: additional channel needed
CS 414 - Spring 2012
Sync in Distributed Environments

Sync information can be delivered using
multiplexed data streams - in-band sync
 Advantage:
related sync information is
delivered together with media units
 Disadvantage: difficult to use for multiple
sources
CS 414 - Spring 2012
Location of Sync Operation
Sync media objects by combining objects
into new media object
 Sync operation placed at sink

 Demand
on bandwidth is larger because
additional sync operations must be
transported

Sync operation placed at source
 Demand
on bandwidth smaller because
streams are multiplexed according to sync
requirements CS 414 - Spring 2012
Clock Synchronization

Sync accuracy depends on
clocks at source and sink
nodes
 Ta = Tav – Nla – Oa
 Tv = Tav – Nlv – Ov

End-to-end delay
 Nla
= EEDa = Tav-Ta-Oa
 Nlv=EEDv = Tav-Tv-Ov
 EEDa = (Ta1-Ta2)/2

NTP (Network Time
Protocol )
CS 414 - Spring 2012
AV Skew Calculation
Select synchronization point in AV streams in
terms of frame number for each stream
 Execute sync operation at sync points
1. Measure

 Arrival
time of sync audio packet TsyncA
 Arrival time of sync video packet TsyncV
2. Calculate:
 AVSkew = TsyncA – TsyncV
3. Alternative measure and calculate
 TsyncA
= TsendA + EEDA; TsyncV = TsendV + EEDV
 EED = RTT/2 = (Trecv – Tsend)/2
CS 414 - Spring 2012
Network Time Protocol

Protocol to sync clocks of computer
systems over packet-switched, variable –
latency data networks
 Uses
UDP port 123
 Designed to resist effects of variable latency (jitter
buffer)
 Designed in 1985 by Dave Mills at U. Delaware
 Can achieve accuracy of 200 µsec
 Based on Marzullo Algorithm
CS 414 - Spring 2012
Marzullo’s algorithm (1984)
(intersection algorithm)




Agreement protocol for
estimating accurate time
from a number of noisy
time sources
If we have estimates
10  2, 12  1, 11 1,
then interval intersection
is 11.5  0.5
If some intervals don’t
intersect, consider
intersection of majority of
intervals
Source: wikipedia
CS 414 - Spring 2012
Clock strata


NTP uses hierarchical system of
“clock strata”
Stratum levels define distance
from reference clock and exist to
prevent cycles in hierarchy

Stratum 0


Stratum1



devices are atomic clocks, GPS clocks,
radio clocks
computers attached to stratum0 devices
Act as servers for timing requests from
Stratum 2 servers via NTP
Stratum2

(similar to Stratum1, but they also have
peering relation to other stratum2
servers
CS 414 - Spring 2012
Precision Time Protocol (PTP)




High Precision time protocol (original IEEE 1588-2002)
for synchronization used in measurement and control
systems residing on a local area network
Accuracy in the sub-microseconds
PTPv2 – IEEE 1588-2008 improves accuracy, precision
and robustness
PTP serves a niche not well served by NTP and GPS
 it achieves accuracy beyond those attainable using
NTP;
 It is cheaper than GPS
 It is accessible to receivers for which GPS signals are
inaccessible.
CS 414 - Spring 2012
PTP Details

IEEE 1588 standards describe a hierarchical masterslave architecture for clock distribution




Ordinary clock – source for root timing reference, called
‘grandmaster’
Boundary clock – intermediate synchronization bridge from one
network segment to another – used by synchronization master to
sync clocks within a network segment
PTP messages use IP-multicast over UDP (IPv4 or
IPv6)
Selection algorithm of best clock source uses properties
such as quality, accuracy, variance, …
CS 414 - Spring 2012
PTP Synchronization

After selection of master source of time,
clocks determine the offset between
themselves and their master
 Let
t represent physical time, given slave device, the
offset o(t) at time t is: o(t) = s(t) – m(t),


m(t) is measured time at master at time t, and
s(t) is measured time at local clock at time t.
 Master
periodically broadcasts the current
time to other clocks (up to 10 measurements
per second in 1588-2008)
CS 414 - Spring 2012
PTP Synchronization
T1’ – T1 = o + d and T2’ – T2 = -o + d, then we find
o = (T1’ - T1 – T2’ + T2)/2
The clock now knows offset o during the transactions and can correct
itself by this agreement with the master clock.
CS 414 - Spring 2012
Other Sync Issues


Sync must be considered during object acquisition
Sync must be considered during retrieval
 Sync

Sync must be considered during transport
 If

possible use isochronous protocols
Sync must be considered at sink
 Sync

access to frames of stored video
delivery to output devices
Sync must consider support of functions such as
pause, forward, rewind with different speeds, direct
access, stop or repeat
CS 414 - Spring 2012
Other Sync Issues


Sync must be considered during object acquisition
Sync must be considered during retrieval
 Sync

Sync must be considered during transport
 If

possible use isochronous protocols
Sync must be considered at sink
 Sync

access to frames of stored video
delivery to output devices
Sync must consider support of functions such as
pause, forward, rewind with different speeds, direct
access, stop or repeat
CS 414 - Spring 2012
Reference Models

We need reference models to
 Understand
various requirements for
multimedia sync
 Identify and structure run-time mechanisms to
support execution of sync
 Identify interface between run-time
mechanisms
 Compare system solutions for multimedia
sync
CS 414 - Spring 2012
Synchronization Reference
Model

Sync model we will be evaluating in detail
is according to Meyer, Effelsberg,
Steinmetz:
 Sync
multimedia objects are classified
according to
Media level
 Stream level
 Object level
 Specification level

CS 414 - Spring 2012
Media Level (1)
Each application operates single
continuous media streams composed of
sequence of LDUs
 Each media stream responsible for intrastream synchronization
 Assumption at this level: device
independence
 Supported operations at this level:

 read(devicehandle,
LDU)
 write(devicehandle,
CS 414LDU)
- Spring 2012
Media Level (2) - Example
window = open(“videodevice”);
movie = open(“file”);
while (not EOF (movie) ) {
read(movie, &LDU);
if (LDU.time == 20)
printf(“Subtitle 1”);
else if (LDU.time == 26)
printf(“Subtitle2”);
write(window, LDU); }
close(window);
close(movie);
CS 414 - Spring 2012
Stream Level (1)
Operates on continuous media streams
and groups of streams
 Models inter-stream synchronization for
need of parallel presentation
 Offers abstractions:

 notion
of streams,
 timing parameters concerning QoS for intra-stream
and inter-stream synchronization
CS 414 - Spring 2012
Stream Level (2)

Supports operations:
 Start(stream),
stop(stream), create-group(list-of-
streams);
 Start(group), stop(group);
 Setcuepoint(stream/group, at, event);

Classifies implementation according to
 Support
for distribution (end-to-end, local)
 Support of type of guarantees (best effort,
deterministic)
 Support of types of supported streams (analog,
digital)
CS 414 - Spring 2012
Object Level (1)


Operates on all types of media and hides
differences between discrete and continuous media
Offers abstractions:
 Complete


sync presentation
Computes and executes complete presentation
schedules that include presentation of noncontinuous media objects and calls to stream level
Does not handle intra-stream and inter-stream
synchronization
 (relies
on media and stream levels)
CS 414 - Spring 2012
Object Level (2) - Example

MHEG – Multimedia Hypermedia Experts Group of ISO


Defines representation and encoding of multimedia and hypermedia
objects (object-based declarative programming language)
Provides abstractions suited to real-time presentations


Provides abstracts for real-time exchange


implemented with minimal buffering
Evaluates status of objects and performs actions (e.g., prepare,
run, stop, destroy)



implemented via multimedia synchronization functionalities
For time-dependent streams – access to stream level
For time-independent streams – direct access the object to present it
Classification of this level according to (a) distribution
capabilities, (b) type of presentation schedule, (c) schedule
calculation
CS 414 - Spring 2012
Specification Level (1)


Open layer included in tools which allow to
create sync specifications
Examples:



Synchronization editors, document editors, authoring systems,
conversion tools
Examples of such tools: multimedia document formatter that
produces MHEG specifications
Classification:




Interval-based spec
Time-axes based spec
Control flow-based spec
Event-based spec
CS 414 - Spring 2012
Specification Level (2) - SMIL




SMIL – World Wide Web
Consortium’s Synchronized
Multimedia Integration
Language
SMIL 2.0 and SMIL 3.0
(developed by CWI, PI: D.
Bulterman)
Features: Animations, Time
Manipulations, Timing and
Synchronization, Content
Control, Meta-information,
Media objects, Linking, Layout,
Transitions
http://www.w3.org/TR/SMIL3/
CS 414 - Spring 2012
SMIL
“Standards, SMIL 2.0, Examples and Comparisons”, D. Bulterman, Siemens
Corporate research, Editor: P. Liu, 2002
CS 414 - Spring 2012
Summary
CS 414 - Spring 2012
Conclusion
Carefully analyze what kind of
synchronization is needed in your
multimedia system and application
 Determine at which level you need
synchronization
 Determine what the synchronization
requirements should be based on prior
experiments

CS 414 - Spring 2012