Transcript Multimedia - Yale "Zoo"

Multimedia Networking
R. Yang
1
Outline
 Admin. and review
 Introduction to multimedia networking
 An architecture of stored multimedia
 Network support of multimedia app.
 Application adaptation
2
Preview: Multimedia Networking
Our goals:
Schedule of the two classes:
 Understand the service
requirements of networked
multimedia applications
 Learn the high level
structures designed for
multimedia networking
 Deal with multimedia in a
best-effort network
 Class 1: Multimedia
applications; application
adaptation in best-effort
networks
 Class 2: New architectures:
IntServ and DiffServ
 How the Internet might
evolve to better support
multimedia?
3
MM Networking Applications
Classes of MM applications:
1) Streaming stored audio
and video
2) Streaming live audio and
video
3) Real-time interactive
audio and video
Fundamental
characteristics:
 Typically delay sensitive


end-to-end delay
delay jitter
 But loss tolerant:
infrequent losses cause
minor glitches
 Antithesis of data,
which are loss intolerant
but delay tolerant.
4
Streaming Stored Multimedia
Streaming:
 media stored at source
 transmitted to client
 streaming: client playout
begins before all data has
arrived
 timing constraint for still-to-be
transmitted data: in time for
playout
5
Streaming Stored Multimedia: Interactivity
 VCR-like functionality: client can
pause, rewind, FF, push slider bar
 10 sec initial delay OK
 1-2 sec until command effect OK
6
Streaming Live Multimedia
Examples:
 Internet radio talk show
 Live sporting event
Streaming
 play can lag tens of seconds
 still have timing constraint (but shorter delay
required)
Interactivity
 fast forward impossible
 rewind, pause possible!
7
Interactive, Real-Time Multimedia
 applications:
IP telephony, video conference,
distributed interactive worlds
 end-end delay requirements:
 audio:
< 150 msec good, < 400 msec OK
• includes application-level (packetization) and network
delays
• higher delays noticeable, impair interactivity
 session initialization

callee be able to advertise its IP address, port
number, encoding algorithms.
8
Outline
 Admin. and review
 A brief introduction to the physical
layer
 Introduction to multimedia networking
 An architecture of stored multimedia
 Network support of multimedia app.
 Application adaptation
9
Streaming from Web Server (1)
 Audio and video files stored





in Web servers
Browser requests file with
HTTP request message
Web server sends file in
HTTP response message
Content-type header line
indicates an audio/video
encoding
Browser launches media
player, and passes file to
media player
Media player renders file
• Major drawback: media player
interacts with server through
intermediary of a Web browser
10
Streaming from Web Server (2)
Alternative: set up connection
between server and player
 Web browser requests and
receives a meta file
(a file describing the
object) instead of
receiving the file itself
 Content-type header
indicates specific
audio/video application
 Browser launches media
player and passes it the
meta file
 Player sets up a TCP
connection with server and
sends HTTP request
Some concerns:
 Media player communicates
over HTTP, which is not
designed with pause, ff,
rewind commands
 May want to stream over
UDP
11
RTSP/RTCP/RTP
Web
browser
HTTP GET
presentation desc.
Web
server
RTSP
media
player
RTCP
media
server
RTP
client
server
RTSP: Real-Time Signaling Protocol
RTCP: Real-Time Control Protocol
RTP: Real-Time Transport Protocol
12
Start: Using HTTP to get Presentation File
Example presentation file
<title>Twister</title>
<session>
<group language=en lipsync>
<switch>
<track type=audio
e="PCMU/8000/1"
src = "rtsp://audio.example.com/twister/audio.en/lofi">
<track type=audio
e="DVI4/16000/2" pt="90 DVI4/8000/1"
src="rtsp://audio.example.com/twister/audio.en/hifi">
</switch>
<track type="video/jpeg"
src="rtsp://video.example.com/twister/video">
</group>
</session>
13
Control: RTSP, Out of Band Control
RTSP messages are sent out-of-band:
 The RTSP control messages use a different port number




(554) than the media stream, and are therefore sent out-ofband
RTSP can be sent over UDP or TCP
Each RTSP message can be sent over a separate TCP
connection
The media stream, whose packet structure is not defined by
RTSP, is considered “in-band”
If the RTSP messages were to use the same port number as
the media stream, then RTSP messages would be said to be
“interleaved” with the media stream
14
RTSP: Initialization and Control
Web
browser
HTTP GET
presentation desc.
Web
server
SETUP
PLAY
media
player
media stream (RTP)
media
server
RTCP
PAUSE
TEARDOWN
client
server
15
RTSP: Exchange Example
C: SETUP rtsp://audio.example.com/twister/audio RTSP/1.0
Transport: rtp/udp; compression; port=3056; mode=PLAY
S: RTSP/1.0 200 1 OK
Session 4231
C: PLAY rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0
Session: 4231
Range: npt=0C: PAUSE rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0
Session: 4231
Range: npt=37
C: TEARDOWN rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0
Session: 4231
S: 200 3 OK
16
Data: Real-Time Protocol (RTP)
 RTP runs in the end systems
 RTP specifies a packet structure for packets
carrying audio and video data: RFC 1889
 RTP packets are encapsulated in UDP segments
 Interoperability: If two Internet phone applications
run RTP, then they may be able to work together
17
RTP Header
8
16
32
32
- Payload Type
- e.g. type 0: (Pulse Coded Modulation) PCM mu-law, 64 Kbps; type
3, GSM, 13 Kbps; type 33, MPEG2 video
Question: why include payload type in each packet?
- Sequence Number
- SSRC field
- Identifies the source of the RTP stream. Each stream in a RTP session
should have a distinct SSRC
- Timestamp field
- Reflects the sampling instant of the first byte in the RTP data
packet
18
RTP Header (2)
8
16
32
32
 Timestamp field



Reflects the sampling instant of the first byte in the RTP data
packet
The timestamp is derived from a sampling clock at the sender
Example
• audio stream sampled every 125 usecs
• audio application generates chunks consisting of 160 encoded
samples
• then the timestamp increases by 160 for each RTP packet when the
source is active
• the timestamp clock continues to increase at a constant rate even
when the source is inactive
19
H.323 Overview
 Foundation for audio and video conferencing across
IP networks
 Targets real-time communication (rather than ondemand)
 Umbrella recommendation from the ITU
 Broad in scope:
 stand-alone devices (e.g., Web phones)
 applications in PCs
20
Overview
Gatekeeper
Internet
PSTN
Gateway
H.323 terminals
Telephone: SS7, Inband
21
H.323 terminals
H.232 Big Picture
Internet
Router
RAS
Gatekeeper
LAN = “Zone”
22
Gateway
PSTN
H.323 terminals
Gateway
Router
Internet
RAS
Gatekeeper
LAN = “Zone”
• Bridge between IP Zone and PSTN (or ISDN) network.
• Terminals communicate with gateways using H.245 and Q.931
23
H.323 terminals
Gatekeeper
Internet
Router
RAS
Gatekeeper
LAN = “Zone”
• The gatekeeper is optional. Can provides to terminals:
• address translation to IP addresses
• bandwidth management: can limit amount of bandwidth consumed by
real-time conferences
• Optionally, H.323 calls can be routed through gatekeeper. Useful for billing.
• RAS protocol (over TCP) for terminal-gatekeeper communication.
24
H.323 Terminal
25
H.323 Endpoints Must Support:
 G.711 - ITU standard
for speech compression
 RTP - protocol for
encapsulating media
chunks into packets
 H.245 - “Out-of-band”
control protocol for
controlling media
between H.323
endpoints.
 Q.931 - A signalling
protocol for
establishing and
terminating calls.
 RAS
(Registration/Admission
/Status) channel
protocol - Protocol for
communicating with a
gatekeeper (if
gatekeeper is present)
26
H.323 Encoding
Audio:
 H.323 endpoints must
support G.711 standard
for speech compression.
G.711 transmits voice at
56/64 kbps.
 H.323 is considering
requiring G.723 =
G.723.1, which operates
at 5.3/6.3 kbps.
 Optional: G.722, G.728,
G.729
Video
 Video capabilities for an
H.323 endpoint are optional.
 Any video-enabled H.323
endpoint must support the
QCIF H.261 (176x144
pixels).
 Optionally supports other
H.261 schemes: CIF, 4CIF
and 16CIF.
 H.261 is used with
communication channels that
are multiples of 64 kbps.
27
Generating Audio Packet Stream in H.323
Audio
Source
Encoding:
e.g., G.711
or G.723.1
RTP packet
encapsulation
UDP
socket
Internet or
Gatekeeper
28
H.245 Control Channel
 H.323 stream may
contain multiple
channels for different
media types.
 One H.245 control
channel per H.323
session.
 H.245 control channel is
a reliable (TCP) channel
that carries control
messages.
 Principle tasks:
Open and closing
media channels.
 Capability exchange:
before sending
media, endpoints
agree on encoding
algorithm
 Note: H.245 for
multimedia conferencing
is analogous with RTSP
for media streaming

29
Information flows
Call Control
Channel
Call Signaling
Channel
Media Control
Channel
H.323
Terminal
H.323
Terminal
RAS Channel
H.323
Gatekeeper
Media Channel
1
Media Channel
2
TCP
UDP
30
Gatekeeper 2/2
 H.323 terminal must
register itself with the
gatekeeper in its zone.
 When the H.323
application is invoked at
the terminal, the
terminal uses RAS to
send its IP address and
alias (provided by user)
to the gatekeeper.
 If gatekeeper is present in a
zone, each terminal in zone
must contact gatekeeper to
ask permission to make a
call.
 Once it has permission, the
terminal can send the
gatekeeper an e-mail
address, alias string or
phone extension. The
gatekeeper translates the
alias to an IP address.
 If necessary, a
gatekeeper will poll other
gatekeepers in other
zones to resolve an IP
address. Process varies
by vendor.
31
Data Control: Real-Time Control Protocol (RTCP)
 Works in conjunction with each real-time RTP
session
 Each participant in an RTP session periodically
transmits RTCP control packets to all other
participants, e.g.,


Receiver-report packets: fraction of packets lost, last
sequence number, average inter-arrival jitter
Sender-report packets: the number of packets sent, the
timestamp and the wall-time of the last sent packet
 RTCP attempts to limit its traffic to 5% of the
session bandwidth
32
Outline
 Admin. and review
 A brief introduction to the physical
layer
 Introduction to multimedia networking
 An architecture of stored multimedia
 Network support of multimedia app.
 Application adaptation in best-effort
Internet
33
Multimedia in Networks:
Fundamental Characteristics
 Many have requirements on rate, i.e. long-term average bandwidth
requirement (e.g. 13kbps phone conversion)
 Many are delay and jitter sensitive
 jitter is the variability of packet delays within the same packet
stream.
 Many are loss tolerant


infrequent losses cause minor glitches that can be concealed
but high loss rate makes the application useless
 Antithesis of data (programs, banking info, etc.), which are loss
intolerant but delay tolerant; multimedia is also called “continuous
media”
utility
rate
rate
34
Why Best-effort Internet May not Work Well?
 Variable bandwidth

dynamic bandwidth sharing
 Potentially long delays
 Assume simple arrival and service processes
(Poisson). Suppose the utilization of a link is  (<1),
the delay is
1
1 
 Variable delay jitters
35
Multimedia Networking:
Two Types of Approaches
Laissez-faire philosophy
 No reservations, no
change of the services
provided by IP
 Several tools



As demand increases,
provision more bandwidth
Application adaptation
Place stored content at
edge of network
Integrated/Differentiated
services philosophy:
 Extends the service
provided by the Internet
 Two approaches

IntServ
• New protocols and router
mechanisms so that each
flow can reserve end-toend bandwidth

DiffServ
• Datagrams are marked
• User pays more to
send/receive 1st class
packets
36
Outline
 Admin. and review
 A brief introduction to the physical layer
 Introduction to multimedia networking
 An architecture of stored multimedia
 Network support of multimedia app.
 Application adaptation in best-effort
Internet



dealing with variable delay
dealing with packet loss
dealing with variable bandwidth
37
Delay Distribution
In general, the distribution of delay is Bell-Shaped [MKT98]
38
Playout Buffer
 Basic idea: delay playing out packets to
accommodate delay jitter
packets
number
packets
generated
packets
received
1
0
delay
time
r
p
p'
 Discussion: For what types of applications will
playout buffer be the most effective?
39
Example: Adaptive Playout Delay Estimation
 TCP-like delay and timeout estimation; adjust playout delay at the
beginning of each check point
40
Outline
 Admin. and review
 A brief introduction to the physical
layer
 Introduction to multimedia networking
 An architecture of stored multimedia
 Application adaptation in best-effort
Internet



dealing with variable delay
dealing with packet loss
dealing with variable bandwidth
41
Forward Error Correction (FEC):
Piggyback A Lower Quality Stream on Next Packet
2
3
 Send a lower resolution audio stream as the
redundant information on the next packet,
e.g. nominal stream at 64 kbps and redundant
stream GSM at 13 kbps
42
Outline
 Admin. and review
 A brief introduction to the physical layer
 Introduction to multimedia networking
 An architecture of stored multimedia
 Application adaptation in best-effort
Internet



dealing with variable delay
dealing with packet loss
dealing with variable bandwidth
43
Basic Idea: Change the Rate of an MM Application
 Video and audio

can generate different encoding rates by
controlling the encoding process
 Question: how do you change the rate of a
multimedia stream?
(262 KB)
44
Video Encoding: Block Transform Encoding
DCT
Quantize
Zig-zag
011010001011101...
Run-length
Code
Huffman
Code
45
Discrete Cosine Transform
4C(u)C(v)
F[u,v] =
n2
n-1 n-1

f(j,k) cos
(2j+1)up
2n
j=0 k=0
where
C(w) =
1
2
1
cos
(2k+1)vp
2n
for w=0
for w=1,2,…,n-1
- Convert from pixel domain to frequency domain
- DCT better at reducing redundancy than Discrete Fourier
Transform but it is more computationally expensive
46
Basis Functions of DCT
47
Example: Block Encoding
DC component
139
144
150
159
144
151
155
161
149
153
160
162
153
156
163
160
DCT
1260 -1 -12
-23 -17 -6
-11 -9 -2
-7
-2
0
-5
-3
2
1
Quantize
original image
AC components
79 0 -2 -1 -1 -1 0 0 -1 0 0 0 0 0 0 0
run-length
code
0
1
0
0
0
2
0
79
-2
-1
-1
-1
-1
0
Huffman
code
zigzag
79
-2
-1
0
0
-1
-1
0
-1
0
0
0
0
0
0
0
10011011100011...
coded bitstream < 10 bits (0.55 bits/pixel)
48
Result of Coding/Decoding
139
144
150
159
144
151
155
161
149
153
160
162
144
156
155
160
153
156
163
160
146
150
156
161
149
152
157
161
152
154
158
162
reconstructed block
original block
-5
-4
-5
-1
-2
1
-1
0
0
1
3
1
1
2
5
-2
errors
49
Examples
Uncompressed
(262 KB)
Compressed
(22 KB, 12:1)
Compressed
(6 KB, 43:1)
50
Encode Rate According to Available Bandwidth
 Steps
 Estimate available bandwidth (how?)
 Encode the stream according to the estimation
of available bandwidth (how?)
51
Backup Slides
52
Layering
 Put the stream into multiple layers
 Each layer is a refinement of the previous
layers
 Example:

Zigzag of the DCT matrix, put the coefficients
into different layers
53
Alternative: Buffering Plus Layering
54
Bandwidth and Buffer Co-allocation
55