Transcript Delivering Audio and Video Over the Web

Delivering Audio and Video
Over the Web
Norman White
Stern School of Business
Business Case
 Multimedia is a powerful way to deliver a
message
 As a listener/viewer uses more senses to get a
message, retention rate goes up.
 Much information is better understood visually
 Question? What are the problems of delivering
audio and video over the internet?
Audio Formats
 WAV – direct digitization of audio
Size and quality depend on “sampling” rate
The higher the sample rate, the larger the file size
for the same length, but the better the quality
 Au
 MPEG
 Many others
Video Formats
 AVI
Windows video format
– Supports variety of CODECs
 CODEC – Compression-Decompression
 Quicktime (MOV)
Apple supported format
 MPEG – Motion Pictures Experts Group
Hard to manipulate, high quality
Digital Audio – ISDN
Integrated Service, Digital Network
 To digitize speech, the telephone standard is to
take 8000 samples a second, each sample
having a value (0-255, I.e. 8bits)
 Hence 1/2 of an ISDN line is 64kb (64000 bits
per second) (allows simultaneous voice and data
transmission)
 Note, this is uncompressed. We will talk about
compression later)
Bit rates for different audio streams

Telephone
G.711 PCM
G.721 ADCMP
G.722
G.728


Bit rate Kbps
64
32
48,56,64
16
CD
CD - DA
MPEG consumer
MPEG studio
1411
192
384
Motion Video Bit Rates (Mbps)

HDTV

Compressed

Digital TV (studio)
34

MPEG2
Broadcast TV (mpeg2)
 VCR (MPEG-1)
 Videoconferencing

2,000
166
3-6
2-4
 1.2
 .1 – 1.5

Some typical network rates
 56k modem - 56K bits per second
 ISDN
- 64kbps (2 channels 128)
 Wireless - 2MB - 11Mb/s
 Cable Modem - 10 Mb/s
 ethernet/token ring 10mb/s, now 100mbs
 T3 - 45 mb/s
 OC3 - 154MB/s
Video Tips
 Very important to capture video with the highest
quality you can
 Consumer Camcorders lose quality quickly as
light level drops
 Remember, audio quality is just as important,
consumer camcorders have poor microphones,
need separate high quality microphone for good
results.
Digital Camcorders
 The new digital Prosumer cameras have much
higher quality, but still some problems
 Recording time limits
Usually 1 hour for consumer cameras, OK for some
applications, but unacceptable if there is no break to
change media
 Blair Witch Project – First major movie to be shot
with Digital Camera
Cost Dropping quickly
Applications expand
Until recently, needed a > $10K Sony beta max for
decent video
 Now $2,000 digital cameras give comparable quality
(and MUCH smaller and lighter)
 Inexpensive disk drives now offer capacity to deal with
“raw” video (100MB per minute, 6 gigabytes for a one
hour avi file)
 Need fast processors to encode, decode, transfer data

More hardware advances
 IEEE 1394 – “Firewire” standard
Allows direct connection of digital devices without a
separate card
400 Mbit/s
Also supplies power to camera
 USB – Universal Serial Bus
Offers 12Mbit transfer
– Next version will be 100 Mbit
Networking Advances
100 Mbit ethernet – Allows streaming of MPEG Video
 “Switched” Ethernet – allows a separate stream to each
individual PC without collisions
 Multicasting router capabilities allow listeners to share a stream
 Cable Modems, ADSL, Satellite Dishes much higher bandwidth
to home, small business
 TCP/IP v6 – Quality of Service support
 Internet 2- 600Mbit connections
 Gigabit switches to handle traffic

Software Advances
 Rapidly improving streaming and compression
techniques to take advantage of more
processing power, new computer capabilities
(special instructions for A/D)
 Inexpensive, powerful video manipulation
software
Demand Explosion
 E-Commerce and B2B now can use on-line
video training, sales clips etc.
 Person to Person IP Video Conferencing
 On-line TV? Movies? Not yet…
Editing Video
 Adobe Premiere
Powerful Mac and PC based editor
 Supports “non-linear” editing
Used to have to edit by splicing tape (linear)
– Copy tape from one machine to another as you edit
Now can hold digital video content on disk and edit
frame by frame
 AVID – Commercial quality edit system
Adobe Premiere Functionality
 Cut / Splice Video clips
 Dissolve between clips
 Multiple audio tracks
 Supports many CODECS
 Background colors
 Titles
 Filters (blur, pan, zoom,sharpen distort etc.)
Premiere Basics
Video clips are shot
 Clips are brought into Premiere
 Clips are dropped on one of the video tracks in
constructor window
 Audio clips added, sound adjusted
 Titles, transitions, filters added
 “Make Movie” produces a new video clip in AVI , QT,
Real Media etc format.
 AVI format can be edited again

Conclusion
Video is now capable of being edited on consumer
class computers
 Need LOTS of disk space

100MBS / minute of video
Need 3 times as much space
Hard drive defragmented
Lot’s of memory
Fast processor (200hz Pentium minimum)
Digital capture card, Firewire or USB to bring video in
Conclusion
 All indications are that Streaming Media
applications will continue to have rapid growth
 Recent Dot-com problems have shut down some
of the early entrants in the consumer market.
BUT!!!
 Audio and video file sizes very large and need
transfer rates much faster than available
bandwidths.
 Many difficult problems need to be solved in
order to use audio and video over the internet.
Network Requirements of audio
and motion video
Problems for live usage
 Error rate
Network may not deliver bits reliably
Shouldn’t happen that often in a digital network due
to error control
 Transit delay
Data is not delivered at a constant speed, or at the
right speed
Audio Characteristics
 Ear is a differentiator
Has high powers of differentiation
Means that small audio problems are easily
recognized
Video Characteristics
 Eye is an integrator
It puts signals together
Hard to recognize multiple intermixed drawings and
interpret them
Missing information is less noticeable
 Means video can tolerate more errors and higher
delays
Implications for multimedia
 In a transmission of mixed audio and video,
audio should have a higher priority
Audio more sensitive to delays
 Note - this is a problem on the internet
Internet currently has no priority mechanism
Compression
 Can we reduce the amount of information we
need to send?
 We can compress it
 two types of compression
Lossy - Can’t exactly recreate original
Lossless - Can exactly recreate original file, bit for
bit.
Compressing different types of
info
 Images/ Graphics
Gif (lossless)
JPEG - Can be lossy
 Sound (first digitize at some sampling rate)
Then compress
 Video
(usually lossy)
Intraframe compression
interframe compression
Video compression techniques

Intraframe compression
Compress the pixels within one frame

Interframe compression
Since video frames often change very little between frames,
we can increase compression by only recording the changed
bits from one frame to the next.
This works extremely well for many situations

Video - 1 frame = 640*480*8 bits = 300K bytes (
10MB/sec 30 fps)
Compression Problems
 Sending compressed data means that problems
in one block (packet) of data may impact
subsequent data blocks that have related
information which is based on the content of an
earlier data block (I.e. keyframes)
Partial solution to network delay
 Add an offset delay to reception of message
 The receiving side will collect N seconds of
information before it starts playing the audio or
video clip
 How big should ‘N’ be?
Depends on effective speed of communications
channel
higher variance means larger N
Other Problems
 Intermedia Synchronization
Need to synchronize audio and video streams
 Compression and burstiness
Compression can introduce burstiness into a nonbursty transmission
Example. Audio uncompressed has a fixed number
of bits per second, compressed it can go down to
zero (periods of silence)
Conclusion
 Network multimedia demands many complex
tradeoffs between bandwidth, errors, delays and
cost
 Compression makes the problem even more
complex
Error rates have to be lower to use compression
 What are some reasonable levels for audio and
video?
Streaming Audio and Video
Problems with audio/video over
the web

File sizes large, perhaps infinite for continuous live
webcasts
Can’t wait for file to completely download
WEB bandwidth unreliable - high variance
 Bandwidth varies from minute to minute
 Need to have complex compression and streaming
protocols to compensate

Not supported by HTTP protocol
– http is basically a download protocol, not sophisticated
Demands much more complex protocol to try to compensate
for problems
Solution
 Move audio and video files to a separate server
which uses a different (more sophisticated)
communications protocol
 Have client application communicate directly
with the media server.
Current Solutions

Real Networks
RTSP - Real Time Streaming Protocol
Needs a separate server
Protocol supports multiple, simultaneous streams (audio,
video, animation, text)
Helper App to play the streams (REAL PLAYER )
SMIL language to describe relationship between streams
Automatic, dynamic bandwidth negotiation
– Server and client continue negotiating transfer rate in real time.
RTSP features
Determines average bandwidth between video server
and client and variance
 Preloads part of file before it starts playing
 Dynamically adjusts needed bandwidth by varying
quality of stream as necessary

Audio switches codecs when bandwidth drops
– CD -> HiFi-> mono – Voice
Video increases compression and drops frames

Objective is to keep a “reasonable” output at all times.
Real Player Encoding
 User specifies bit rates to encode in the file at
encoding time.
 Real media file (.rm) is generated that contains
data necessary to generate any of the supported
bit streams
 More bit streams larger files on server
 More bit streams better possibility of playback
Example
User is connected to server with a connection that is
varying between 30 and 45 K bits per second
 User plays a 60 second video clip that was encoded at
30, 40, 50, 100 K bits per second
 Server and Player will negotiate a 40 K bit per second
stream
 Player will buffer (predownload part of file) enough to
ensure the stream will play continuosly even if
throughput is only 30K bits per second

How long does Real Player wait
before starting to play?
Stream is 60*40K bits/second = 2,400,000 bits
 At 30K bits per second it will take 2,400,000/30,000
seconds = 80 seconds
 Need to wait 20 seconds before starting to play stream,
i.e load up 20 seconds of material (20*40,000 =
800,000 bits or 100,000 characters
 Note the many design limitations here.

Is there enough memory
How long can user wait before giving up
What happens if bandwidth changes and they need to
renegotiate …
Other Solutions
 Microsoft ASF - Advanced Streaming Format
Only supports one stream, need 3rd party tools to
combine multiple streams into ASF format
Works with Microsoft NETSHOW
Supports multiple bitrates in one stream
includes high bandwidth intranet option
Publishing Streaming Media with
Real Networks

Need 3 components
WEB Server
– Create links to .RAM, or .SMI files
– RAM files point to REAL Networks server
Real Networks Server
–
–
–
–
Files are housed here
Special “Streaming” support
Multicast support
Handles Live Streams
Browser Helper APP
– Real Player G2 - can run standalone
Example ..
 First, need to capture and encode Video/audio in
Real Media (.rm) Format
 Run Encoder software
Specify audio and video “codecs” to use
– Each bitrate has a different codec
– Can use multiple bitrates
Move Encoded file to Real Networks Server
Create Links on WEB server to point to file
Video file TEST.RM
 create link to .ram file on WEB server
<a href=“TEST.ram”> Click here to see video</a>
 create TEST.ram file
rtsp://videos.stern.nyu.edu:554/TEST.rm
What Happens

User clicks on link
Link is a .ram file, so helper app is invoked to process the file
Helper App (REAL PLAYER 8) starts and is passed the URL
of the 1 line ram file by the web server
REAL PLAYER 8 reads (HTTP GET request) ram file and
then issues RTSP connection to video server.
Video Server and Real Player now have a separate
TCP/IP connection (port 554) to communicate over
using the Real Time Streaming Protocol.
 Why so complicated?

Because web servers only support http protocol
Synchronized Media
 What happens if I want to synchronize several
different streams. Need to use SMIL
Synchronized Media Interpretation Language?
 The link needs to point to a .SMI file instead of a
RAM file.
 Simple example, put an intro in front of the video
 I.e. play intro file followed by TEST.rm
SMIL Example
<smil>
<root> </root>
<body>
<seq>
<rtsp://videos.stern.nyu.edu:554/file1.rm ….> first file
to play
 <rtsp://videos.stern.nyu.edu:554/file2.rm …> second
file to play
 </seq>
 </body>
 </smil>





More Advanced SMIL
Smil can also play material in parallel in different
windows
 Example would be a REAL TEXT presentation in one
window, with synchronized Real pics in another window,
and synchronized audio playing as the slides (and text)
changes.
 Real Presenter – Real Networks add-on ($29.95) to
Powerpoint to allow streaming Powerpoint narration.

Some links
 WWW.REAL.COM
 WWW.ABC.COM
Real Networks Tools
Beta versions available for free
 Real Player 8
Helper App to play Real player audio and video
 Real Publisher
Set of tools to encode audio/video to support a
variety of speeds (network and computer)
Supports SURESTREAM ™
– Automatically switches speeds
 REAL SERVER
Host content
Other Features
 REAL TOOLS
REAL PIX - Stream images
REAL TEXT - Stream Text
REAL FLASH - Stream Macromedia Flash
SMIL - Synchronize all of the above
Final Project Ideas
 Stern TV Station - Multiple Channels
Live events
Archived events
News?
 Video classes with power point slides
coordinated with talk.