Transcript Delivering Audio and Video Over the Web

Delivering Audio and Video
Over the Web
Norman White
Stern School of Business
Agenda
•
•
•
•
•
Audio/video formats
Rates of transmission
Hardware & software advances
Network requirements
Streaming audio & video
Business Case
• Multimedia is a powerful way to deliver a
message
• As a listener/viewer uses more senses to get
a message, retention rate goes up.
• Much information is better understood
visually
• Question? What are the problems of
delivering audio and video over the internet?
Audio Formats
• WAV – direct digitization of audio
– Size and quality depend on “sampling” rate
– The higher the sample rate, the larger the
file size for the same length, but the better
the quality
• Au
• MPEG
• Many others
Video Formats
• AVI
– Windows video format
• Supports variety of CODECs
– CODEC – Compression-Decompression
• Quicktime (MOV)
– Apple supported format
• MPEG – Motion Pictures Experts Group
– Hard to manipulate, high quality
Digital Audio – ISDN Integrated
Service, Digital Network
• To digitize speech, the telephone standard is
to take 8000 samples a second, each sample
having a value (0-255, I.e. 8bits)
• Hence 1/2 of an ISDN line is 64kb (64000 bits
per second) (allows simultaneous voice and
data transmission)
• Note, this is uncompressed. We will talk
about compression later)
Bit rates for different audio
streams
• Telephone
–
–
–
–
G.711 PCM
G.721 ADCMP
G.722
G.728
• CD
– CD - DA
– MPEG consumer
– MPEG studio
• Bit rate Kbps
–
–
–
–
64
32
48,56,64
16
– 1411
– 192
– 384
Motion Video Bit Rates (Mbps)
• HDTV
– Compressed
• Digital TV (studio)
– MPEG2
• Broadcast TV (mpeg2)
• VCR (MPEG-1)
• Videoconferencing
• 2,000
– 34
• 166
– 3-6
• 2-4
• 1.2
• .1 – 1.5
Some typical network rates
•
•
•
•
•
•
•
56k modem - 56K bits per second
ISDN
- 64kbps (2 channels 128)
Wireless - 2MB - 11Mb/s
Cable Modem - 10 Mb/s
ethernet/token ring 10mb/s, now 100mbs
T3 - 45 mb/s
OC3 - 154MB/s
Video Tips
• Very important to capture video with the
highest quality you can
• Consumer Camcorders lose quality
quickly as light level drops
• Remember, audio quality is just as
important, consumer camcorders have
poor microphones, need separate high
quality microphone for good results.
Digital Camcorders
• The new digital Prosumer cameras have
much higher quality, but still some problems
• Recording time limits
– Usually 1 hour for consumer cameras, OK for
some applications, but unacceptable if there is no
break to change media
• Blair Witch Project – First major movie to be
shot with Digital Camera
Cost Dropping quickly
Applications expand
• Until recently, needed a > $10K Sony beta
max for decent video
• Now $2,000 digital cameras give comparable
quality (and MUCH smaller and lighter)
• Inexpensive disk drives now offer capacity to
deal with “raw” video (100MB per minute, 6
gigabytes for a one hour avi file)
• Need fast processors to encode, decode,
transfer data
More hardware advances
• IEEE 1394 – “Firewire” standard
– Allows direct connection of digital devices
without a separate card
– 400 Mbit/s
– Also supplies power to camera
• USB – Universal Serial Bus
– Offers 12Mbit transfer
• Next version will be 100 Mbit
Networking Advances
• 100 Mbit ethernet – Allows streaming of MPEG Video
• “Switched” Ethernet – allows a separate stream to
each individual PC without collisions
• Multicasting router capabilities allow listeners to
share a stream
• Cable Modems, ADSL, Satellite Dishes much higher
bandwidth to home, small business
• TCP/IP v6 – Quality of Service support
• Internet 2- 600Mbit connections
• Gigabit switches to handle traffic
Software Advances
• Rapidly improving streaming and
compression techniques to take
advantage of more processing power,
new computer capabilities (special
instructions for A/D)
• Inexpensive, powerful video
manipulation software
Demand Explosion
• E-Commerce and B2B now can use online video training, sales clips etc.
• Person to Person IP Video
Conferencing
• On-line TV? Movies? Not yet…
Editing Video
• Adobe Premiere
– Powerful Mac and PC based editor
• Supports “non-linear” editing
– Used to have to edit by splicing tape (linear)
• Copy tape from one machine to another as you edit
– Now can hold digital video content on disk and
edit frame by frame
• AVID – Commercial quality edit system
Adobe Premiere Functionality
•
•
•
•
•
•
•
Cut / Splice Video clips
Dissolve between clips
Multiple audio tracks
Supports many CODECS
Background colors
Titles
Filters (blur, pan, zoom,sharpen distort etc.)
Premiere Basics
• Video clips are shot
• Clips are brought into Premiere
• Clips are dropped on one of the video tracks
in constructor window
• Audio clips added, sound adjusted
• Titles, transitions, filters added
• “Make Movie” produces a new video clip in
AVI , QT, Real Media etc format.
• AVI format can be edited again
Conclusion
• Video is now capable of being edited on
consumer class computers
• Need LOTS of disk space
–
–
–
–
–
–
100MBS / minute of video
Need 3 times as much space
Hard drive defragmented
Lot’s of memory
Fast processor (200hz Pentium minimum)
Digital capture card, Firewire or USB to bring
video in
Conclusion
• All indications are that Streaming Media
applications will continue to have rapid
growth
• Recent Dot-com problems have shut
down some of the early entrants in the
consumer market.
BUT!!!
• Audio and video file sizes very large
and need transfer rates much faster
than available bandwidths.
• Many difficult problems need to be
solved in order to use audio and video
over the internet.
Network Requirements of
audio and motion video
Problems for live usage
• Error rate
– Network may not deliver bits reliably
– Shouldn’t happen that often in a digital
network due to error control
• Transit delay
– Data is not delivered at a constant speed,
or at the right speed
Audio Characteristics
• Ear is a differentiator
– Has high powers of differentiation
– Means that small audio problems are
easily recognized
Video Characteristics
• Eye is an integrator
– It puts signals together
– Hard to recognize multiple intermixed
drawings and interpret them
– Missing information is less noticeable
• Means video can tolerate more errors
and higher delays
Implications for multimedia
• In a transmission of mixed audio and
video, audio should have a higher
priority
– Audio more sensitive to delays
• Note - this is a problem on the internet
– Internet currently has no priority
mechanism
Compression
• Can we reduce the amount of
information we need to send?
• We can compress it
• two types of compression
– Lossy - Can’t exactly recreate original
– Lossless - Can exactly recreate original
file, bit for bit.
Compressing different types of
info
• Images/ Graphics
– Gif (lossless)
– JPEG - Can be lossy
• Sound (first digitize at some sampling rate)
– Then compress
• Video
(usually lossy)
– Intraframe compression
– interframe compression
Video compression
techniques
• Intraframe compression
– Compress the pixels within one frame
• Interframe compression
– Since video frames often change very little
between frames, we can increase compression by
only recording the changed bits from one frame to
the next.
– This works extremely well for many situations
• Video - 1 frame = 640*480*8 bits = 300K
bytes ( 10MB/sec 30 fps)
Compression Problems
• Sending compressed data means that
problems in one block (packet) of data
may impact subsequent data blocks that
have related information which is based
on the content of an earlier data block
(I.e. keyframes)
Partial solution to network
delay
• Add an offset delay to reception of message
• The receiving side will collect N seconds of
information before it starts playing the audio
or video clip
• How big should ‘N’ be?
– Depends on effective speed of communications
channel
– higher variance means larger N
Other Problems
• Intermedia Synchronization
– Need to synchronize audio and video
streams
• Compression and burstiness
– Compression can introduce burstiness into
a non-bursty transmission
– Example. Audio uncompressed has a fixed
number of bits per second, compressed it
can go down to zero (periods of silence)
Conclusion
• Network multimedia demands many complex
tradeoffs between bandwidth, errors, delays
and cost
• Compression makes the problem even more
complex
– Error rates have to be lower to use compression
• What are some reasonable levels for audio
and video?
Streaming Audio and Video
Problems with audio/video
over the web
• File sizes large, perhaps infinite for
continuous live webcasts
– Can’t wait for file to completely download
• WEB bandwidth unreliable - high variance
• Bandwidth varies from minute to minute
• Need to have complex compression and
streaming protocols to compensate
– Not supported by HTTP protocol
• http is basically a download protocol, not sophisticated
– Demands much more complex protocol to try to
compensate for problems
Solution
• Move audio and video files to a
separate server which uses a different
(more sophisticated) communications
protocol
• Have client application communicate
directly with the media server.
Current Solutions
• Real Networks
– RTSP - Real Time Streaming Protocol
– Needs a separate server
– Protocol supports multiple, simultaneous streams
(audio, video, animation, text)
– Helper App to play the streams (REAL PLAYER )
– SMIL language to describe relationship between
streams
– Automatic, dynamic bandwidth negotiation
• Server and client continue negotiating transfer rate in real
time.
RTSP features
• Determines average bandwidth between
video server and client and variance
• Preloads part of file before it starts playing
• Dynamically adjusts needed bandwidth by
varying quality of stream as necessary
– Audio switches codecs when bandwidth drops
• CD -> HiFi-> mono – Voice
– Video increases compression and drops frames
• Objective is to keep a “reasonable” output at
all times.
Real Player Encoding
• User specifies bit rates to encode in the
file at encoding time.
• Real media file (.rm) is generated that
contains data necessary to generate
any of the supported bit streams
• More bit streams larger files on server
• More bit streams better possibility of
playback
Example
• User is connected to server with a connection
that is varying between 30 and 45 K bits per
second
• User plays a 60 second video clip that was
encoded at 30, 40, 50, 100 K bits per second
• Server and Player will negotiate a 40 K bit per
second stream
• Player will buffer (predownload part of file)
enough to ensure the stream will play
continuosly even if throughput is only 30K bits
per second
How long does Real Player
wait before starting to play?
• Stream is 60*40K bits/second = 2,400,000 bits
• At 30K bits per second it will take 2,400,000/30,000
seconds = 80 seconds
• Need to wait 20 seconds before starting to play
stream, i.e load up 20 seconds of material (20*40,000
= 800,000 bits or 100,000 characters
• Note the many design limitations here.
– Is there enough memory
– How long can user wait before giving up
– What happens if bandwidth changes and they
need to renegotiate …
Other Solutions
• Microsoft ASF - Advanced Streaming
Format
– Only supports one stream, need 3rd party
tools to combine multiple streams into ASF
format
– Works with Microsoft NETSHOW
– Supports multiple bitrates in one stream
– includes high bandwidth intranet option
Publishing Streaming Media
with Real Networks
• Need 3 components
– WEB Server
• Create links to .RAM, or .SMI files
• RAM files point to REAL Networks server
– Real Networks Server
•
•
•
•
Files are housed here
Special “Streaming” support
Multicast support
Handles Live Streams
– Browser Helper APP
• Real Player G2 - can run standalone
Example ..
• First, need to capture and encode
Video/audio in Real Media (.rm) Format
• Run Encoder software
– Specify audio and video “codecs” to use
• Each bitrate has a different codec
• Can use multiple bitrates
– Move Encoded file to Real Networks
Server
– Create Links on WEB server to point to file
Video file TEST.RM
• create link to .ram file on WEB server
– <a href=“TEST.ram”> Click here to see
video</a>
• create TEST.ram file
– rtsp://videos.stern.nyu.edu:554/TEST.rm
What Happens
• User clicks on link
– Link is a .ram file, so helper app is invoked to
process the file
– Helper App (REAL PLAYER 8) starts and is
passed the URL of the 1 line ram file by the web
server
– REAL PLAYER 8 reads (HTTP GET request) ram
file and then issues RTSP connection to video
server.
• Video Server and Real Player now have a separate
TCP/IP connection (port 554) to communicate over
using the Real Time Streaming Protocol.
• Why so complicated?
– Because web servers only support http protocol
Synchronized Media
• What happens if I want to synchronize
several different streams. Need to use SMIL
– Synchronized Media Interpretation Language?
• The link needs to point to a .SMI file instead
of a RAM file.
• Simple example, put an intro in front of the
video
• I.e. play intro file followed by TEST.rm
SMIL Example
•
•
•
•
•
<smil>
<root>
</root>
<body>
<seq>
<rtsp://videos.stern.nyu.edu:554/file1.rm
….> first file to play
• <rtsp://videos.stern.nyu.edu:554/file2.rm …>
second file to play
• </seq>
• </body>
• </smil>
More Advanced SMIL
• Smil can also play material in parallel in
different windows
• Example would be a REAL TEXT
presentation in one window, with
synchronized Real pics in another window,
and synchronized audio playing as the slides
(and text) changes.
• Real Presenter – Real Networks add-on
($29.95) to Powerpoint to allow streaming
Powerpoint narration.
Some links
• WWW.REAL.COM
• WWW.ABC.COM
Real Networks Tools
Beta versions available for free
• Real Player 8
– Helper App to play Real player audio and video
• Real Publisher
– Set of tools to encode audio/video to support a
variety of speeds (network and computer)
– Supports SURESTREAM ™
• Automatically switches speeds
• REAL SERVER
– Host content
Other Features
• REAL TOOLS
–
–
–
–
REAL PIX - Stream images
REAL TEXT - Stream Text
REAL FLASH - Stream Macromedia Flash
SMIL - Synchronize all of the above
Final Project Ideas
• Stern TV Station - Multiple Channels
– Live events
– Archived events
– News?
• Video classes with power point slides
coordinated with talk.