Transcript 21Morrison

Assessing Audio Visual Quality
P905 - AQUAVIT
Assessment of Quality for audio-visual signals
over Internet and UMTS
Geoff Morrison, BT Labs
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Contents
•
•
•
•
Why we are doing this
What we want to achieve
How we are going about it
Who is doing it
– BT, CSELT, Berkom
• When are we doing it
– February 1999 to December 2000
• The Results
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Rationale
• Coding algorithms, such as MPEG-4 and H.263, can
compress audio and video signals to bit rates
<1 Mbit/s as available over mobile and IP networks.
• The impact of impairments, from compression and
transmission, on users' opinion must be understood.
• Audiovisual communication must be evaluated under
representative conditions and real tasks, such as
conversational services and AV database retrieval.
• Network performance must be monitored to guarantee
a minimum QoS. Need single-ended models to check
audio and video transmission quality in real time.
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Compression Impairments
• Digital TV in studios is 216 Mbit/s
• Lossless compression typically only halves the bit
rate
• To reach very low rates requires lossy audio and
video compression, introducing significant distortions
Is the resulting application usable for purpose?
How best to assign a given total bit rate between
video and audio?
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Transmission Impairments
• AV bits <> data
• Audiovisual content has a “timeliness” aspect
– Critical in conversational applications
– Important in database retrieval applications
• Error detection and retransmission protocols not ideal
for audiovisual content
– variable delay
– variable throughput
• Compressed AV is very sensitive to transmission
errors, packet loss
– more compression  more sensitivity
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Error Handling
• Error resilience tools for encoder
– structure compressed data so that damage is localised
– optimise encoder options for the network performance
• Forward Error Correction
– improve apparent network performance seen by decoder
• Error concealment at decoder
– reconstruct approximations for damaged parts of pictures
and sound
• Combinations of above
How well do these work in practice?
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Project Objectives
• Understand the potential of audio, video and AV
communication at bit rates < 1Mbit/s for UMTS and IP.
• Develop tools and techniques for simulation of
complete audiovisual transmission systems.
• Develop quality evaluation methods for complete
audiovisual transmission systems.
• Investigate the relationship between system parameters
(network and terminal) and subjective quality.
• Provide guidelines for the system parameters to be
used for new applications.
• Encourage the use of audiovisual communication on
these dominant networks of the future
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Work Items
•
•
•
•
•
Define the tests to be performed
Develop/implement a UMTS test bed
Develop/implement an IP test-bed
Conduct subjective tests
Make objective measurements and relate them to
subjective results
• Disseminate the findings
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Definition of tests
• Literature survey for available test methods
• Select appropriate methods
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
UMTS Test Bed
•
•
•
•
•
•
•
•
•
•
Video codec (MPEG-4 and/or H.263)
Audio codec (MPEG AAC, scalable and non-scalable)
Error resilience tools (MPEG-4)
MPEG-4 multiplex
Transmission errors - simulated and perhaps actual
Error recovery
Demux
Optimisation of parameters
Tools to make objective measurements
Production of material for subjective and objective tests
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
UMTS Test Bed
• Baseline non real-time test bed
– MPEG-4 baseline video codec, AAC and G.723.1 audio, AL
layer and MPEG DMIF
• Enhanced non real-time test bed
– Error resilience tools, layered audio
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
UMTS Test Bed
Client
Server
Server Application
Audio decoder
Network interface
Network interface
Network
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Video decoder
IP Test Bed
•
•
•
•
•
•
•
•
•
Video codecs (H.263 and/or MPEG-4)
Audio codecs (G.723.1, MPEG AAC scalable and n/s)
Error resilience tools (ITU-T, MPEG, IETF)
Packetisation
Transmission errors - simulated, measured profiles and
actual
Error recovery
Optimisation of parameters
Tools to make objective data measurements
Production of material for subjective and objective tests
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
IP Test Bed
• Baseline non real-time test bed
– G.723.1, H.263, simplified RTP, simple network impairment
simulator
• Enhanced non real-time test bed
– more codecs including MPEG, error resilience, complete RTP,
error concealment, enhanced network simulator
• Enhanced real-time test bed
– real time image capture, encoding and decoding
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
IP Test Bed
Encoder
Encoder Application
Audio Encoder
Decoder
Server Application
Video Encoder
Audio Decoder
Network Interface
Heidelberg, 11-12 May 1999
Video Decoder
Network Interface
Network
AIMS’99 Workshop
Decoder Application
Subjective tests
• “Conventional” subjective tests covering a suitably
wide range of variables
–
–
–
–
–
the codecs
bit rate, video frame rate, audio bandwidth
Error conditions (packet loss, delay variability, …)
Protocols (CBR, VBR, ABR, RSVP, ..)
Application (one-way, conversational)
• Users may accept lower quality in return for mobility.
Testing will seek an estimation of the threshold of
acceptability of audio and video quality in mobile
applications
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Objective measurements
• Objective assessment of audio quality
– TOSQA, PAMS, ...
• Objective assessment of video quality
– model of human vision
– other algorithms to objectively estimate user’s opinion
• Compare subjective and objective assessments
• Investigate relationship between parameter values
and subjective audio, video and AV quality
assessments
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Double Ended Model
• Measuring instrument needs both original input to
encoder and delivered version at decoder output
System
under
Test
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Meter
Single Ended Model
• Measuring instrument needs only decoder output
– more suitable for live performance monitoring
System
under
Test
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Meter
Project Resources
20 man-months
Project
Management
Dissemination of
Results
Definition of
Tests
UMTS Testbed
IP Testbed
Subjective Tests
Objective Tests
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Results
• Reports due in Q4 of 2000.
• The findings should be of benefit to at least 3 groups:
– Network Providers. Results such as minimum desirable bit
rates and error characteristics should influence the planning
and operation of IP and UMTS networks intended to carry
audiovisual services.
– Application Providers. The results will provide reliable
guidance about which applications and operating parameters
will satisfactorily meet customers' expectations and those
which will not.
– Researchers in the field of "human factors". It is expected
that the project will advance the state of the art in perceptual
modelling and in the understanding of cross-modal effects.
AIMS’99 Workshop
Heidelberg, 11-12 May 1999