Transcript Network Fault Events - Ohio Supercomputer Center

Measuring Interaction QoE in Internet Videoconferencing
Prasad Calyam (Presenter)
Ohio Supercomputer Center, The Ohio State University
Mark Haffner, Prof. Eylem Ekici
The Ohio State University
Prof. Chang-Gun Lee
Seoul National University
MMNS, November 1st 2007
Outline
•
Background
•
•
•
•
•
Voice and Video over IP (VVoIP) Overview
Network QoS and End-user QoE in VVoIP
Streaming QoE versus Interaction QoE
Network Fault Events
Multi-Activity Packet Trains (MAPTs) methodology
•
•
Participant Interaction Patterns
Traffic Model for MAPTs emulation
•
Vperf tool implementation of MAPTs
•
Performance Evaluation
•
Concluding Remarks and Future Work
2
Outline
•
Background
•
•
•
•
•
Voice and Video over IP (VVoIP) Overview
Network QoS and End-user QoE in VVoIP
Streaming QoE versus Interaction QoE
Network Fault Events
Multi-Activity Packet Trains (MAPTs) methodology
•
•
Participant Interaction Patterns
Traffic Model for MAPTs emulation
•
Vperf tool implementation of MAPTs
•
Performance Evaluation
•
Concluding Remarks and Future Work
3
Voice and Video over IP (VVoIP) Overview
 Large-scale deployments of VVoIP are on the rise
 Video streaming (one-way voice and video)
 MySpace, Google Video, YouTube, IPTV, …
 Video conferencing (two-way voice and video)
 Polycom, MSN Messenger, WebEx, Acrobat Connect, …
 Challenges for large-scale VVoIP deployment
 Real-time or online monitoring of end-user Quality of Experience (QoE)
 Traditional network Quality of Service (QoS) monitoring not adequate
 Network QoS metrics: bandwidth, delay, jitter, loss
 Need objective techniques for automated network-wide monitoring
 Cannot rely on end-users to provide subjective rankings – expensive and
time consuming
4
Network QoS and End-user QoE
 End-user QoE is mainly dependent on the combined impact of network factors
 Device factors such as voice/video codecs, peak video bit rate (a.k.a. dialing speed)
also matter
Network QoS
End-user QoE
 Our study maps the network QoS to end-user QoE for a given set of commonly
used device factors
 H.263 video codec, G.711 voice codec, 256/384/768 Kbps dialing speeds
5
Voice and Video Packet Streams
 Total packet size (tps) – sum of payload (ps), IP/UDP/RTP
header (40 bytes), and Ethernet header (14 bytes)
 Dialing speed is
voice codec
;
= 64 Kbps fixed for G.711
 Voice has fixed packet sizes (tpsvoice ≤ 534 bytes)
 Video packet sizes are dependent on alev in the content
6
Video alev
 Low alev
 Slow body movements and constant background; E.g. Claire video sequence
 High alev
 Rapid body movements and/or quick scene changes; E.g. Foreman video sequence
 ‘Listening’ versus ‘Talking’
 Talking video alev(i.e., High) consumes more bandwidth than Listening video alev (i.e., Low)
Claire
Foreman
7
End-user QoE Types
 Streaming QoE
 End-user QoE affected just by voice and video impairments
 Video frame freezing
 Voice drop-outs
 Lack of lip sync between voice and video
 Interaction QoE
 End-user QoE also affected by additional interaction effort in a conversation
 “Can you repeat what you just said?”
 “This line is noisy, lets hang-up and reconnect…”
 QoE is measured using “Mean Opinion Score” (MOS) rankings
8
Network Fault Events
 “Best-effort service” of Internet causes network fault events that
impact application performance
 Cross traffic congestion, routing instabilities, physical link failures, DDoS attacks
 Our definition of network fault events is based on the “Good”, “Acceptable”
and “Poor” (GAP) performance levels for QoS metrics causing GAP QoE
 Type-I: Performance of any network factor changes from Good grade to
Acceptable grade over a 5 second duration
 Type-II: Performance of any network factor changes from Good grade to Poor
grade over a 10 second duration
9
Related Work
 Characteristics of network fault events well understood
 Bursts, spikes, complex patterns – lasting few seconds to a few minutes (Markopoulou et.
al., Ciavattone et. al.)
 Measuring Streaming QoE impact due to network fault events has been
well studied
 ITU-T E-Model is a success story for VoIP QoE estimation
 Designed for CBR voice traffic and handles only voice related impairments
 ITU-T J.144 developed for VVoIP QoE measurement
 “PSNR-based MOS” – Requires original and reconstructed video frames for frameby-frame comparisons
 Offline method - PSNR calculation is a time consuming and computationally
intensive process
 Online VVoIP QoE measurement proposals
 PSQA (G. Rubino, et. al.), rPSNR (S. Tao, et. al.)
 Measuring Interaction QoE impact due to network fault events has NOT
received due attention
 Need for schemes to measure interaction difficulties in voice and video conferences
presented by A. Rix, et. al.
10
Problem Summary
 Given:
 Voice/video codecs used in a videoconference
 Dialing speed of the videoconference
 Network fault event types to monitor
Multi-Activity Packet Trains Methodology
 Develop:
 An objective technique that can measure Interactive VVoIP QoE
 Real-time measurement without involving actual end-users, video
sequences and VVoIP appliances
 An active measurement tool that can: (a) emulate VVoIP traffic on a network
path, and (b) use the objective technique to produce Interaction QoE
measurements
Vperf Tool
11
Outline
•
Background
•
•
•
•
•
Voice and Video over IP (VVoIP) Overview
Network QoS and End-user QoE in VVoIP
Streaming QoE versus Interaction QoE
Network Fault Events
Multi-Activity Packet Trains (MAPTs) methodology
•
•
Participant Interaction Patterns
Traffic Model for MAPTs emulation
•
Vperf tool implementation of MAPTs
•
Performance Evaluation
•
Concluding Remarks and Future Work
12
Proposed Solution Methodology
 “Multi-Activity Packet Trains” (MAPTs) measure
Interaction QoE in an automated manner
 They mimic participant interaction patterns and video activity levels
as affected by network fault events
 Given a session-agenda, excessive talking than normal due to
unwanted participant interaction patterns impacts Interaction QoE
 “Unwanted Agenda-bandwidth” measurement and compare with
baseline (consumption during normal conditions)
 Higher values indicate poor interaction QoE and caution about
potential increase in Internet traffic congestion levels
 Measurements serve as an input for ISPs to improve network
performance using suitable traffic engineering techniques
13
Proposed Solution Methodology (2)
‘repeat’
‘disconnect’
‘reconnect’
‘reorient’
Type-I and Type-II fault detection
14
Participant Interaction Patterns
 Assumption: Question (Request) and Answer (Response) items in a
session agenda
 Side-A listening when Side-B talking, and vice versa
Normal – PIP1
15
Participant Interaction Patterns (2)
 Participant Interaction Patterns (PIPs) using MAPTs for a “Type-I”
network fault event
Repeat – PIP2
16
Participant Interaction Patterns (3)
 Participant Interaction Patterns (PIPs) using MAPTs for a “Type-II”
network fault event
Disconnect/Reconnect/Reorient – PIP3
17
Participant Interaction Patterns (4)
 Our goal is to measure the Unwanted Agenda-bandwidth and Unwanted Agenda-time
measurements after MAPTs emulation of the Q & A session agenda
18
Traffic Model for MAPTs Emulation
 Traffic Model for probing packet trains obtained from trace-analysis
 Combine popularly used low and high alev video sequences and model them
at 256/384/768 Kbps dialing speeds for H.263 video codec
 Low – Grandma, Kelly, Claire, Mother/Daughter, Salesman
 High – Foreman, Car Phone, Tempete, Mobile, Park Run
 Modeling
 Video Encoding Rates (bsnd) time series
 Packet Size (tps) distribution
 Derived instantaneous inter-packet times (tps) by dividing instantaneous
packet sizes by video encoding rates
19
Video Encoding Rates (bsnd) Modeling
 Time-series modeling of the bsnd data using the classical
decomposition method
 We find a Second order moving average [MA(2)] process
model fit
 θ1 – MA(1) parameter
 θ2 – MA(2) parameter
Low alev
High alev
20
Video Packet Size (tps) Distribution Modeling
 Distribution-fit analysis on the tps data
 We find a Gamma distribution fit
 α – shape parameter
 β – scale parameter
For High alev
at 256 Kbps
21
Traffic Model Parameters for MAPTs Emulation
22
Outline
•
Background
•
•
•
•
•
Voice and Video over IP (VVoIP) Overview
Network QoS and End-user QoE in VVoIP
Streaming QoE versus Interaction QoE
Network Fault Events
Multi-Activity Packet Trains (MAPTs) methodology
•
•
Participant Interaction Patterns
Traffic Model for MAPTs emulation
•
Vperf tool implementation of MAPTs
•
Performance Evaluation
•
Concluding Remarks and Future Work
23
Vperf Tool Implementation of MAPTs
 Per-second frequency of “Interim Test Report” generation
 Interaction QoE reported by Vperf tool - based on the progress of the
session-agenda
24
Example – Session Agenda and Network Factor Limits File
25
Outline
•
Background
•
•
•
•
•
Voice and Video over IP (VVoIP) Overview
Network QoS and End-user QoE in VVoIP
Streaming QoE versus Interaction QoE
Network Fault Events
Multi-Activity Packet Trains (MAPTs) methodology
•
•
Participant Interaction Patterns
Traffic Model for MAPTs emulation
•
Vperf tool implementation of MAPTs
•
Performance Evaluation
•
Concluding Remarks and Future Work
26
MAPTs Emulation at different Dialing Speeds
256 Kbps
384 Kbps
768 Kbps
27
MAPTs Measurements Evaluation
 Increased the number of Type-I and Type-II network fault events in a
controlled LAN testbed for a fixed session-agenda
 NISTnet network emulator for network fault generation
 Recorded Unwanted Agenda-Bandwidth and Unwanted Agenda-Time
measured by Vperf tool
(a) Impact of Type-I Network Fault Events
on Unwanted Agenda-Bandwidth
(b) Impact of Type-II Network Fault Events
on Unwanted Agenda-Bandwidth
28
MAPTs Measurements Evaluation (2)
(c) Impact of Type-I and Type-II Network
Fault Events on Unwanted Agenda-Time
29
Outline
•
Background
•
•
•
•
•
Voice and Video over IP (VVoIP) Overview
Network QoS and End-user QoE in VVoIP
Streaming QoE versus Interaction QoE
Network Fault Events
Multi-Activity Packet Trains (MAPTs) methodology
•
•
Participant Interaction Patterns
Traffic Model for MAPTs emulation
•
Vperf tool implementation of MAPTs
•
Performance Evaluation
•
Concluding Remarks and Future Work
30
Conclusion
 We proposed a Multi-Activity Packet Trains methodology
 Mimic participant interaction patterns and video activity levels as
affected by network fault events
 MAPTs provide real-time objective measurements of Interaction QoE
in a large-scale videoconferencing system
 Without requiring end-users, actual video sequences, VVoIP appliances
 Defined new Interaction QoE Metrics
 Unwanted Agenda-Bandwidth, Unwanted Agenda-Time
 Implemented MAPTs in an active measurement tool called Vperf
and evaluated Interaction QoE measurements on a network testbed
31
Future Work
 Our work is a first-step towards measuring how network fault
events impact Interaction QoE in videoconferencing sessions
 We considered basic participant interaction patterns and
network fault event types
 Future scope could include several other participant interaction
patterns and network fault event types
 E.g. MAPTs for network fault events that cause lack of lip-sync
 Human subject testing to more accurately map and validate
network fault event types and participant interaction patterns
32
Thank you for your attention!
☺
Any Questions?
33