Transcript Network Assurance and Testing During the Migration to VoIP

Analyze
Assure
ST-09 Network Assurance
and Testing During the
Migration to VoIP
TMC Developers Conference
San Francisco
Aug 03rd, 2005
Andy Huckridge
Spirent Communications.
Chair, Interop WG, MSF
Accelerate
Agenda
•
•
•
•
•
Spirent overview
•
Metrics for Measuring Voice and Video Quality and
Performance
•
Good test methodology
Key implementation issues
What is Triple Play / Converged networks?
Specifics on testing SIP
Network Impairments and Parameters that Voice
and Video Affect Quality
Analyze
Assure
Accelerate
Spirent Communications
• Spirent is the test solution leader
– 1,800 employees in 14 countries
– More than 1,500 customers
– Sales and service capabilities in
30 countries
Analyze
Assure
Accelerate
Implementation steps - Lab
Services
Deployment
Performance
Analysis
Industry
standards
Manufacturing
Quality Assurance
Functionality &
Conformance Testing
Performance
Testing
• Characterize system before
trial
• Validate system scalability
• Identify capacity limits
• Measure call performance
• Automate regression testing
Analyze
Assure
Accelerate
Implementation steps - Network
Service
Assurance
Initial Deployment
Network Certification
Pilot Networks
Widespread
Network
Deployment
• Facilitate vendor selection
• Identify performance ceilings
• Enable accurate capacity planning
• End-to-end service assurance testing
• Improve operational performance
• Improve customer satisfaction
Analyze
Assure
Accelerate
Key Implementation Issues
•
•
•
•
•
Circuit to packet migration
Scalability and Performance
Voice quality
Interoperability and conformance
Budget pressures
Analyze
Assure
Accelerate
Before you deploy!
• Network
Equipment Manufacturers (Chips,
IP-PBX, Gateways, MSs & SSs)
–
–
–
–
Characterize your system before trial
Validate system scalability
Identify capacity limits
Measure call performance
• Service
Providers
(NSPs, SPs, ITSPs)
–
–
–
–
Define criteria for vendor selection
Identify performance ceilings
Accurately plan for your capacity needs
End-to-end service assurance testing
Analyze
Assure
Accelerate
Data Transmission
“Non-Real-Time” Applications
Web
HTTP
Data Base
MS SQL
Oracle
Telenet
Name
Resolving
DNS
Email and
Messaging
POP
SMPT
Exchange
File Transfer
FTP
Music Downloading
Home control
Data Examples: Internet access, Email, File Transfer, Portals, Database Applications,
Gaming, Government Services, Online Commerce
Analyze
Assure
Accelerate
Voice and Video
“Real-Time Applications”
VoD
VoIP, IP Telephony, Video Telephony
G.711, G.729, G.728, G.726, G.723
H.261, H.263,
SIP, SIP-T, H323, Skinny, MGCP,
MEGACO/H.248
Multi-Media
RTP
H.264,Microsoft AVI, QuickTime (.mov)
Windows Media (.wmv, .asf), RealMedia (.rm),
IP Music/Audio/Radio
Real-Time Online
Communications
Instant Messenger
Webex
Netmeeting
SIP
H.323
Gaming
Single / Multiplayer
IPTV Services
Broadcast, On-Demand,
Bi-directional / Interactive
MPEG1, MPEG2,
MPEG4, VC1, H264
Voice Applications: Phone service integrated with video
Video Applications: Broadcast TV, video on demand, distance learning
Analyze
Assure
Accelerate
Converged Triple Play: Data, Voice and Video
With Network Impairments
Data
Video
Dialed
On
Ring
Digits
Hook
Off
Dial
Good
Hook
#
Back
Hello
Bye
Configure
Configure
Notify
Connect
Analyze
Assure
Connect
Accelerate
Signaling
Path
Impairments
Voice
Good
On
Off
Hook
Ring Conversation
Disconnect
Hello
Bye
Hook
can be heard in the voice conversation
Testing SIP Conformance
•
•
•
•
•
Comprehensive and scriptable SIP call flows
Complete configurable SIP signaling messages
SIP protocol analysis
Simplified flow diagrams with visual analysis
Comprehensive conformance test suites
Analyze
Assure
Accelerate
Testing SIP Conformance
• ETSI TS 102-027-1 v2.12,Tiphon:
– RFC 3261 user agent, proxy and redirect server
compliance
• Graphical SDL and TTCN tools
– Create, edit, compile and execute simulation scripts
and conformance tests
• Additional SIP messages beyond RFC 3261
– Included in torture tests
• Additional tests as defined by the SIP Forum
Analyze
Assure
Accelerate
Testing with Configurable SIP
Configurable SIP call setup and call teardown
• Configurable call flows and messages
• Incoming message filter
– Adaptive signaling syntax for SIP
– Improves interoperability with new drafts and
non-conformant proprietary implementations
Analyze
Assure
Accelerate
Testing with Configurable SIP
•
Configurable messages:
 Invite, ACK, bye, register
 Responses: 1xx, 2xx, 3xx, 4xx, 5xx, 6xx
•
Configurable timers, message intervals
•
Enable and disable optional messages:
 Re-invite, cancel, options, message, info, notify, subscribe,
unsubscribe, update, refer, Prack
 Fix erroneous incoming messages “on the fly” with the
“search and replace” method
 Allows interoperability with SIP devices (including drafts,
non conformant, prototype)
Analyze
Assure
Accelerate
SIP Message Registration Screen
Shots
Analyze
Assure
Accelerate
SIP Message Origination
Analyze
Assure
Accelerate
SIP Message Termination
Analyze
Assure
Accelerate
Incoming Message Filters
Analyze
Assure
Accelerate
TOS for SIP Signaling
Analyze
Assure
Accelerate
Diffserv for SIP Signaling
Analyze
Assure
Accelerate
Testing SIP Robustness
• Robustness testing
– Passed: does not crash, stable, or acceptable results
– Failed: crashes, unstable, or unacceptable results
• Security testing
– It is crucial to identify SIP security holes
• SIP testing tool
– Tests SIP robustness and security
– Comprehensive negative test suites for SIP
Analyze
Assure
Accelerate
Real Signaling with real RTP
•
•
•
•
•
•
Capability to do signaling with audio
Capability to perform real time measurements
Capability of using signaling without audio
Problems of not using real signaling
Problems of not using real RTP streams
Real time objective metrics
Analyze
Assure
Accelerate
Testing SIP-T
VoIP Network
MGC
MGC
SIP-T
SS7
SS7
SIP-T
H.248/Megaco
PSTN
SIP-T
POTS
Analyze
Assure
PSTN
SIP Proxy
RTP/RTCP
SS7
GR303
ISDN
CAS
V5
H.248/Megaco
Trunking
Gateway
Trunking
Gateway
1000Base-SX/LX
10/100/1000Base-T
Accelerate
POTS
SIP-T Performance Testing Suites
Performance testing
– Validate and stress-test SS7 ISUP and SIP
interworking with optional media, over thousands of
emulated user agents
• SIT-T testing
– Configurable SIP-T calls with intelligent protocols
• QoS and CoS testing
– Optional TOS/Diffserv and VLAN options in SIP-T
media calls, used to measure QoS with PESQ and emodel
• Feature testing
– Automated and configurable SIP call set-up,
teardown, flows, messages
Analyze
Assure
Accelerate
Network Impairments and Parameters
that Affect Voice and Video Quality
•
•
•
•
•
•
•
•
•
Network Architecture
Types of Access Links
QoS controlled Edge Routing
MTU Size
Packet Loss (Frame Loss)
Out of order packets
One Way Delay (Latency)
Variable Delays (Jitter)
Background Traffic
(Congestion, Bandwidth,
Utilization, Network Load,
Load Sharing)
Analyze
Assure
Accelerate
•
•
•
•
•
•
Timing Drift
Route Flapping
Signaling protocol mismatches
Network faults
Link Failures
Voice Only Impairments
–
–
–
–
Echo
Voice coding algorithms
A/D and D/A Conversion
Noise – Circuit and External
• Video Only Impairments
– Video coding algorithms
– Fixed vs Variable Frame Rate
IP Network Architecture
Source
Device A
LAN A
Local Access
B
Local Access
A
64 kbit/s
*128 kbit/s
256 kbit/s
1000BaseX
*384 kbit/s
* 100BaseT Switch
512 kbit/s
100BaseT Hub
*768 kbit/s
10BaseT
*T1 (1.536 kbit/s)
* WLAN (~4 Mbit/s) E1 (1.920 kbit/s)
---------------------E3 (34 Mbit/s)
Occupancy level
*T3 (44 Mbit/s)
Packet loss
ADSL (~256 kbit/s)
*Cable (~256 kbit/s)
Fiber (1-10 Gbit/s)
-------------------Occupancy level
QoS edge router
Core IP Network
Route flapping
One-way delay
Jitter
Packet loss
64 kbit/s
*128 kbit/s
256 kbit/s
*384 kbit/s
512 kbit/s
*768 kbit/s
*T1 (1.536 kbit/s)
E1 (1.920 kbit/s)
E3 (34 Mbit/s)
*T3 (44 Mbit/s)
ADSL (~2 Mbit/s)
*Cable (~3 Mbit/s)
Fiber (1-10 Gbit/s)
-------------------Occupancy level
QoS edge router
* Case used in impairment tables
Affects
Data, Voice and Video Quality
Analyze
Assure
Accelerate
LAN B
1000BaseX
* 100BaseT Switch
100BaseT Hub
10BaseT
* WLAN (~4 Mbit/s)
---------------------Occupancy level
Packet loss
Destination
Device B
Network Operating With Constant Delay
Affects
Voice and Video Quality
Analyze
Assure
Accelerate
End to End Delay Sources
Core
Network
Originating
LAN
Originating
Gateway
Edge
Router
Fixed
• Encoding
• Switching
• Buffer
Variable
• VAD
• Packetizing
WAN
Fixed
• Switching
•Propagation
•Serialization
• Serialization
WAN
Fixed
Fixed
• Switching
• Decoding
Variable
Variable
• Voice contention
Variable
• Voice contention
• De-jitter buffer
• Data Contention
• Voice contention
• Data Contention
• Video Contention
• Data Contention
• Video Contention
• Packet loss
Concealment
•Video Contention
Affects
Voice and Video Quality
Analyze
Terminating
Gateway
Fixed
• Serialization
• Look ahead
Edge
Router
Core
Network
Routers
Fixed
Fixed
Terminating
LAN
Assure
Accelerate
•
•
•
•
Algorithmic delay
Serialization delay
Propagation delay
Component delay
Echo Impairment on Converged network
IP
Phone
TELR
Delay in IP Network
makes Echo sound worse
Tail Circuit
MG
T1
Link
IP Network
2 Wire
V
PBX
POTS
Phone
ERLE
IP
Phone
ERL
Echo Canceller in MG
reduces Echo Level
ERLE – Echo Return Loss Enhancement
ERL – Echo Return Loss
TELR – Talker Echo Loudness Rating
Analog
4-Wire Link
RX
T1 Link
Affects
Voice Quality
TX
RX
E&M
TX
PBX
Analyze
Assure
Accelerate
Analog
2-Wire Link
Hybrid
Transformer
POTS
Phone
Impedance
Mismatch
Echo Impairment on Converged network
Electrical Coupling
• Impedance Mismatch (Hybrid)
Acoustical Coupling
• Speakerphone
Affects
Voice Quality
Converged
Network
Path
PathABto
toBA
Echo
EchoPath
PathSide
Side
AB
(250ms)
(250ms)
Echo is caused by impedance mismatches in hybrid circuits (2w to
4w) and feedback between the telephone mouth piece and ear piece
Analyze
Assure
Accelerate
Effect of Delay on Voice Quality
Voice Quality
> 25ms Echo Cancellation Required
<150 ms (with echo
cancellation): acceptable
150-400 ms:
acceptable if
delay expected
> 400 ms
unacceptable for
most applications
Analyze
Assure
Accelerate
Effect of Echo Level on Voice Quality
Less Echo
More Echo
TELR – Talker Echo Loudness Rating
(Signal to Echo Ratio)
Analyze
Assure
Accelerate
Less Echo
More Echo
Affects
Voice Quality
Network with Variable Delays (Jitter)
• Variable processing delay
– A busy router or switch will take longer to look up the routing (address)
table
• Queuing delay
– Network congestion
Affects
Delay (ms)
Voice and Video Quality
Analyze
Assure
Accelerate
Time (s)
Bad
Severe
Delay (ms)
Good
Delay (ms) Delay (ms)
Jitter Characteristics
Time (s)
Affects
Voice and Video Quality
Analyze
Assure
Accelerate
Packet Loss
Example: Queue Management
Threshold
Affects
Voice and Video Quality
Bit Bucket
RED (Random Early Discard)
Analyze
Assure
Accelerate
Speech Compression Impairment
Voice Quality
G.711
Best Quality
Common Compression Types: G.711, G.729, G.728, G.726, G.723, AMR, EVRC
Analyze
Assure
Accelerate
VAD – Voice Activity Detection
Timing may be different
No VAD
Affects
Voice Quality
VAD
Data is intentionally not sent during times of Silence
Analyze
Assure
Accelerate
Impact Of Packet Size
Affects
Data, Voice and Video Quality
10 Bytes
= 10 ms Speech
20 Bytes
Normal size for VoIP applications
= 20 ms Speech
40 Bytes
= 40 ms Speech
80 Bytes
= 80 ms Speech
• Typically Packets are kept small for best results
• Many equipment manufacturers use dynamic packet size
to optimize for network conditions
Analyze
Assure
Accelerate
Mechanisms for Assuring QOS
Data
Voice
QoS Class
(Y.1541)
Applications (Examples)
Node Mechanisms
0
Well
Managed
Real-Time, loss sensitive,
Jitter sensitive, high
interaction (VoIP, VTC,
IPTV
Strict QoS. Guaranteed no
over subscription on links.
1
Best
Effort
Real-Time, Jitter sensitive,
interactive (VoIP, VTC).
2
Video
Triple Play
•
•
•
•
Transaction Data, Highly
Interactive, (Signaling)
Separate Queue with
preferential servicing,
Traffic grooming
Separate Queue, Drop
priority
Network Techniques
Constrained Routing and
Distance
Less constrained Routing
and Distances
Constrained Routing and
Distance
3
Transaction Data,
Interactive
Less constrained Routing
and Distances
4
Low Loss Only (Short
Transactions, Bulk
Data, Video Streaming)
Long Queue, Drop priority
Any route/path
5
Internet
Traditional Applications of
Default IP Networks
Separate Queue (lowest
priority)
Any route/path
Class of Service (COS) ITU-T Y.1541 defines the 5 classes of service and their application
Type of Services (TOS)
Affects
TOS and COS are both elements with in an IP Packet
Data, Voice and Video Quality
DIFSER and RSVP provide mechanisms to improve QOS
Analyze
Assure
Accelerate
TIA-921 and ITU-T G.NIMM
Test Profiles Based on QoS (Y.1541) Classes
Different test profiles for different Service Level Agreements (SLAs)
Impairment Type
Units
Range
Impairment Type
Units
Range
Impairment Type
Units
Range
Jitter
ms
+/- 50
Jitter
ms
+/- 75
Jitter
ms
One Way Latency
ms
50 to 100
0 to +/250
One Way Latency
ms
50 to 200
One Way Latency
ms
50 to 400
Sequential Packet Loss
#sequential
packets
Random
loss only
Sequential Packet Loss
#sequential
packets
2 to 5
Sequential Packet Loss
#sequential
packets
2 to 500
Rate of Sequential Loss
sec-1
0 to 2
Rate of Sequential Loss
sec-1
< 10-1
Rate of Sequential Loss
sec-1
Random Packet Loss
%
0 to 0.05
Random Packet Loss
%
0 to 2
Random Packet Loss
%
0 to 20
Out of Sequence Packets
%
0 to 0.001
Out of Sequence Packets
%
0 to 0.1
Out of Sequence Packets
%
0 to 20
Profile A
Well Managed Network
Table 2
Analyze
Assure
Accelerate
Profile B
Best Effort Managed Network
Table 3
Profile C
Un-Managed Network
Table 4
Early Voice Quality Testing
Analyze
Assure
Accelerate
Voice Quality Testing
• Active (Intrusive) Testing
–
–
–
–
–
Sends, Receives and compares Wave Files to measure voice quality
MOS (Mean Opinion Score)
PSQM, PSQM+ (Perceptual Speech Quality Measurement)
PESQ (Perceptual Evaluation of Speech Quality)
R-Value and J-MOS derived from PESQ
• Passive Testing
– R-Value – ITU-T P.VTQ
•
•
•
•
Measures Voice Quality on RTP Packets
Based on E-model
Japan – J-MOS
Similar Techniques can be used to measure Video Quality
– P.563 (ITU-T recommendation) 3SQM, P-Stream
• Measures Voice Quality of Voice traffic based on Audio Siginal
• Provides an estimate of PSQM
Analyze
Assure
Accelerate
Active (Intrusive)
Voice Quality Testing
MOS, PSQM, PSQM+, PESQ, R-Factor (PESQ Derived)
DUT
Send Wave Files
Receive Wave Files
Example: (ITU-T Female
Nice File with Pilot Tone)
Measures Voice Quality by Comparing Sent and Received Wave files
Sent (Green)
and Received
(Orange)
wave files
Expanded Sent
(Green) and
Received
(Orange) wave
files
PESQ Score vs Number of PESQ Measurements
Values are different for Male, Female, different
Wave Files and different Languages
Analyze
Assure
Accelerate
Passive Voice and Video Quality Testing
R-Factor/Emodel
Measure:
Video Quality
MOS-LQ
MOS-CQ
MOS-PQ
J-MOS
Network R
User R
Burst statistics
Diagnostic data
IP
Network
PSTN
E1/T1/E3/T3
/PRI/GR303,
V5,SLC96
RTP
Trunking
Gateway
RTP
Measure:
ITU-T P.VTQ
Analyze
Assure
Accelerate
Video Quality
MOS-LQ
MOS-CQ
MOS-PQ
J-MOS
Network R
User R
Burst statistics
Diagnostic data
IP Telephone
Passive Voice Quality Testing
P.563 (P-Stream, 3SQM)
RTP
DUT
Receive Audio
Estimates Voice Quality based on 3
Characteristic of Received Audio
Analyze
Assure
Accelerate
Voice Quality Measurements
Emodel
PAMS
5
P.861
PSQM/PSQM+
0
P.862
PESQ
4.5
~3.88
~3.65
~3.40
~3.13
~2.84
1
Analyze
Assure
Accelerate
6.5
-0.5
Sample Voice Quality Test Results
MOS
PSQM
PESQ
G.711
Created by inducing
packets lost
G.723.1 (6300 bps)
Comparison of Scores for G.711 and G.723.1 (6300 bps)
Analyze
Assure
Accelerate
Video Quality Measurement
• Video Compression
– Video Compress schemes affect the video quality
– H.261, H.263, H.264, VC1, MPEG-1, MPEG-2, MPEG-4,
Microsoft AVI, Windows Media (.wmv, .asf), RealMedia (.rm),
QuickTime (.mov)
• Interactive real-time applications (e.g., video
•
conferencing, voice over IP) are sensitive to latency and
Frame Rate
Typical Video Quality Metrics
– SNR
– Objective MOS
– Edge Noise
– Blockiness
– Jerkiness
– Blur
– Error Blocks
– PSNR
– Object Retention
– Spatial Resolution
– Color Reproduction
– Temporal Resolution
Accuracy
Analyze
Assure
Accelerate
Video Quality Measurement
Blockiness
Blockiness
Analyze
Assure
Accelerate
Original
Video Quality Measurement
Reference and Blocky Video
Blocky
Analyze
Assure
Accelerate
Original
(Reference)
Video Quality Measurement
Blur
Blur
Analyze
Assure
Accelerate
Original
Video Quality Measurement
Noise
Noise
Analyze
Assure
Accelerate
Original
Video Quality Measurement
Spatial (Pixel) Resolution
128X128
32X32
8X8
Spatial Resolution
Analyze
Assure
Accelerate
Department of Computer Science
University of Canterbury
http://www.cosc.canterbury.ac.nz/people/mukundan/covn/Imgresl.htm
Video Quality Measurement
Temporal (Motion) Resolution
Vertical-Width (v)
HorizontalWidth (h)
Temporal-Width (t)
Fk
Fk+1
Fk+2
Fk+3
Video Frames
Analyze
Assure
Accelerate
Fk+4
Fk+5
Video Quality Measurement
Models
• Video Quality Metrics (VQM)
•
•
•
•
•
– ITU-T SG9 and VQEG are working on standard
TV Model - optimized for higher bit-rate digital television
systems with no frame dropping (e.g., MPEG-2)
Videoconferencing Model - optimized for lower bit-rate
videoconferencing systems that drop frames (e.g., H.261,
H.263).
General Model - optimized for a wide range of video
quality (videoconferencing, TV)
Developer Model - optimized for a wide range of video
quality (videoconferencing, TV) with the added constraint
of fast computation.
PSNR Model - based on the traditional peak signal-tonoise-ratio (PSNR) calculation.
Analyze
Assure
Accelerate
Video Quality Measurement
Techniques
• Full Reference (ITU-T J.144R and BT.1683)
– Video quality is calculated by comparing the received video with
the complete original video
• Reduced Reference (ANSI T1.801.03-2003 and ITU-T
J.143)
– Spatial and temporal information are calculated from original
video and transmitted to the receiving end
– Video quality is calculated by comparing the received video with
the reduced reference
• No Reference – Passive
– Video quality on based only on the received information (picture
content)
– Video quality is derived from RTP packet information similar to RFactor (E-Model) for voice quality
Analyze
Assure
Accelerate
Video Quality Measurement
Full Reference
Original Video
Received Video
Transmitted Video Signal
Processor
Processor
Original Video
Video Quality Score
Full Reference
•MOS
•Blockiness
• Blur
• PSNR
Analyze
Assure
Accelerate
Video Quality Measurement
Reduced Reference
Transmitted Video
Received Video
Processor
Processor
Reduced-Reference Signal
Low Rate
Video Quality Score
•MOS
Reduced-Reference
•Spatial (Pixel) and Temporal (Motion) Information
•Blockiness
• Blur
• PSNR
Analyze
Assure
Accelerate
Video Quality Measurement
No-Reference
Transmitted Video
Processor
Received Video
Processor
No-Reference
Picture Content
or
Passive monitoring of RTP
Video Quality Score
•MOS
•Blockiness
• Blur
• PSNR
Analyze
Assure
Accelerate
Types of Testing
Type of tests
Test these DUTs
•
•
•
•
•
•
IP PBX
Gateways
IP Phone
Servers
Firewalls
IAD
•
•
•
•
•
•
•
•
Video Quality
Payload types
•
•
•
•
•
Voice quality
Functional
Scalability
Troubleshooting
Conformance
–
–
–
–
–
Triple Play (Data,
Voice and Video)
Analyze
Assure
• PSTN
–
H.323
–
MGCP
–
Megaco/H.248 –
Skinny
–
Accelerate
–
SIP
Voice
Data
Fax
Modem
Interoperability
With these interfaces
•
GigE 1000Base-SX,
1000Base-LX
•
•
•
•
10/100/1000Base-T
With these protocols
• IP
Video
CAS
PRI
SS7
NFAS
V5
GR303
Analog
T1/E1
T3/E3
Types of Testing
•Call Establishment
–Start Dial Signal Delay
–Post Dial Delay
–Call Duration
–Ring Duration
•Call Disconnect
–Connection Disconnect Delay
–Release on Request
•Call Statistics
–Connection set-up failures
–Connection premature disconnect
–Call completion percentages
•Transport Layer Measurements
–One-way Transmission Time
–Roundtrip Transmission Time
–Jitter
–Packets out of order
–Packet Loss
Analyze
Assure
Accelerate
•Speech Quality Measurements
–PESQ
–PSQM
–MOS
–R Factor
–Echo Delay
–Round Trip Delay
–Echo Return Loss
–Signal Pass Noise
–Noise Level
•Video Quality Measurements
–MOS
–Blockiness
– Blur
– PSNR
Video Telephony Testing
Distributed Testing
• Isolate Network Problems
• Results Over Time
Call Quality Summary
Poor
2%
Fair
24%
Bad
1%
Call Quality Summary
Nearly All Users
Dissatisfied
2%
Many Users
Dissatisfied
10%
• Results by Group
Good
NotRate
Recommended
Call Completion
by Day Fair
1%
Poor
Bad
Very
Satisfied
One Way Delay by
Call
Group
20%
Some Users
Dissatisfied
14%
Good
73%
Packet Loss by Call Group
180
100
160
140
90
1.40
120
% Complete
Call Setup Time by Call Group
% Complete
80
Mon
Tue
Wed
Thu
Fri
Sat
Sun
98
60
97
99
99
96
99
99
40
0.80
20
1.00
350
0
Delay (ms)
NYC - DAL
NYC - CHI
122
98
0.40
0.20
0.00
% Loss
Call Setup Time (ms)
0.60
300
80
250
70
Jitter by Call Group
NYC - SJ
DAL - CHI
DAL - SJ
CHI - SJ
106
173
132
112
200
60
150
50
100
NYC - CHI
0.10
50
0.43
0
Call Setup Time (ms)
100
40
NYC - DAL
NYC - SJ
0.40
30
20 - DAL
NYC
150
80
DAL - CHI
DAL - SJ
CHI - SJ
1.23
0.35
0.64
60
5.00
40- CHI
NYC
NYC - SJ
DAL - CHI
DAL - SJ
CHI - SJ
175
130
313
110
105
20
4.00
0
Jitter (ms)
Jitter (ms)
MOS
10
0
Jitter by Hour
CHI-to-DAL
Call Quality Summary by Hour
CHI-to-DAL
Jitter (ms)
75
Satisfied
53%
1.20
100
Jitter (ms)
80
% Loss
85
Delay (ms)
% Complete
95
12 1 2 33.004
A A A A A
NYC - DAL
NYC - CHI
M M M M M
43
51 2.00
Jitter (ms) 77 69 67 65 68
5 6 7
A A A
NYC - SJ
M M M
41
68 69 70
8 9 10 11 12
A A A A P
DAL - CHI
M M M M M
73
73 75 76 93 92
1
P
M
2 3 4
P P P
DAL - SJ
M M M
54
100 82 83 81
5
P
M
6 7 8
P P P
CHI - SJ
M M M
45
80 79 79 79
9 10 11
P P P
M M M
79 79 78
MOS
1.00
12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11
AM AM AM AM AM AM AM AM AM AM AM AM PM PM PM PM PM PM PM PM PM PM PM PM
MOS 3.5 3.7 3.6 3.4 3.8 3.4 3.6 3.8 3.5 2.9 2.8 2.3 1.9 1.8 2.9 2.9 3.1 2.9 3.4 3.5 3.5 3.6 3.6 3.7
Analyze
Assure
Accelerate
Good test methodology
Implementation, Validation & Observation
– Conformance testing
• IETF 3261 & new SIP RFC’s
– Stress testing
• Scriptable call flow
• Bulk signaling with real RTP
– Robustness testing
• SIPPING Torture Test
• PROTOS / ETSI TIPHON
– Visual protocol analysis
• Application & content decoding
Analyze
Assure
Accelerate
Analyze
Analyze
Assure
Accelerate
Assure
Accelerate