Transcript Case Study

Case Study
Type in a sub-title agenda here
Agenda
• Business Problem & Background
• Solution Approach
• Along the Way: Learnings
• Summary
3
Trends in Enterprise Call Centers
• Cost reduction
• Multiple Outsourcers (agents)
• Multiple Geographies (global)
• Multiple Networks (transport)
• Multiple Technologies (on call center premises)
4
Call Distribution Evolution – ’80s
Single Enterprise/
Single Site
1980’s
Avaya
Enterprise Customer Enterprise Customer
Long Distance
Network
NORTEL
ASPECT
Single enterprise &
single site ACD
Automatic Call
Distributors
Although business strategies have evolved, call center technology has not kept pace.
Many concepts developed in the early 80’s still exist in TDM and IP solutions
5
Call Distribution Evolution – ’90s
Single Enterprise/
Multiple Sites
1990’s
Genesys
Enterprise Customer Enterprise Customer
Long Distance
Network
Cisco
Network Routing & CTI
Single enterprise with
multiple call centers
In the 1990s, vendors addressed single enterprise and multi-site
routing with network routing solutions
6
Call Distribution Evolution – Now
Client Enterprise
Client Enterprise
Client Enterprise
Outsourcer Enterprise
Outsourcer Enterprise
Life has become a
Mess….
Multiple Enterprises
Multiple Sites
Multiple Networks
Multiple Technologies
Multiple Operations
Outsourcer Enterprise
New headache, needing a new aspirin
7
Implementation Models
• Black Hole
– Call sent to offshore outsourcer directly from the network
– No direct Visibility, Control or Quality
• Forced Footprint
– Outsourcer implements Enterprise chosen Technology (e.g. ICM or
Genesys)
– Provides a level of dynamic control over call routing and call reporting
(e.g. at pre-routing phase, reports of agent handling through routing
software)
• Mini-Telco
–
–
–
–
–
Bring the call into a domestic ACD
Peel off calls to go to domestic teams or offshore teams
Extensive control over routing and reporting
Can do compliance recording
Difficult to do agent monitoring
8
Vendor Management
• Outsourcer provides Reporting and Service Level
Management for Blackhole implementations
– Typically, this means end of day reports with metrics like SL,
abandons, max queue depth, etc.
• Enterprise has insight into overall call center performance
through historic reports
– Some get more depth in reporting
• These features cost more, and provided by outsourcer
based on their technology implementation
– The more features that the outsourcer provides the enterprise,
the higher the cost (software licenses for switch vendor’s
reporting software)
9
Issues
• Management (Visibility, Control, Quality) of call
delivery to captive and outsourced call centers
• Assessment of outsourcer performance
• Contract management
– Staffing level
– Customer satisfaction on service levels
• Non-capex solution required
10
Solution Requirements
•
•
•
•
•
•
An outsourcer management gateway
Matches any deployed technology
Zero footprint at Enterprise & Outsourcer
Integrates with existing call center solutions
Delivered as a managed service
Cost effective, pay as you go model
11
Solution Approach
Technology Trends
•
•
•
•
•
Migration of enterprises to converged networks
Emergence of SIP in the VoIP space
Downward cost curves of VoIP equipment
Leverageable building blocks
Standards
– VoiceXML
– CCXML
13
The Solution: Version 1
• Match demand (calls) with supply (trunks leading to
outsourcers)
• Let call center use its own ACD
• Not be responsible for transport (carriers do that)
• Get visibility into all calls
– Impossible to do this with TDM
– Being in the SIP signaling path throughout the call tells all
• Provide some control over call destinations
– Define capacity based trunks to send calls
– Perform dynamic load-balancing across trunks (towards agents)
– Tackle the set of problems that are unique to Outsourcing
• Provide enterprises with silent monitoring capability
14
Separate Application from Network
Application Servers
CCXML
SIP
Application Server executes the logic, and controls the call within the network
through CCXML documents
15
Inside the Network
HTTP/CCXML
Call Control
Gateways
Media
Gateway
SIP
Media
Servers
16
Application Servers and Networks
Application Server
SIP
CCXML
SIP
SIP
Application Server controls calls within multiple service provider networks
using CCXML documents
17
What Happened
• Funded the idea of a Management Gateway
– Nobody would think of funding yet another ACD
•
•
•
•
Built the team
Built the techology
Went to market
Customers said they needed to be able to deliver calls
from the mid-point based on agent availability
• Crucial customer feedback on viability of such a solution
– ACD reports not useful if the full lifetime of the call is not known
– Call center metrics get skewed (to excellent!) if the non ACD
queuing time not included
18
The Solution: Version 2
• Added call delivery based on agent presence
– State of line not needed
• Found first customers
• Incorporated additional market feedback to
include call transfers: blind/consult/conference
• Added many additional ACD oriented features
19
Extension to Architecture
Application Servers
CCXML
SIP
Added Agent Login and Presence from browser-based desktop applet
20
Sample Enterprise Implementation
C1
C2
305555....
ATL
AC
O (CR)
MIA
585550....
b
6M
3 VZB DS3
JFK
VPOP
1.5M
S3
ROC
770555....
b
b
6M9Mb
S1 (ES)
MCO
Sykes
IPLC
VZB MPLS
S1 (Manila)
ATL
Verizon PSTN
1 5 Mb
Internet
b
9M
6M
Sprint MPLS
OMA
402555[1-4]...
W
b
15Mb
DEN
b
6M
3 VZB DS3
4075551[5-9]..
LAX
VPOP
6 Mb
PHX
602555....
T (Manila)
6M
b
SJC
PHX
AC
Manh
4692617[0-3]..
S2 (Dom.
Rep)
Rich
21
Along the Way:
Learnings
Choice of Technologies
• Linux as a server platform
– Experience has been par excellence
• Call Control Gateway implementation
– SIP B2BUA
– Java or C++: C++
– Choice of language determined by developers available
• SIP stack
– Open source: generally bare metal, required a lot of maintenance
– Commercial stack: we chose Dynamicsoft
• Previous experience with it at Telera (Genesys)
– General experience
• Definitely required in-house maintenance
• Robustness issues under load, race conditions, etc.
23
Choice of Technologies
• Media Server
– Desired to use any available VoiceXML media server
– Conferencing features not standards based
– Chose Snowshore/Brooktrout/Cantata/Dialogic Linuxbased software media server
• Application server implementation
– Java is an excellent choice
– Scales and performs very well
• Used Cisco MGs for development
24
Deployment Experiences: Interop
• First implementation in a carrier’s network
required interop testing with Lucent Telica
• Making the CCG work well with the Telica
(Interop test)
– Straight call worked fine (as expected)
• Issues with REINVITES, codec negotiations, header contents
(E.164 number vs label), etc.
– Took longer than most, as interpretation of SIP can
cause behavioral issues
• Made changes to CCG adapt faster than Lucent could issue
patches
25
Tenant Identification & Trunks
• Every tenant in system had a unique (internal) dial prefix
• Number with dial prefix delivered to the CCG which
needed the digit string to identify SIP trunk group and
number to be presented
• Developed a re-direct server that performed number
translation, and route identification
• Also needed to deploy CCG on separate IP address per
tenant
– Carrier billing required this
– Linux supports multiple IPs: CCGs instantiated per tenant
26
Post-Deployment Experiences
– Customer encountered one-way audio issues (on
some calls)
• Isolated a specific SBC to capture traffic (a difficult task in a
carrier network) and proved that RTP stream was intelligible
in one direction
• Incidents disappeared after MG software updated
– Session timers
• Telica reinvite appearing on our second leg using Cseq 1
• Turned out to be a stack bug (and calls got dropped when
enough of these invites got dropped by stack)
27
Learnings: SIP & VoIP Products
• Interop tested with numerous SIP products:
–
–
–
–
Media Gateways
Phones (hard, soft, G.711, G.729)
Registrars
Hosted IP PBX (Broadsoft, Sylantro)
• Straight call always worked
• Reinvites as a B2BUA can be tricky to debug especially in
a carrier network
• Voice quality and echo cancellation issues are very thorny
– One fixed through vendor DSP firmware update
– A set of such intermittent problems fixed themselves when MG
updated
28
WAN Reliability
• Even the best Internet backbones have packet
loss
– Moved to MPLS after achieving a certain volume of
customers
• Application tolerance to packet loss
– Needed tuning to retry and back off
– Balance real time responsiveness with compensation
for network
29
Summary
In Conclusion
• A real Enterprise problem that needed solving
• Could not even think of tackling this problem
without SIP
• Interesting problems to solve along the way due
to initial maturity of technologies
• A lot has happened in 4 years
31
Summary
• Problem & Background
• Solution Approach
• Along the Way: Learnings
• Summary
32