Transcript Migrating from Centrex to IP Telephony at Penn

Production Quality Open Source VoIP:
Next Generation Telephony at Penn
Mike Palladino & Deke Kassabian
University of Pennsylvania
Educause National 2006
October 2006
ISC Networking &
Telecommunications
Educause National 2006
Copyright Michael Palladino & Dikran Kassabian [2006]. This work
is the intellectual property of the authors. Permission is granted for this
material to be shared for non-commercial, educational purposes,
provided that this copyright statement appears on the reproduced
materials and notice is given that the copying is by permission of the
author. To disseminate otherwise or to republish requires written
permission from the authors.
ISC Networking &
Telecommunications
Educause National 2006
Agenda
• Background, Organization, Business and
Motivation Discussion – Mike Palladino
• Project and Technology – Deke Kassabian
ISC Networking &
Telecommunications
Educause National 2006
Background: Penn
• A private, Ivy League university in
Philadelphia, PA founded in 1740
• 22,000 students/4,000 faculty/13,000 staff
• 7500 students live on campus
• 9000+ students, staff and faculty live in the
surrounding community
• Health System has separate management
– IT division outsourced/15,000 users
ISC Networking &
Telecommunications
Educause National 2006
Background: Information Systems &
Computing
• ISC is the central computing group
– 300 technical staff/contractors
• CIO (Robin Beck) reports to Provost and EVP for
administration
– Three main groups
• Administrative Systems
• Support & Security
• Networking & Telecommunications
ISC Networking &
Telecommunications
Educause National 2006
Background: Networking &
Telecommunications
•
•
•
•
•
•
100 technical staff/contractors
100% direct charge budget
25,000 phone lines/12,000 voicemail users
50,000 IP addresses/14,000 email accounts
9,000 analog video connections
40+ services from wiring, to digital video, to R&D for
emerging technologies
• Internet2 GigaPop (MAGPI)/38 institutional subscribers
ISC Networking &
Telecommunications
Educause National 2006
Our Vision
• A unified staff delivering excellent, integrated
(data, voice and video) systems and services
over one converged, highly reliable, robust IP
network capable of handling today’s
applications and tomorrow’s innovations.
ISC Networking &
Telecommunications
Educause National 2006
Leadership Role-Convergence Readiness
• Organizational
• Technical
• Financial
ISC Networking &
Telecommunications
Educause National 2006
Organization
• Networking, Telecommunications and Video
Groups merged in 2000
• Much staff re-education has taken place
• In the past 6 years, 109 staff/contractors changed
• New organization is nearly fully integrated and
increasingly well cross-trained
ISC Networking &
Telecommunications
Educause National 2006
Organizational Convergence
• Consciously avoided silos through integration of
data, voice and video staff with similar jobs
– Centralized Finance Group
– Consolidated asset and contract management
– Centralized help desk
– Single Operations & Infrastructure Groups
– Single Video Group (analog, streaming, digital,
production, podcasts)
– Single Network Management Group
– Emerging services group developing
ISC Networking &
Telecommunications
Educause National 2006
Technical Convergence
• Implemented through Next Generation PennNet
Project
– Provide infrastructure foundation for next generation
data, voice and video services
– Eliminate single points of failure
– Network redundancy to every campus building
– Reduce catastrophic disaster recovery time from 2
weeks to under 2 hours
– Self-healing routing core and services
ISC Networking &
Telecommunications
Educause National 2006
Campus Map
ISC Networking &
Telecommunications
Educause National 2006
Next Generation PennNet
NAP4
NORTHERN TIER
SANSOM EAST
NAP5
WESTERN TIER
LEVY
NAP2
CENTRAL TIER
HUNTSMAN
NAP3
SOUTHERN TIER
MOD5
ISC Networking &
Telecommunications
Educause National 2006
NAP-CH
COLLEGE HALL
NODE ROOM
NAP1
EASTERN TIER
VAGELOS
Financial Convergence
•
•
•
•
•
•
All budgets consolidated into one
100% direct charge budget
No capital budget
Replacement cycles built into the charges
All services financially self-sufficient.
User participation & full budget transparency
through Network Planning Task Force
(www.upenn.edu/computing/group/nptf)
ISC Networking &
Telecommunications
Educause National 2006
Voice Services at Penn Today
• Most users have Centrex service
– Local Exchange Carrier -provided business
telephone service
– Long list of available features
– Switch at Telco Central Office
– On-campus copper cabling
• Some departments have PBXs and small
key systems
ISC Networking &
Telecommunications
Educause National 2006
Telecommunications Infrastructure
• Local Exchange Carrier brings copper loop infrastructure
to campus Rate Demarcation Points (RDPs)
• Penn maintains in-ground and in-building copper cable
infrastructure from RDP to building, building to closet,
closet to wallplate
• This infrastructure is very old, and in some cases is failing
resulting in outages and requiring expensive repairs
• Replacement of these cables would be $3-5M
ISC Networking &
Telecommunications
Educause National 2006
ISC Networking &
Telecommunications
Educause National 2006
Rate Demarcation Points
ISC Networking &
Telecommunications
Educause National 2006
Business Drivers
• Aging phone-cabling infrastructure
– In-ground cabling is 25-75 years old
– Cable faults are increasing
– Cable repairs are expensive and time consuming
• Operational inefficiencies
– Separate cabling infrastructure
– Moves, adds and changes are time consuming and
costly (3000/yr @ $125= $375k)
• Legacy services costly to support
– ACD
– Voice mail
ISC Networking &
Telecommunications
Educause National 2006
Business Drivers (Continued)
• Vendor Management
–
–
–
–
–
–
Service delivery issues
High usage rates
Limited application integration/flexibility
Prefer vendor independence
Prefer open standards
Prefer open source
ISC Networking &
Telecommunications
Educause National 2006
Voice Services Goals
• To convert 25,000 analog voice customers to
Integrated Communications (VoIP, Voicemail, etc.)
on a converged IP network with added
functionality and lower costs in 5 years or less
• Cost–effective, reliable voice communications
• Vendor independence
• Single integrated network infrastructure
• Flexibility and new functionality to meet evolving
demands
ISC Networking &
Telecommunications
Educause National 2006
Voice Services Strategy
• Negotiate low-cost phone line and usage contracts
• Develop architecture to enable competition among
service providers for connectivity and usage
• Eliminate legacy voice services and position for
Integrated Communications (ACDs, Voicemail, etc.)
• Programs with 2 cellular phone service providers
– Over $125k of revenue-sharing annually
– Over $300k savings annually to Penn community
ISC Networking &
Telecommunications
Educause National 2006
Leadership Challenges: VoIP
• Penn’s research into integrated communication and VoIP
started in late 1990s
• There was a lot of activity with minimal progress
• Focused leadership was required
– Integrated Communications was made our top
departmental priority
– A program director whose sole responsibility was the
VoIP project was hired
– Project teams were immediately set up
ISC Networking &
Telecommunications
Educause National 2006
Project and Technology
Deke Kassabian
Director, Engineering & Services
University of Pennsylvania
ISC Networking &
Telecommunications
Educause National 2006
History of VoIP at Penn
• Protocol research dates back to late 1990s
• Worked first with H.323-based VoIP, and later with
Session Initiation Protocol (SIP)
• Worked with a range of applications:
– Cisco Call Manager, an enterprise solution (2001-2002)
– BroadSoft’s Broadworks, a carrier-class VoIP software
solution, piloted on campus (2001-2003)
– Externally hosted IP Centrex solutions (2003-2004)
– Open-source SIP Express Router with Asterisk Comedian
Voice Mail (2003-Present)
ISC Networking &
Telecommunications
Educause National 2006
Key Open Source Question
• Does IP Telephony lend itself to open source server
software?
– Can it be reliable enough?
– Will we have the feature set we need?
– Will we be able to maintain it well?
• Experience with open source server software for Email and
Web services encouraged us to take this as a serious option
ISC Networking &
Telecommunications
Educause National 2006
The Breakthrough
• Held a VoIP Project launch meeting with
managers and team leads
• Identified biggest issues
• Committed significant local staff resources
• Hired a project manager
ISC Networking &
Telecommunications
Educause National 2006
Project Teams
Steering
Committee
Technical
Facilities
Process
Business
Network
Architecture
Installations
Operations
Application
Architecture
Intake
Support
Telephony
Architecture
Billing
Communications
Provisioning
Penn iPhone
ISC Networking &
Telecommunications
Educause National 2006
Open Source Strategy
• Avoid over-customization
– Use existing functionality where possible
– Maintain active relationship with developers
– Try to get any essential changes of general interest
built back into mainline code
• Notable Asterisk customization
– IMAP access to voice mail message store for true
unified messaging from phones and email clients
ISC Networking &
Telecommunications
Educause National 2006
VoIP Building Blocks
•
•
•
•
SIP Clients or User Agents
SIP Proxy Servers
Media & Feature Servers
Gateways and ITSP
services
ISC Networking &
Telecommunications
Educause National 2006
SIP User Agents (UAs)
• Hardware SIP phone, in the
familiar desk set form factor.
Penn currently uses the
Cisco 7940 phone.
• Software that acts as a SIP
client or user agent and runs
on a desktop, laptop or
handheld. Shown here is
eyeBeam from Counterpath.
ISC Networking &
Telecommunications
Educause National 2006
SIP Proxy Servers
• Server that accepts “registration” from valid
users
• Allows client to signal their call information
• Handles call set up and gets out of the way
• Penn uses SER (iptel.org)
and OpenSER (openser.org)
ISC Networking &
Telecommunications
Educause National 2006
Media & Feature Servers
• Server that handles “media” such as voice
mail and music on hold
• Provides for some additional call features
• Co-exists with provisioning services
• Penn currently uses Asterisk
(www.digium.com)
ISC Networking &
Telecommunications
Educause National 2006
Gateway
IP
Gateway
• Interconnect between campus network and PSTN
• Penn currently uses a Cisco 36xx router
Campus
Network
ISC Networking &
Telecommunications
ISDN PRIs
PSTN
Educause National 2006
General VoIP Design
ISC Networking &
Telecommunications
Educause National 2006
VoIP Topology
ISC Networking &
Telecommunications
Educause National 2006
Penn iPhone Roadmap
1H 2006
Phase 1.0
2H 2006
Phase 1.X
1H 2007
Phase 2.0
2H 2007
Phase 3.0
Phase 3.0
• Octel voice mail migration
Phase 2.0
• Feature Release 2.0
• Softphone Pilots - I
Phase 1.0
• VoIP Campus Pilot Begins
• Penn iPhone web portal
• Penn Directory lookups
from iPhone
Phase 1.X
• Centrex-Asterisk Pilot(s)
• IMAP Message Store
• ITSP/LNP Production
• Limited Multi-line Pilots
• New Handset Testing
• PiPS Maintenance
• Security Recommendations
• Handset Recommendations
• 600+ phones deployed
Educause National 2006
• New Handset Production
• Softphone Pilots - II
• iPhone Communications
• Multi-line Pilots
• 2000 phones deployed
General
Availability
05
20 -0
05 1
20 -0
05 2
20 -0
05 3
20 -0
05 4
20 -0
05 5
20 -0
05 6
20 -0
05 7
20 -0
05 8
20 -0
05 9
20 -1
05 0
20 -1
05 1
20 -1
06 2
20 -0
06 1
20 -0
06 2
20 -0
06 3
20 -0
06 4
20 -0
06 5
20 -0
06 6
20 -0
06 7
20 -0
06 8
-0
9
20
total iPhones
Campus-wide Pilot Rollout
Deployed Penn iPhones
800
700
600
500
400
300
200
100
0
Month
ISC Networking &
Telecommunications
Educause National 2006
Current State of Service
• Production-grade
– Redundant servers, gateways and PRIs
– Single-line features, email/voicemail integration
– 911 support equal to legacy system
• Layer 2 QoS, separate VLANs & subnets
• Full service- installation, support & billing
ISC Networking &
Telecommunications
Educause National 2006
Managing Penn iPhone
• Web “provisioning interface” to allow
– Installation staff to configure phones for
deployment
– Support staff to provide local support
– End-users to select and maintain configuration
options
ISC Networking &
Telecommunications
Educause National 2006
Penn iPhone Web Services
(Phase 1.0)
Self-service
Voice mail
Change Email
Password
Do Not Disturb
Call Forward
Reject Anon.
Blocked Caller ID
More . . .
ISC Networking &
Telecommunications
Educause National 2006
ISC Networking &
Telecommunications
Educause National 2006
ISC Networking &
Telecommunications
Educause National 2006
Unified Messaging
• Some users prefer getting voice mail in email.
Others really need to access voice mail by phone.
• Wanted to access voice mail via email and via the
telephone without message replication
• Penn-developed extension to Asterisk for IMAP access to
voice mail for true unified messaging from phones and
email clients
– included in version 1.4 with Fall 2006 release
ISC Networking &
Telecommunications
Educause National 2006
Emergency Services & Location
• Ability to route 511/911 calls, including
support for location information
• Three phases towards improvement
– Phase 1 - Fixed location phones
– Phase 2 - VoIP phones movable by IT Staff,
with web form location reporting
– Phase 3 - Dynamic location updates
ISC Networking &
Telecommunications
Educause National 2006
ISC Networking &
Telecommunications
Educause National 2006
Security & Privacy
• Key project elements for S&P
– Configuration security - protecting phone
configuration and user details
– Gateway security - securing gateways so
only authorized users make calls
– Call privacy - minimizing the eavesdrop
risks
ISC Networking &
Telecommunications
Educause National 2006
Current Development Projects
• Multi-line and ring-group support
• Using ITSPs (Internet Telephony Service
Providers) for off campus calls
• Migrating Centrex users to our next
generation voice mail
• Power Over Ethernet (PoE) network designs
• Evaluating additional VoIP handsets
ISC Networking &
Telecommunications
Educause National 2006
New Roles for Campus IT Staff?
• Local tech support staff troubleshoot basic
connectivity, occasionally power cycle phones
• Submit trouble tickets when necessary
• Act as proxy for the user in the web interface,
to change feature settings and forwarding
numbers
ISC Networking &
Telecommunications
Educause National 2006
Collaboration within Higher-Ed
• Several Internet2 universities are using a
very similar approach
• Sharing plans, code and best practices
through monthly collaboration calls
• Open to participation by other groups
seriously engaged in similar efforts
ISC Networking &
Telecommunications
Educause National 2006
A Few Key Challenges
• “Recruiting” pilot customers that fit the service
profile, are in a suitable location, and can provide
detailed feedback
• Identifying supportable, high function VoIP
handsets
• Managing very substantial staff workload
• Moving fast enough to avoid costly telephone
cable upgrades
ISC Networking &
Telecommunications
Educause National 2006
Lessons & Recommendations
• Make project a priority across the organization
• Cross-train staff, and allow time for
adjustment. Augment staff with consultants in
key areas
• Pilot one technology at a time (open-source
vs. in house commercial vs IP centrex)
• Create/maintain a voice development
environment
ISC Networking &
Telecommunications
Educause National 2006
Lessons & Recommendations
•
•
•
•
Roll out pilot and new features in your own
department first, as a pre-pilot
Replace existing phones with VoIP phones
on users desk (don’t simply add)
Structure installations to be “cookie cutter”
Pilot not only technology, but full support
and communications process. Involve all
service delivery and support personnel
ISC Networking &
Telecommunications
Educause National 2006
Looking To The Future
• With the elimination of legacy services and
networks, we anticipate:
–
–
–
–
–
Increased reliability
Improved functionality
Higher levels of operational efficiencies
Telecommunications cost reductions of 10-30%
Better customer service
ISC Networking &
Telecommunications
Educause National 2006
Production Quality Open Source VoIP:
Next Generation Telephony at Penn
Mike Palladino & Deke Kassabian
University of Pennsylvania
Educause National 2006
October 2006
ISC Networking &
Telecommunications
Educause National 2006