Transcript CEN Network Technology Briefing – July 2006

CEN Network Technology
Briefing – July 2006
Briefing Agenda
• Describe UConn’s Leadership in State, National and
Regional Advanced Research and Education Networks
– Connecticut's Optical Network Backbone and Architecture
– Discussion of UConn's role in providing service to CEN users
• Overview of network content initiatives in K12, higher
education and government on these networks (online
learning, video, e-portfolio, etc)
• The relationship between the CEN, Internet2, the NOX,
Abilene, NEREN and the National Lambda Rail
CEN Services for K12 & Libraries
•
•
•
•
•
•
•
•
Every School district gets an optical drop
On Network Peering to all other CEN sites
Primary Internet Service Provider
Internet2
Firewall
Child Protection Filtering
Domain Name Service
Generally redundant links to each site
CEN Services for Higher Eds
•
•
•
•
•
•
Redundant Optical drop to every campus
On Network Peering to all other CEN sites
Optional Commodity Internet Services
Optional Internet2 services
Optional access to NEREN fabric
Future video, disaster recovery services
CEN Paying Customer Connectivity
• Who’s on Now:
– UConn (8)*
– CSU (5)*
– CommTech
System *
– Charter Oak State*
– Albertus Magnus*
– Yale *
– Trinity *
– Wesleyan *
– UNH *
– Conn College *
– USCGA *
– Rensellaer *
•
–
–
–
–
–
–
–
Sacred Heart *
U Hartford *
Fairfield *
Quinnipiac *
Mystic Aquarium *
Vbrick *
American School for
the Deaf *
– Connecticut Public
Television *
– St. Joseph’s *
– Mitchell *
Who is next:
–
–
–
–
–
St. Vincent’s
Commtech (4)
U Bridgeport
Lyme Academy
Williams School
CEN Technologies
• Optical backbone on leased dark fiber
• CWDM on congested fiber paths
• Ethernet based Network
– Large frame size capacity (MTU of 9216)
– MPLS Enabled Core for Layer-2 cut-through
– IP Multicast
– Capacity to deploy IP v6 overlay
CEN Dark Fiber Backbone
• Fibertech Networks – “On Network” Dark – Existing backbone areas where
CEN purchased by the pair
– “Lateral Build” Dark – 12 strands built for CEN with no
electronics
– Erate Leased Ethernet – Built for CEN as a GBIC
based ethernet service
• Singlemode Fiber, SMF28
– LX/LR (<10 km)
ZX/ER (10>70 km)
CEN Dark Fiber Backbone
ENFIELD
Current 1 Gbps Backbone
Current Backbone to be upgraded to 10 Gbps
Backbone Under Construction
Litchfield
Troop A
Hartford
UConn Law
BGP RR
East Hartford
DOIT
Mansfield
UConn
BGP RR
Waterbury
Rowland
Center
BGP RR
East Hartford
DOIT
New Britain
CCSU
Southbury
Troop B
Danbury
WCSU
BGP RR
STAMFORD
UConn Stamford
Meriden
Middletown
Ansonia
Bridgeport
Troop G
BGP RR
WEST HAVEN
Qwest
Norwich
Hamden
SCSU
NEW LONDON
Qwest
Hub Site Types:
• Telecom POPS (2)
– West Haven, New London
• State Police Locations (4)
– Meriden, Southbury, Litchfield, Bridgeport
• College Data Centers (9)
– Danbury, Hamden, Hartford, Storrs, Norwich,
Middletown, Stamford, Enfield, New Britain
• Borrowed Space (3) – Ansonia, Waterbury
Hub Site Specs
• Design with short fiber lateral before fiber diversity,
preferably only building entrance
• Type A Sites (Critical & typically w/ 10G)
– Powering
• 4 hours battery with automatic generator backup
• 8 hours battery
– Assured 7x24 Access
• Type B Sites (Backup Service only)
– 8 hours battery
– Less favorable access conditions
CEN Dark Fiber Backbone
ENFIELD
B
Current 1 Gbps Backbone
Current Backbone to be upgraded to 10 Gbps
Backbone Under Construction
A*
Litchfield
Troop A
Hartford
UConn Law
A
Waterbury
Rowland
Center
A
BGP RR
East Hartford
DOIT
A
A
Mansfield
UConn
BGP RR
A
A
A
BGP RR
East Hartford
DOIT
New Britain
CCSU
Southbury
Troop B
A*
Meriden
Danbury
WCSU
A
A
BGP RR
STAMFORD
UConn Stamford
B*->A
B
A
Bridgeport
Troop G
B*->A
Middletown
Ansonia
A
Norwich
A
BGP RR
WEST HAVEN
Qwest
A*
A
Hamden
SCSU
NEW LONDON
Qwest
New London
Ansonia
West Haven
Waterbury
Meriden
Backbone Architecture
•
•
•
•
Massive over-provisioning to allow multiple link failures with no service impact,
typically 10G on primary backbone
Physical and logical meshing implemented where possible
9216 MTU Size on all core links
MPLS Tag Switching on all interfaces
–
•
•
•
•
All MPLS enabled devices in OSPF Area 0 on all interfaces
BGP Peering for VPNV4 routes only to 5 geographically separated route reflectors
No policy routing, ospf weighting or access lists if possible (let traffic flow its default
path)
Prefix Management
–
–
–
–
•
MPLS TTL Propogation disabled except for troubleshooting
Global routing table only for on-network connectivity
All customer routes in virtual routing tables
Global multicast only to support MPLS MDT trees
Customer networks also prefer to use OSPF in VRF’s, not using area 0
Failure Responsiveness
–
–
Link State notification on all backbone links should force immediate routing convergence
Longest failures should be based on BGP timers
Fiber Tributary Design
• Higher Education Sites
Higher Ed
Typically 10GigE
GigE LX or ZX
Higher Ed
Hub Site
Hub Site
Higher Ed
Higher Ed
Fiber Tributary Design
• K12 Site Design
Typically 10GigE
GigE LX or ZX
K12 Site
K12 Site
K12 Site
K12 Site
Hub Site
Hub Site
Tributary Design
HIGHER ED SITES
• 7000 series software based •
routers
•
• OSPF routed /30’s per port
• Each campus dual-homed to •
two hub sites
• MPLS runs to the edge device •
• >1500 MTU
•
• BGP to the edge
•
K12 SITES
3550 series L3 switches
OSPF Shared /28’s on
backbone vlan
Up to 4 (6) sites per tributary
between two hub sites
No MPLS
1500 MTU
No BGP
Backbone Construction
• t
Level(3) Conduit Route
•
•
•
•
•
130 Mile state controlled duct
108 Strand Cable Installed
96 Singlemode
12 LEAF
48 Spliced through
• We own the cable
Firewall, Filtering & Server Block
ENRT043E
ENRT043D
CEN Firewall Group VRF Import/Export redundancy
K12-East VRF
Dark Fiber
Connected K12 Sites
K12-West VRF
Dark Fiber
Connected K12 Sites
K12-FWG43B – VRF
Aggregate for “inside
routes”
Gig Link between “inside” routers shares routes
across for redundancy
ENRT043D
Checkpoint Firewall
FWG-43B
Import
Weight
3500
NET-FWG1 – VRF
Aggregate for “inside
routes”
Import
ISP-ONNET
Weight
(VRF table for all
1000
on-network CE
addresses) ISP-QWESTVRF For Qwest ISP
ENRT156H Default
Route Only
Import
Weight
750
Import
Weight
2800
Import
Weight
3000
ISP-INTERNET2-NOX
VRF For Internet2
Routes
(12,000 routes)
ISP-WILTEL-HTFD
VRF For Wiltel ISP
Routes
(default route only)
ISP-QWEST-NL
VRF For Qwest ISP
ENRT095H Default
Route Only
ENRT043E
BGP Peers from “Outside” VRF NET-FWG1
where the Internet routes are mixed in to these
two “inside routers”. Default route is sent in to
the inside via BGP and customer network
prefixes are sent out. Import maps based on
BGP communities tagged on the inside VRF’s
assure the correct return path through the
correct firewall from the ISP vrf’s for stateful
inspection.
Wiltel GigE
G2/12
DNS #2
URL Server #2
SBC-ATM for
SBC Opteman
Links
K12-FWG43A – VRF
Aggregate for “inside
routes”
ENRT-FWG43B
Customer K12 Router
6509 Sup II / Does not run
MPLS
Runs BGP /OSPF
FWG 43B
K12-SBC
VRF for
Opteman Links
FWG 43A
ENRT-FWG43A
Customer K12 Router
6509 Sup II / Does not run
MPLS
Runs BGP /OSPF
DNS #1
URL Server
WhatsUp
Checkpoint Firewall
FWG - 43A
Import
Weight
3500
NET-FWG1 – VRF
Aggregate for “inside
routes”
Import
ISP-ONNET
Weight
(VRF table for all
1000
on-network CE
addresses) ISP-QWESTVRF For Qwest ISP
ENRT156H Default
Route Only
Import
Weight
750
Import
Weight
2800
Import
Weight
3000
ISP-INTERNET2-NOX
VRF For Internet2
Routes
(12,000 routes)
ISP-WILTEL-HTFD
VRF For Wiltel ISP
Routes
(default route only)
ISP-QWEST-NL
VRF For Qwest ISP
ENRT095H Default
Route Only
Filtering, Firewall, Server Block
• Design for Full redundancy
– Working towards no customer downtime when
a cluster fails or goes off line
– Building a business continuity function so
East Hartford can go away without customer
impact
Servers:
• Cenmon (Cricket, techsupport site, log
server, DNS)
• N2H2 Admin & N2H2 URL Servers (2)
• TFTP/FTP
• DNS Servers (2)
• Radius Servers (2)
• VOIP Server
• Firewall Management Station
Internet Services Architecture
• Currently 4 Commodity ISP’s
– Wiltel Hartford – 1 Gbps – Newark, NJ
– Qwest New London – 622 Mbps – Boston, MA
– Qwest West Haven – 622 Mbps – New York, NY
– NEREN/OSHEAN – 1 Gbps – Boston, MA
• 2 Paths to Internet2/NOX
– NEREN Storrs to NOX – 1 Gbps
– Qwest New London – OC3
CEN Dark Fiber Backbone
ENFIELD
INTERNET PROVIDER DRAINS
B
Current 1 Gbps Backbone
Current Backbone to be upgraded to 10 Gbps
Backbone Under Construction
A*
Litchfield
Troop A
Hartford
UConn Law
A
Waterbury
Rowland
Center
A
BGP RR
East Hartford
DOIT
A
A
Mansfield
UConn
BGP RR
A
A
A
BGP RR
East Hartford
DOIT
New Britain
CCSU
Southbury
Troop B
A*
Meriden
Danbury
WCSU
A
A
BGP RR
STAMFORD
UConn Stamford
B*->A
B
A
Bridgeport
Troop G
B*->A
Middletown
Ansonia
A
Norwich
A
BGP RR
WEST HAVEN
Qwest
A*
A
Hamden
SCSU
NEW LONDON
Qwest
ISP Architecture
• All ISP routing entities (VRF’s) can run to
nearest ISP egress point in event of
cohesive network collapse.
• Try not to rate limit in any instance,
customers allowed to burst within reason
• Goal is zero customer-impacting downtime
Internet Provider Load Balance
• Qwest WH
• Qwest NL
• Wiltel Htfd
•
•
•
•
•
•
•
•
•
•
Connecticut State
University
Community Colleges
UConn Health Center
CIR = 135 Mbps @
$39/mbps/mo
Backup for Wiltel
Averaging 135-140
mbps peak
All other UConn
CIR = 135 Mbps @
$39/mbps/mo
•
All K12 & Libraries
All other higher ed
campuses
CIR = 200 Mbps @
$29/mbps/mo
•
Backup for West
Haven
•
Backup for Qwest
links
•
Averaging 180 Mbps
peak
•
Averaging 600 Mbps
peak
These are our provider costs, not including salaries, benefits, program
management, NEREN, collocation, etc. Please consider confidential!
A Revolutionary
Idea in Networking
“Old North Church Project”
Northeast Research and Education Network Proposal
NEREN BUFFALO
NEREN SYRACUSE
-
-
NEREN ROCHESTER
NEREN SPRINGFIELD
-
1 Federal St
NEREN WORCESTER
474 Main St
NEREN CAMBRIDGE
Northern Crossroads Colocation Space
300 Bent St
NEREN ALBANY
194 Washington
St
NEREN
PROVIDENCE
275 Promenade
NEREN STORRS
University of Connecticut
Route 44, Mansfield, CT
NEREN NYC
32 Avenue of the Americas
NYSERnet Colocation Space
New York, New York
NEREN HARTFORD
State of Connecticut Data Center
101 E. River Dr, East Hartford
Vendor Fiber Routes
Connecticut, Rhode Island and Massachusetts have purchased the route from Manhattan
to Cambridge through Stamford, Storrs, Providence, Springfield and Albany for the Old
North Church Project
NEREN Geography
32 Avenue of the Americas,
NYC
601 West 26th Street, NYC
60 Hudson Street, NYC
230 Congress Street,
Boston
300 Bent Street, Cambridge
Along Mass Pike, Lee
Albany
375 Promenade,
Providence
450 Main Street, Worcester
54 Meadow Street, New
Haven
RT 44, Grand Union, Storrs
101 East River Drive, E.
Hartford
Stamford
Pomfret
NEREN Technology
• Currently Gigabit Ethernet from Hartford to
Boston to Springfield
• DWDM Multiplexing Planned
– 32 lambdas of minimally 2.5 Gbps capacity
– Likely 10Gbps Ethernet lambda deployment
– Some interest in Infinera O-E-O products
• Sparse network utilizing state
infrastructure for local distribution
CEN OPERATIONAL THOUGHTS
• When in doubt, broadcast it out
– Internal staff email list [email protected]
– Customer list: [email protected]
• No core changes without discussion
• Our change window is 5-7 AM with 5 day customer notice
• Edge sites more tolerant of customer requested timing
– Remember K12 Daisy-chain convergence issues.
Questions/Contact Information
John Vittner
860-622-2241
[email protected]
Robin Brown
860-622-2139
[email protected]