Transcript The SWITCHlambda Project

SWITCHlambda - Experiences with national
dark fibers @ SWITCH
Willi Huber / Urs Eppenberger
[email protected] / [email protected]
2003 © SWITCH
The SWITCHlambda Project
Project goals:
• Provide fore front network services to the
Swiss Universities at low recurring cost
• Network scalable to multiples of 10 Gbps with
low additional cost
• Long living infrastructure, no need to change
provider every couple of years
• Independence of carrier market
2003 © SWITCH
2
SWITCHlambda Project
Basic decisions:
• Replacement of the STM-1 links by GigabitEthernet
connections based on dark fiber
• Long term lease of dark fiber, indefeasible right of use
(IRU) for at least 10 years
• Use of DWDM on main trunk lines
• Redundant wide area links
• Simple network design (no SDH, no ATM): Use of
Ethernet-Interface all over the place
2003 © SWITCH
3
SWITCHlambda Network Design
• Hierarchy of technologies
– DWDM on main trunk lines (16 l possible today, expandable if later
needed)
– CWDM or single GE to smaller sites off the backbone
• Transparent optical path between Zurich and Geneva
– No regeneration between the two main sites of SWITCH
• Optical path on main trunks designed for 10 Gbps / l
– Chromatic dispersion compensated
– Polarization mode dispersion under control
• Bi-directional use of fibers
– lower fiber costs
2003 © SWITCH
4
SWITCHlambda Network end of 2003
350 km
Basel
Zurich
Bern
Lausanne
Geneva
dark fiber based links
(approx. 1200 km)
leased GE link
2003 © SWITCH
5
Bi-directional Use of Fiber
• Only one fiber used for the DWDM system
• Second fiber free to connect smaller sites inbetween
Universities with cheap GE technology
DWDM node
DWDM node
n * 1 GE / 10 GE implemented with DWDM
Router
Router
P2P GigabitEthernetconnections
Router
Router
2003 © SWITCH
One pair of fiber
6
Typical SWITCHlambda node
Bi-directional transmission mode: i.e. a single fiber is used for
signals in both directions, leaving the second fiber free for future
use.
directional
coupler
optical
amplifiers
dispersion
compensator
add/drop
circuit
to/from
site A
dispersion
compensator
optical
amplifiers
directional
coupler
to/from
site C
Rx Tx
Rx Tx
GigE
Mux/
Demux
GigE
Mux/
Demux
local
connections
site B
2003 © SWITCH
7
Sorrento Gigamux
2003 © SWITCH
8
Sorrento CWDM
JumpStart-400 CWDM System
– 8 wavelengths (4 receive, 4 transmit)
– Bi-directional operation
– Optional backup path
SWITCH uses currently 1 pair of Jumpstarts for two
p2p GE connections over on single fiber.
2003 © SWITCH
9
Bi-directional GigabitEthernet
Cheap solution for p2p GE connection over one single
fiber with colored GBICs and passive optical devices
Switch /
Router
Splitter
Signal reflection (e.g. bad connector)
Coloured
GBIC
OADM
Switch /
Router
Splitter
Coloured
GBIC
OADM
2003 © SWITCH
10
Testing 10 Gbps/channel capability
Goals
• verify equipment capability under real life conditions
• verify optical design (dispersion compensation)
• verify fiber quality for 10 Gbps application (PMD and
other adverse characteristics)
Realization
• DWDM interface cards on loan from Sorrento (10GE
WAN-PHY)
• IP equipment from Force10 (2002) and Cisco (2003)
• traffic generation with Linux based workstations and
the free software “mgen”
2003 © SWITCH
11
10 Gbps tests
10 Gbps/l, 360 km, bi-directional transmission
1. Test with signal regeneration optical - electrical optical in the middle (at Bern)
2. Test with transparent optical signal Zurich - Geneva
3. Same as 2., but with additional attenuation of 3 dB of
optical signal at the sender
Each test: 8 Gbps of traffic during 35 hours, 0 bit errors
 BER < 8 * 10-15
 SWITCHlambda ready for 10 Gbps !
2003 © SWITCH
12
Pros and Cons: Technical View
• Stable, long living
infrastructure
• High future potential
• This new stuff is fun
2003 © SWITCH
• none
13
Is this reliable?
• Stable infrastructure with occasionally long
interruptions
– Interruptions mainly due to planned maintenance
– True fiber cuts are rare, but last for hours if not days (e.g. fiber
on high voltage power lines, fiber along railway)
– Experience of a big Swiss carrier: Fiber cuts happen mostly in
the local loop (city)
• Our experiences so far
– Several interruptions due to planned maintenance (lasting up
to 1 day)
– Broken optical amplifier (interruption of 4 hours)
» Redundant links are mandatory for a good sleep!
2003 © SWITCH
14
No connection, what now?
• 24/7 monitoring by Ascom Netwatch
– ping, SNMP
– alarm SWITCH staff
– basic analysis
• SWITCH NOC for analysis and action decision
• University IT staff
– power problems
– physical equipment check
• Onsite support by Ascom Netwatch
– localize cable problems with OTDR (optical time domain
reflectometer)
– Cisco hardware
• Onsite support by Deltanet for Sorrento DWDM
2003 © SWITCH
15
Pros and Cons: Operational View
• we are the boss
2003 © SWITCH
• Big effort for negotiations
with a lot of contract
partners
• no single provider to
blame in case of
connection loss
• need a car to fix problems
instead of telnet
16
Network Cost, general Remarks
• Cost for wide area links
– Well known in advance
• Local loop cost
– Vary a lot from place to place
– Good (say cheap) solutions need a lot of planning and patience
– Building own infrastructure is economically better than renting,
but not possible in all cities
– In some cities there is a de facto monopole of the city’s industrial
works
– The railways are an excellent partner for wide area links: low local
loop cost because the schools are mostly located near railway
stations
2003 © SWITCH
17
Network Cost, general Remarks (2)
• Cost for optical equipment
– DWDM is expensive
– Long 10 Gbps/l DWDM trunks are very expensive (dispersion
compensation, amplification)
– CWDM stuff is getting cheap
• Routers / Switches
– Ethernet interfaces are cheaper by factors than SONET ones
2003 © SWITCH
18
Cost Structure
Depreciation period used for the calculations:
Fiber optic infrastructure (wide area & local loop): 10 years
Fiber optic equipment DWDM & CWDM:
5 years
12 %
6%
wide area fiber
31 %
local loop fiber
DWDM equipment
CWDM equipment
maintenance & support
7%
44 %
Total yearly cost: 2.05 MEUR
Cost include:
Investment years 2001-2004
10 Gbps between the major sites
maintenance & support
2003 © SWITCH
19
Comparison with predecessor
SWITCHng SWITCHlambda
Years covered:
1999-2002
2001-2004
Technology:
ATM/STM-1
Sites connected: 13 universities
dark fiber, GE & 10 GE
21 universities
Yearly cost:
2.05 MEUR
2003 © SWITCH
2.95 MEUR
20
Isn’t it cheapter to buy it?
We have not done a serious market review lately or
even negotiated for capacity to get the real prices
But we have a figure we assume is the best current
offer on the market for high capacity links: 100 kEUR
– limited to 300 Mbit/s
– local loop must exist
For 21 sites: 2.1 MEUR
 Similar cost as SWITCHlambda,
but less flexibility and performance.
2003 © SWITCH
21
Pros and Cons: Economic View
• Low recurring cost
• Low marginal cost for
additional bandwidth
2003 © SWITCH
• High investment
• Uncertainity about
development of
bandwidth prices
22