Transcript Layer 3 to Layer 1: Lessons Learned

Going Optical with Movaz Building and Maintaining BiSON
Scot Colburn - NCAR
WESTNET 2006
Three lessons:
• Layer 3 is math, Layer 1 is Physics.
– Layer 3 uses math, statistical tricks to share a pipe
– WDM uses physics tricks to share a fiber
• Movaz – Relatively young company, recently
acquired by Adva, sells “Heathkit” WDM nodes
• BiSON – Bi-State Optical Network connecting
Northern Colorado and Southern Wyoming.
– Phases: GBICs, CWDM, DWDM, 10G
– Motivation: BRIN, Lariat, TeraGrid, BPOP, FRGP.
6/14/2006
WestNet 2006
2
Networking moves down the stack
“There is a theory which states that if ever anyone
discovers exactly what the Universe is for and why
it is here, it will instantly disappear and be replaced
by something even more bizarre and inexplicable.
There is another theory which states that this has
already happened.”
-Douglas Adams (1952 - 2001)
6/14/2006
WestNet 2006
3
Remember 10Base5?
6/14/2006
WestNet 2006
4
10Base5/2 begot Bridges, Routers
• Broken or miscrimped connectors could break
network, failures very difficult to locate
• Had to put station taps precisely between
nodes
• Then, bridges divided L1 domains, increased
manageability. L3 divided bridges.
• 10BaseT fixed some of these multidrop issues
by being point-to-point.
6/14/2006
WestNet 2006
5
WDM – “Physic”-al networking
• Incredibly clear glass for optical fibers
• Physical magic of EDFA (Erbium Doped Fiber
Amplifier)
– But EDFA’s generate ASE noise (Amplified
Spontaneous Emission)
• Inexpensive, high quality lasers
– But jitter, modulation prevent high bit-rates
• Single mode reduces dispersion
– But bit-rates limited by CD and PMD
• All rather unrelated to layer 3 experience
– All engineering tricks to cope with physical realities
6/14/2006
WestNet 2006
6
New physical concepts, new
terminology. And Telco terms, too
• Electrical/Optical Engineering:
– dBs, power, SNR, attenuation, EDFAs, ASE
• Telco:
– Tribs, Grooming, PMs, Alarms, BITS
– In service upgrade?
6/14/2006
WestNet 2006
7
Aren’t ROADMs kinda bridges?
Are MEMs cross connects kinda routers?
• ROADMs (Reconfigurable Optical Add/Drop
Multiplexers) will make optical simple, right?
– Filter bands, equalize power, connect rings
• MEMS cross-connects will let GMPLS really
work, provision waves/paths almost the way PNNI
provisioned SVCs, right?
– WSS – Wavelength Selectable Switch
6/14/2006
WestNet 2006
8
Movaz
6/14/2006
WestNet 2006
9
Movaz OLD (Optical Line Driver)
6/14/2006
WestNet 2006
10
Movaz OLD PM (Performance Monitoring)
BLDR-MOVAZ-1> sh pm po 13-1 opt
Port <1-1-13-1> Optical PMs
Card Type: OLDUEV20 1310
PM
Value
Low Threshold
High Threshold
OPR OSC
-33.7
-38.0
-6.0
dBm
OPT OSC
1.9
0.0
4.0
dBm
LBC OSC
7
-
EDFA IP
-22.8
23.7
EDFA GAIN
OOR
Time: 1991/01/01 04:31:56 MDS
Unit
60
mA
-25.0
-2.0
dBm
22.0
24.5
dB
EDFA OP
2.3
-
-
dBm
EDFA LPA
4
0
240
mW
EDFA LBCA
28
0
800
mA
EDFA LTA
25
20
30
EDFA LPB
19
0
120
mW
EDFA LBCB
38
0
365
mA
EDFA LTB
25
20
30
degC
degC
OPR
-20.2
-
-
dBm
OPT
0.8
-
-
dBm
VOA ATTEN
1.7
-
-
dB
6/14/2006
WestNet 2006
11
MOVAZ XCVR (Transceiver)
6/14/2006
WestNet 2006
12
Movaz UDS SIM
6/14/2006
WestNet 2006
13
Movaz Shelf
6/14/2006
WestNet 2006
14
BiSON – Bi State Optical Network
6/14/2006
WestNet 2006
15
BiSON history
• Our Quest for fiber (chronological order)
–
–
–
–
Denver-Boulder from ICG for NCAR
Boulder-Fort Collins from ICG, PRPA for CSU
Laramie-Denver from Level3 for Uwyo
Laramie-Fort Collins to close Bison Ring
• Bought McCleod fiber outright, CSU repairs it
6/14/2006
WestNet 2006
16
BiSON just kinda grew up
• No WDM in first path
– Cisco CWDM (powerful) GBIC
• Movaz CWDM in second path (Bldr-CSU)
• Movaz DWDM in third path (UWyo-Dnvr)
• In fourth path, Movaz DWDM but the OSC
doesn’t reach
– No APE (Automatic Power Equalization)
– VGA OLD or Raman OLD in our future?
6/14/2006
WestNet 2006
17
BiSON
diagram
6/14/2006
WestNet 2006
18
BiSON ring design
• UWyo proposed classic DWDM redundant wave
per site. Very expensive.
– Remember that CWDM bit? Would need upgrade
• Instead, designed Layer 2 ring:
– 2 GigEs connect each site to the next.
– "If the only tool you have is a hammer, you tend to see
every problem as a nail.“ - Abraham Maslow
• In retrospect, UWyo’s design would’ve been nice
– We have potential for L2 packet loops.
– Bridged VLAN ring passes through four nodes with
three different network administrators.
6/14/2006
WestNet 2006
19
Bison L3/L2
6/14/2006
WestNet 2006
20
BiSON evolution
• Layer 2 instability
– Observed ~5000 arps/sec – switch console was quite
sluggish.
– FRGP’s connected M20 router was distressed
– Ultimately, found unidirectional path on Movaz GigE
• Solutions considered:
–
–
–
–
–
6/14/2006
Rate-limit broadcasts? Not a good Layer2 solution
Rapid spanning tree? Now on by default
MAC level filters? Looping packets likely allowed
Wait for NLR’s L2 network to figure it out?
Eliminate L2 loops
WestNet 2006
21
Next BiSON steps?
• Wyoming adding spur to Cheyenne
• CU renting 1GE for NLR access
• Probably put 10G around ring for something
– Must replace CWDM segment with DWDM
– Bridged? No - Probably non-stop 10G waves
• Or deploy dedicated 2.5 G (2xGE) waves
– Like 10BaseT is point-to-point, maybe L3 GE links
should also not be multi-drop?
• Keep existing shared bridged vlans as interBiSON links?
6/14/2006
WestNet 2006
22
Everybody gets their
own 2.5 Gbps wave
UWyo
CSU
FRGP
BPOP
6/14/2006
WestNet 2006
23
To make a Lambda redundant
network
• Must replace CWDM segment with DWDM
– $107K estimated
• “Replacement” BiSON waves cost $51K
– Assuming SIMS are reused at Level3
• New waves cost $61.6K (= $51K + $10.6K)
– UDC SIMs cost $5.3K – need two per wave
6/14/2006
WestNet 2006
24
10G wave add
• Denver-Boulder next 10G is ~$40K cheap,
because existing TeraGrid 10G blazed the
trail
• Rest of ring?
– Need $107K for CWDM conversion
– ~$45K for VGA amps for Laramie-Fort Collins
– Need to ask Movaz for quote?
6/14/2006
WestNet 2006
25