Transcript NetherLight day-to-day experience

NetherLight day-to-day experience
APAN lambda networking BOF
Erik Radius
Manager Network Services
SURFnet bv
Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27
SURFnet
• Provides the Dutch National Research Network
• Not for profit company, 50 employees
• 150 connected organizations, 500,000 users
• Turnover (2002): 30 million Euro
• Infrastructure services:
– innovation paid for by government
– cost effective exploitation for higher education and
research
Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27
SURFnet5
• Partners BT and Cisco
• 15 PoPs connected by
thirty 10 Gbit/s lambdas
• Dual stack IPv4 and IPv6
since 2001
• More than 100 institutes
connected at Gbit/s level
• 500,000 users
Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27
NetherLight
• Optical Internet exchange point in Amsterdam
• Built and operated by SURFnet
• Experiments with light path provisioning in a
multi domain environment
• www.netherlight.net
Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27
NetherLight Network 2001
• 2.5Gbit/s lambda between StarLight, Chicago, USA
and NetherLight, Amsterdam, NL
• Lambda terminated on Cisco ONS15454 muxes,
– WAN side: SONET framed: OC48c
– LAN side: GbE interfaces to computer clusters
NetherLight
StarLight
GbE
2.5G lambda
GbE
GbE
Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27
NetherLight Network at iGrid2002
New York
10 Gbit/s IEEAF
Amsterdam
NetherLight
10 Gbit/s
Level3
Chicago
StarLight
2.5 Gbit/s
SURFnet
DWDM
SURFnet
2.5 Gbit/s
SURFnet
Geneva
CERN
Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27
Dwingeloo
Astron/Jive
NetherLight equipment in A’dam
Lambdas to Chicago,
Geneva, Prague etc
Ethernet switch
(Cisco 6509)
DWDM over dark fiber
to JIVE
DWDM line terminal
(Cisco ONS15252)
L1 (SONET)
multiplexer
(Cisco ONS15454)
optical cross connect
(Calient PXC)
10GBase-LR
1000BaseT
1000BaseSX
1000BaseLX
(SM fiber,
(copper)
(MM fiber,
(SM fiber,
l = 1310nm)
l = 850nm)
l = 1310nm)
Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27
Computer clusters
UvA/NIKHEF
International lightpath network 1Q2004
10 Gbit/s
New York
MANLAN
Stockholm
NorthernLight
IEEAF
10 Gbit/s
2.5 Gbit/s
10 Gbit/s
2x10
Gbit/s
Tokyo
WIDE
2.5 Gbit/s
Chicago
IEEAF
10 Gbit/s
2x10
Gbit/s
SURFnet
10 Gbit/s
Amsterdam
DWDM
SURFnet
NSF
10 Gbit/s
10 Gbit/s
2.5 Gbit/s
10
Gbit/s
10
Gbit/s
SURFnet
10 Gbit/s
Tokyo
APAN
London
UKLight
lambda service path
service
path
TechsIP
in Paradise
2004,
Honolulu / Lambda Networking BOF / Jan 27
Dwingeloo
ASTRON/JIVE
Geneva
CERN
2.5 Gbit/s
Prague
CzechLight
Day-to-day lightpath use
• Part of lambda capacity statically configured
for IP production traffic
– Router-to-router SONET circuits between Amsterdam and
Chicago for SURFnet and others (e.g. NSF link)
– Link & interface state monitored by SURFnet NOC
• Part of lambda capacity for lightpath research
community
– Changed on weekly/monthly basis upon request
– Has a ‘test’ status; is hardly monitored by NOC
Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27
NSF funded 10G lambda US - EU
Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27
Application for a lightpath
• Researchers can apply for a lightpath
by an e-mail to [email protected]
• Information in the application should include:
– Test purpose
– End points A and Z [CERN, StarLight, Stockholm]
– Interface type [1GbE: SX or LX, 10GbE WAN PHY]
– Number of lightpaths needed
– Lightpath capacity in multiples of SONET STS-6 (311Mb/s):
[STS-6 / STS-12 / STS-24 / STS-48 / STS-192] *
• Full GbE capacity: STS-24 *
– VLAN ID (if applicable)
– Start date, end date
– Contact details (e-mail, phone)
* Bandwidth granularity depends on transport equipment, e.g. Cisco ONS15454
Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27
Day-to-day operations (1)
• SURFnet NOC manages production network plus
NetherLight
– Part of NetherLight has production status
– NetherLight test circuits get lower priority than SURFnet5
IP network
• Some learning curve to get NOC engineers familiar
with transport (SONET) troubleshooting
– ONS15454 network operations: craft terminals only
• DWDM link to Dwingeloo (ASTRON): very stable, no
pro-active management of optical system yet
– DWDM (ONS15252) craft terminal is not very user-friendly
Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27
Day-to-day operations (2)
• International lambda network monitoring
– SNMP monitoring by NOC for SONET alarms and port errors
– BER level monitoring on ONS15454 SONET ports
• BER parameter is part of SLA with carriers
– GbE ports: errors mostly attributable to LAN troubles
– No IP traffic monitoring possible on SONET ports
• ONS15454 cannot look inside SONET circuits, only provide
overall quality information (BER, errored-seconds etc)
• No trouble with clocking observed yet
– SONET network should have one master clock
– ONS15454 internal Stratum III clock is sufficient,
no external clock needed
Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27
Day-to-day operations (3)
• Layer0: optical cross connect
– Calient Photonic Cross Connect, automated patch panel
– Research status only, managed by University of Amsterdam
– Topology discovery difficult
• Layer1: SONET layer
– Very flexible GbE path setups on +- biweekly basis
– 10G end-to-end also possible, for couple of months at a
time
– Multi-domain setup works very well by building half-circuits
– ONS15454 networks from Canet and SURFnet pre-wired at
StarLight
Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27
What did we learn / need to work on
• Point to point lambdas are a powerful service
– High-performance research groups are pulling at us after getting a
taste of 10GE all-the-way…
• Straightforward to build a hybrid network architecture on top
of optical:
– Some lambdas to interconnect your IP prod. routers
– Some lambdas for new high speed few-to-few usage
• Current optical-electrical-optical equipment can allocate sublambdas to individual applications
– Ethernet-over-SONET based, this will get more advanced soon
(we need to learn what LCAS can do here)
• Management is still cumbersome, learning curve for all of us
– How to make this dynamic? Or self-provisionable…?
Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27
Light Path Provisioning in SURFnet6
Operations
Management
Broker
OGSA
Applications
Web
service
Layer 3 flow
data collection
and analysis
Light Path Provisioning
via Control Plane
SURFnet6
Layer 2 / Layer 1 network
Light Path
Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27
SURFnet6
Routers
[email protected]
www.netherlight.net
www.surfnet.nl
Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27