Transcript 20050503-NLRBoF-Sweeny

National LambdaRail
Layer 2 and 3 Networks
3 May 2005
Internet2 Spring Members Meeting
Layers 2 & 3 Users BOF
NLR L2 Services Summary
Goals
Provide circuit-like options for users who can’t use,
or don’t need, a 10G Layer1 wave.
Experiment on future national L2 network
capabilities.
– Create distributed nationwide broadcast domain.
– Create tools and procedures for automated and
user-controlled provisioning of L2 resources.
– Create network management tools for time-of-day
based resource reservation.
Joint Techs - Feb. 13, 2005
2
NLR Layer 2 locations
Jacksonville:
Atlanta:
Raleigh:
WashDC:
NYC:
Pittsburgh:
Cleveland:
Chicago:
Kansas City:
Denver:
Seattle:
Sunnyvale:
Los Angeles:
El Paso:
Houston:
Tulsa:
Baton Rouge:
Albuquerque:
(Sites in blue are already connected)
Level3, 814 Phillips Hwy
Level3, 345 Courtyard, Suite 9
Level3, 5301 Departure Drive
Level3, 1755/1757 Old Meadow Road Suite: 111, McLean VA.
MANLAN, 32 Avenue of the Americas, 24th Floor
Level3, 143 South 25th
Level3, 4000 Chester Avenue.
Level3, 111 N. Canal, Suite 200
Level3, 1100 Walnut Street, MO.
Level3, 1850 Pearl St, Suite 4
PacWave, 1000 Denny Way (Westin)
Level3, 1360 Kifer Road Suite: 251
Equinix, 818 W. 7th Street, 6th Floor
Wiltel, 501 W. Overland
Wiltel, 1124 Hardy St.
Wiltel, 18 W. Archer
Wiltel, No Address Yet
Level3, 104 Gold St.
Joint Techs - Feb. 13, 2005
3
Layer 2 Initial Logical Topology
SEA
CHI
SVL
DEN
NYC
PIT
CLE
KAN
WDC
RAL
LAX
ALB
TUL
ATL
ELP
JAC
Cisco 6509 switch
10GE wave
BAT
HOU
Joint Techs - Feb. 13, 2005
4
NLR L2 Hardware
Cisco Catalyst 6509-NEBs-A
Chassis Height: 36.65 in. 93.09cm
Chassis Width: 17.20 in. 43.68cm
Chassis Depth: 20.32 in. 51.61cm
Weight:
• 270 lb fully loaded
DC Power: 2 -2500 WDC power supplies
We will need 1 60A feed for each
Joint Techs - Feb. 13, 2005
5
Generic NLR L1, L2 and L3 PoP
Layout
Colo
West
15808
15454
East
6509
NLR demarc
CRS-1
DWDM
10G wave, link or port
1G wave, link or port
Joint Techs - Feb. 13, 2005
6
NLR L2 Services Summary
Initial Services
Dedicated Point to Point Ethernet – VLAN
between 2 members with dedicated
bandwidth from sub 1G to multiple 1G.
Best Effort Point to Multipoint – Multipoint VLAN
with no dedicated bandwidth.
National Peering Fabric – Create a national
distributed exchange point, with a single
broadcast domain for all members. Initial user
ports will be 1GE.
Joint Techs - Feb. 13, 2005
7
NLR L2 Services Summary
Expected Near Term Services
Dedicated Point to Multipoint : Dedicated
bandwidth for multipoint connections
Scavenger: Support less-than-best-effort
forwarding. This would be useable for all
connections.
Connections: Support 10GE user ports
Joint Techs - Feb. 13, 2005
8
NLR L2 Services Summary
Possible Long-range Services
User-controlled Web-based provisioning and
configuration Allow users to automatically create
new services, or reconfigure existing services on
the network using a web-based tool.
Time-sensitive provisioning – Allow users to
have L2 connections with bandwidth dedicated
only at certain times of day, or certain days.
Joint Techs - Feb. 13, 2005
9
Layer 3
Joint Techs - Feb. 13, 2005
10
NLR L3 Services Summary
Base Features, Day 1
BGP to each member, with their 2 connections configured
any way they’d like. Options include but aren’t limited to:
primary/backup, route-based load balanced, or 1 for
traffic/1 for measurement.
ISIS core IGP
IPv4 Multicast: PIM, MSDP, MBGP by default.
IPv6: BGP peering by default
Security: Each member will be able to black-hole traffic to
one of their hosts using special BGP community
Joint Techs - Feb. 13, 2005
11
NLR L3 Services Summary
Features Document
Day One Connection
Each member gets a 10GE connection and a VLAN
backhauled over the L2 network to a second node.
General operations of the network, including base features
(configuration with no experiment running), connections,
and communication of experiments.
Future Possibilities
Peering with other R&E networks.
Commodity Internet Connections or peering.
Joint Techs - Feb. 13, 2005
12
NLR Layer 3 locatons
Atlanta:
Level3, 345 Courtyard, Suite 9
WashDC:
Level3, 1755/7 Old Meadow Rd Suite: 111,McLean VA.
NYC:
MANLAN, 32 Avenue of the Americas, 24th floor
Chicago:
Level3, 111 N. Canal, Suite 200
Denver:
Level3, 1850 Pearl St, Suite 4
Seattle:
PacWave, 1000 Denny Way
Los Angeles: Equinix, 818 W. 7th Street, 6th Floor
Houston:
Wiltel, 1124 Hardy St.
Joint Techs - Feb. 13, 2005
13
Layer 3 Initial Logical Topology
showing planned member connections
PNWGP
SEA
NLR L2
ITN
Cornell
CIC
ITN
ITN
NLR L2
UCAR/FRGP
NYC
NLR L2
PSC
NLR L2
CHI
DEN
CENIC
LAX
MAT
P
Duke/NC
OK
UNM
NLR L2t
NLR L2
ATL
Cisco CRS-1 router
10GE wave
WDC
NLR L2
LA
LEARN
NLR L2
GATech
FLR
HOU
Joint Techs - Feb. 13, 2005
14
NLR L3 Hardware
Cisco CRS-1 Half-rack
Chassis Height: 38.5 in. 99.06cm, (with base
cosmetics)
Chassis Width: 17.5 in. 44.45 cm
Chassis Depth: 36.6 in 93.0 cm;
(40.5 in. 100.844 cm, including full cosmetics)
Weight:
330.8 lb (138 kg) chassis with fans, PDU, and
blanks (as shipped)
650 lb (294.8 kg) chassis as shipped, including
power shelves, all line cards, and route processors
Clearance: 3 feet front and back required.
Four-post mounting is strongly preferred.
DC Power: 7900 WDC
Needs 3 65A feeds per power supply
Joint Techs - Feb. 13, 2005
15
CRS1 base configuration
CRS-1 (half-rack)
2 route processors (RPs)
4 switch fabric cards
2 Power Entry Modules
2 control plane software bundle licenses (IOS-XR) with crypto
2 memory modules for each RP (required) – 2GB each
1 8x10GE line card
1 multi-service card (MSC)
1 8x10GE PLIM
1 line card software license
1 extra MSC
1 extra line card software license
8 XENPAK 10G-LR optics
fan trays and air filters
Joint Techs - Feb. 13, 2005
16
CRS1 base configuration
Sites that had at least 7 of their 8 10GE interfaces assigned
at initial installation receive a second 8x10GE, including
the MSC and software license.
Chicago
Denver
Houston
These location have a total of 12 XENPAK 10G-LR optics
modules.
We call the first configuration “A”, and the configuration
with the additional 8x10GE type “B”. The NLR layer 3
network will be comprised initially of 5 type “A” routers
and 3 type “B” routers.
Joint Techs - Feb. 13, 2005
17
Layer 3 Engineering subcommittee recommendations
•
Add logical router capability to select sites
Logical routers are a collection of line cards and Route Processors (RP) which together form a
distinct router within the chassis. In order to provide additional logical routers for hardwarelevel separation of services, additional line cards and RP resources must be installed. Logical
routers may allow multiple potentially destabilizing experiments to be run simultaneously on
the NLR network with the least
•
Add 1GE availability to select sites
This provides a lower speed and less expensive class of interface to which connectors can
attach allows for temporary connections, direct monitoring and measurement connections, and
direct attachment of reasonably priced experimental tools such as traffic generators and
protocol testers.
•
Add additional 10GE to select sites
There are three main reasons for recommending this addition. First, with the base
configuration, all but one site is already oversubscribing the slot from day 1. Also, some sites
are already completely full and have no room for expansion, even with oversubscription.
Although oversubscription may be fine for many experimental scenarios, it would be better to
have the option of oversubscription or not, depending on the project at hand. Because the
costs are so heavily loaded into the line cards, it is probably unrealistic to expect the “next”
connector to absorb the costs of moving to a second 8x10GE card, and it will be a while before
10GE is supported in SPA format. Third, similar to the 1GE line cards recommended above, it
provides a more interesting architecture with greater fanout and provides another line card
which can be used in the case of logical routers
Joint Techs - Feb. 13, 2005
18
NLR L3 Services Summary
Private Test-lab Network Connections.
Route Advertiser Connections: Get active
commodity routing table for experiments, but no
actual commodity bandwidth drainage.
Pre-emptible Connections: Allow other types of
connections to use unused ports on a temporary
basis, such as for a conference or measurement
project.
Joint Techs - Feb. 13, 2005
19
NLR L3 Services Summary
Future Options
MPLS (Traffic Engineering, L2/L3 VPNs): This can be
done on an experimental basis Day 1. May be rolled into
Base Feature list if there is general interest.
IPv6 Multicast: Will be implemented as support and
standards mature.
User-Controllable Resource Allocation: Will be
supported as experiments, and rolled into the base
feature list if there is general use and interest.
Joint Techs - Feb. 13, 2005
20
NLR L3 Services Summary
Service Expectations Document (Exhibit F, Attachment C)
Set user expectations for service on L3 network
Make clear the experiment support model
Service Expectations,
SLA isn’t good measure since the network may
appear “down” because of experiments.
Network may not have same uptime as
production network, but will have same level of
service and support as production network
Joint Techs - Feb. 13, 2005
21
NLR L3 Services Summary
Experiment Support
Each experiment will have a representative from the
L2/L3 Support Center and a representative from the
ESC.
If necessary, the prospective experiment will be sent to
the NNRC for review.
L2/L3 Services staff will be responsible for scheduling
network assets for experiments and will see the
experiment through to completion.
In general, experiments will be scheduled on a firstcome-first-served basis.
Joint Techs - Feb. 13, 2005
22
NLR L3 Services Summary
Network Conditions
Way of communicating the current state of the network to
users.
Users may choose to have their interfaces automatically
shutdown under any Network Conditions they desire.
Users will receive notification of changes to Network
Condition, with focused communication to those who will
be turned on or off because of it.
Tools will be available for users to monitor and track
Network Conditions.
Joint Techs - Feb. 13, 2005
23
Network Conditions
NetCon 7- No Experiment Currently Active
NetCon 6- Experiment Active, No Instability
Expected
NetCon 5- Possible Feature Instability/No
General Instability Expected
NetCon 4- Possible Network Instability
NetCon 3- Congestion Expected
NetCon 2- Probable Network Instability/Possible
Impact on Connecting Networks
NetCon 1- Network Completely Dedicated
Joint Techs - Feb. 13, 2005
24
NLR Layer 2/3
Service Center projects
Out of band monitoring/access network
Database development
General support applications
Specific support applications
Experimentation support
Monitoring and measurement support
Joint Techs - Feb. 13, 2005
25
Joint Techs - Feb. 13, 2005
26
Joint Techs - Feb. 13, 2005
27
Joint Techs - Feb. 13, 2005
28
Joint Techs - Feb. 13, 2005
29
NLR Layer 2/3 - discussion
What do users want/need?
Tools?
User groups?
Monitoring and measurement ability?
Full routes? Control over route propagation?
Control over protocols and timers?
Direct access to login and configure routers?
A router “ghost” service?
Instruction/workshops?
Commodity access or ISP collaboration?
Collaboration with projects like PlanetLab and WAIL?
Joint Techs - Feb. 13, 2005
30
NLR User Resources
•
•
•
•
http://noc.nlr.net
[email protected]
[email protected]
[email protected]
Joint Techs - Feb. 13, 2005
31