Transcript Some Future Trends

CANARIE
http://www.canarie.ca
CA*net 4 Design Document
Last Revised April 22 2001
Version 1.20
OBGP documentation and latest version of this
document can be found at
http://www.canet3.net
[email protected]
Tel: +1.613.785.0426
The Concept for CA*net 4
 Conventional optical networks are built on the paradigm that a central entity
has control and management of the wavelengths
 It therefore must have control of the edge device for the setup and tear down of
the wavelengths
 Will central control and management scale to millions of edge device and
thousands of optical wavelengths?
 Customer empowered optical networks are built on the paradigm that customer
owns and controls the wavelengths (Virtual Dark Fiber)
 Customer controls the setup, tear down and routing of the wavelength between
itself and other customers
 Customer may trade and swap wavelengths with other like minded customers
ultimately leading to wavelengths as market commodity
 How do you design a network architecture if the routing and control of
wavelengths is under the control of the customer at the edge?
 Network is now an asset, rather than a service
 Analogy to time sharing computing in the early 1970s versus customer owned
computers or client-server computing
Condo Fiber & Wavelengths
 Condo fiber means that separate organizations own individual strands of
fiber in a fiber cable
 Each strand owner responsible for lighting up the strand
 Collectively responsible for sharing costs of maintenance on fiber cable,
relocation etc
 Condo wavelengths
 Number of parties share in the cost of a single strand and that light up
with an agreed upon number of wavelengths
 Wavelengths are portioned based on percentage ownership
 With condo fiber and condo wavelengths institutions can treat network as an
asset just like purchasing a computer, rather than a service as today
Research Network Issues
 Research and Education networks must be at forefront of new network
architecture and technologies
 Should be undertaking network technology development that is well ahead
of any commercial interest
 But any network architecture can only be validated by connecting real users
with real applications and must solve real world problems
 Test networks per se are not sufficient
 There is a growing trend for many schools, universities and businesses to
control and manage their own dark fiber
 Can we extend this concept so that they can also own and manage their
own wavelengths?
 Will “empowering” customers to control and manage their own networks result
in new applications and services similar to how the PC empowered users to
develop new computing applications?
CA*net 4 Research Objective
 To deploy a network architecture where the GigaPOPs and institutions at the
edge manage and control their own fiber and their own wavelengths
 Condominium fiber and condominium wavelengths
 To deploy a novel new optical network of distributed optical IXs that gives
GigaPOPs and communities at the edge of the network (and ultimately their
participating institutions) the ability to setup and manage their own wavelengths
across the network and thus allow direct peering between GigaPOPs on
dedicated wavelengths and optical cross connects that they control and manage
 To allow the establishment of wavelengths by the GigaPOPs and their
participating institutions in support of eScience and grid applications to support
true peer-to-peer networking
 To allow connected regional and community networks to setup peering
relationships with CA*net 4 for collaborative research and education and
eScience applications
 To partner with private sector in building “carrier neutral” distributed optical
Internet exchange facilities across Canada and developing new services in
fungible wavelengths to enable customer empowered optical networks
Current View of Optical Internets
ISP
AS 4
Carrier controls and
manages edge devices
AS 1
Optical VLAN
NNI
AS 1
Customer
AS 5
AS 3
UNI
Big Carrier Optical Cloud using MPS or
ASON for management of wavelengths
for provisioning, restoral and protection
AS 2
Customer Empowered Metro
Network
City C
City A
OBGP switch
Carrier Neutral IX
& OBGP switch
Condo Wavelengths
Carrier Neutral IX &
OBGP switch
City B
OBGP switch
Condo Dark Fiber
Condo Wavelengths
Future Optical Networks
Massive peering at the edge
Customer D
Customer A
Condo Wavelength
Customr A elects to cross
connect with Customer C
rather than D
Customer C
Customer B
Condo Fiber
CA*net 4 Research Areas
 New optical technologies that support customer empower networking
 OBGP, CWDM, hybrid optics and HWDM, customer controlled optical
switches
 BGP scaling issues
 Object Oriented Networking
 Wavelengths and optical switch treated as an object and method to be
incorporate into middleware
 Or treated as fungible product
 Distributed Computing Applications and Grids
 Wavelength Disk Drives (WDD)
 eScience
 Grids for weather forecasting, forestry management, education, health,
etc
Object Oriented Networking
 Combines concepts of Active Networks and Grids
 See DARPA
 See Globus
 Customer owns sets of wavelengths and cross connects on an optical switch
 Network elements can be treated as a set of objects in software
applications or grids
 Complete with inheritances and classes, etc
 Rather than distributed network objects ( e.g. Java or Corba) distributed
object networks
 In future researchers will purchase networks just like super computers,
telescopes or other big science equipment
 Networks will be an asset – not a service
 Will be able to trade swap and sell wavelengths and optical cross connects
on commodity markets
Advantages of OON
 With massive peerings to the edge, the loss of one peer is not catastrophic
 No need for restoral or protection paths or ring architectures
 Networks look more like “star bursts” rather than “ring of rings”
 See C Labovitz ACM Sigcomm Aug 2000 – massive peering helps
faster convergence
 May solve problem of scaling large networks
 Today M carriers building meshed networks to N customers with
resultant M*N2 requirement for wavelengths
 With OON the global requirement for wavelengths grows at X*N where
X= average number of wavelengths per customer
Example OON
 Earthquake Visualization Grid
 Globus Middleware Begin
 Establish connection to other grid participants
 Network Object – wavelength to STAR LIGHT – Chicago
 Network Object – wavelength to Research center Amsterdam
 Network Object – wavelength to SDSC Visualization Computer
 Network Object – wavelength to Seismology Center Calgary
 Link objects and create grid
 Run Visualization
 Release Network objects
 Globus Middleware End
 Earthquake Visulization End
Napster OON
 University in Canada
willing to exchange these
wavelengths
 Montreal-NY blue
 NY-Amsterdam red
 Chicago-Montreal green
 Wants these wavelengths
 Chicago-SDSC - purple
 SDSC- Hawaii - red
 Chicago- Chile - yellow
 University in Chicago
willing to swap these
wavelengths
 Chicago to SDSC – purple
 SDSC- Hawaii – Red
 Chicago to Argonne - Blue
 Wants wavelength to
these locations
 Chicago to Toronto - yellow
The eScience Vision
 “National grand challenge" e-research projects are on the
horizon: with the next generation network, interconnecting to
school and community networks, Canadian researchers could use
the thousands of computers in schools and communities
distributed across Canada
 Students at schools and ultimately members of the public could
be full participants in basic reserach
 The next generation research network should be designed to
encourage and enable projects such as these
Wavelength Disk Drives
St. John’s
Regina
Calgary
CA*net 3/4
Winnipeg
Charlottetown
Montreal
Fredericton
Vancouver
WDD Node
Halifax
Toronto
Ottawa
Computer data continuously circulates around the WDD
eScience Grid
WDD Grid
Customer D
Customer A
Customers autonomously create
WDD ring for high
performance applications
Customer C
Customer B
Wavelength Disk Drives
 CA*net 4 will be “nation wide” virtual disk drive for grid
applications
 Big challenges with grids or distributed computers is
performance of sending data over the Internet
 TCP performance problems
 Congestion
 Rather than networks being used for “communications” they
will be a temporary storage device
 Ideal for “processor stealing” applications
OBGP
 Proposed new protocol to support control and management of wavelengths
and optical switch ports
 Control of optical routing and switches across an optical cloud is by the
customer – not the carrier – true peer to peer optical networking
 Use establishment of BGP neighbors or peers at network
configuration stage for process to establish light path cross connects
 Customers control of portions of OXC which becomes part of their
AS
 Optical cross connects look like BGP speaking peers – serves as a
proxy for link connection, loopback address, etc
 Traditional BGP gives no indication of route congestion or QoS, but with
DWDM wave lengths edge router will have a simple QoS path of
guaranteed bandwidth
 Wavelengths will become new instrument for settlement and exchange
eventually leading to futures market in wavelengths
 May allow smaller ISPs and R&E networks to route around large ISPs that
dominate the Internet by massive direct peerings with like minded
networks
Opportunity for carrier and
industry partners
 To participate in a novel new Internet architecture that will allow
customers to manage and control their own wavelengths anywhere across
the network
 Very attractive technology for Tier 2 ISP, research networks and ASPs
 Yahoo and Cable and Wireless have already started down this path
 It will allow them to create their own network topologies
 To provide a valuable new service for customers that will allow them to
reduce Internet transit costs by as much as 75%
 To develop new value added services in IX brokering and management
 To develop new fungible trading services in bandwidth trading and
brokering
 To experiment with new long haul optical technologies that will
dramatically reduce cost of long haul transmission
CA*net 4 Possible Architecture
Layer 3 aggregation service
Optional Service Available to any GigaPOP
St. John’s
Calgary Regina
Winnipeg
Large channel
WDM system
Charlottetown
Vancouver
Europe
Montreal
Customer controlled
Seattle optical switches
Fredericton
Halifax
Ottawa
Chicago
Toronto
New York
Wavelength Scenarios
Workstation to Workstation Wavelength
CWDM
GigaPOP to GigaPOP Wavelength
Regina
BCnet
Vancouver
Campus
OBGP
switch
Winnipeg
St. John’s
RISQ
Halifax
Calgary
Montreal
Seattle
Toronto
Wavelength Setup
AS 2- AS 5 Peer
AS 3
12
10
University
Regional Network
3
13
AS 1
2
15
4
AS 5
14
AS 1- AS 6 Peer
AS 2
5
7
9
1
AS 4
Regional Network
Dark Fiber
8
AS 6
6
ISP router
Wavelength Object owned by primary customer
Wavelength Subcontracted by primary customer to a third party
University
Wavelength Logical Mapping
AS 2- AS 5 Peer
AS 3
12
10
University
Regional Network
3
13
AS 1
2
15
4
AS 5
14
AS 1- AS 6 Peer
AS 2
5
7
9
1
AS 4
Regional Network
Primary Route
Backup Route
8
AS 6
6
University
ISP router
Resultant Network Topologies
University
BGP Peering on switches
at the edge
Packet Forwarding in
the core
9
AS 1
15
Regional Network
13
12
8
AS 5
1
2
10
7
2
3
10
1
7
14
9
AS 2
8
AS 6
5
6
5
Regional Network
Potential OBGP Peering
ISP router
University
Possible CA*net 4 Node
Optional Aggregating Router
8 Channel GbE CWDM
to next CA*net 4 node
4 Channel GbE CWDM
to local GigaPOP
CA*net 4 switch
2xGbE
10xGbE
Carrier A
OC48 DWDM
Carrier B
OC192 DWDM
Carrier A
OBGP
Switch
Physical Wavelength Configs
ORAN A
ORAN B
ORAN C
ORAN D
OBGP links
Dark fiber +CWDM
CA*net 4
1
4
3
2
2xGbE
2xGbE
6
OC48 DWDM
7
Carrier A
5
10xGbE
OC192 DWDM
10xGbE
2xGbE
8
Carrier B
Logical Wavelength Configs
ORAN A
ORAN B
ORAN C
ORAN D
OBGP links
CA*net 4
1
GbE over CWDM
3
2
4
Carrier A
GbE over 2GbE over OC-48 DWDM
5
Carrier B
GbE over 10GbE over OC-192 DWDM
Possible Wavelength Assignment
 Illustrative purposes only
 Assume 150 wavelength system across Canada
 50 wavelengths assigned to provincial networks based on a number of
criteria including ability to extend wavelengths into provincial network and
requirements for high bandwidth applications
 ORANs encouraged to extend wavelengths to individual institutions
 Institutions encouraged to deploy optical switches
 On all cross sections a minimum of 100 wavelengths dedicated to CA*net 4
and carrier partners
 2 wavelengths dedicated to CA*net 4 layer 3 aggregation service (looks
like old CA*net 3)
 10 wavelengths (and OCX ports) reserved for temporary applications
like Grids or eScience
 A wavelength and OXC port bartering and exchange mechanism so that
ORANs can swap wavelengths will be an important requirement
Example Physical Architecture











CANARIE builds heterogeneous network made from many sources e.g (illustrative
purposes only):
dark fiber from St. John to Halifax using ULH 16 channel POS
dark fiber condo from Halifax to Fredericton using 16 channel 10GbE
Condo wavelengths from RISQ from Edmonston to Ottawa sharing 32 channel 10GbE
system
dark fiber from Ottawa to Winnipeg with Onet using 16 channel POS at OC-192
Condo wavelengths from Bell Canada from Montreal to Chicago as part of a 140
wavelength system
wavelengths from Telus from Chicago to Winnipeg as part of a 140 wavelength system
dark fiber from GT Telecom from Winnipeg to Calgary using 16 channel 10GbE
wavelengths from Shaw from Calgary to Vancouver as part of a 32 channel 10GbE
systems
wavelengths from 360 Networks from Halifax to London as part of a 400 wavelength
system
Wavelengths from Teleglobe from Seattle to Honolulu – Sydney – Tokyo - Seoul
OBGP Variations
1.
OBGP Cut Thru

2.
OBGP Optical Peering

3.
External router controls one or more switch ports so that it can establish direct
light path connections with other devices in support peering etc
OBGP Optical Transit or QoS

4.
OBGP router controls the switch ports in order to establishes an optical cut
through path in response to an external request from another router or to carry
out local optimization in order to move high traffic flows to the OXC
To support end to end setup and tear down of optical wavelengths in support of
QoS applications or peer to peer network applications
OBGP Large Scale

To prototype the technology and management issues of scaling large Internet
networks where the network cloud is broken into customer empowered BGP
regions and treated as independent customers
OBGP Optical Peering
 Primary intent is to automate BGP peering process and patch panel process
 Operator initiates process by click and point to potential peer
 Original St. Arnaud concept
 Uses only option field in OPEN messages
 Requires initial BGP OPEN message for discovery of OBGP neighbors
 Virtual BGP routers are established for every OXC and new peering
relationships are established with new BGP OPEN message
 Full routing tables are not required for each virtual router
 No changes to UPDATE messages
 No optical transit as all wavelengths are owned by peer
 Uses ARP proxy for routers on different subnets
 Wade Hong Objects concept







Uses an external box (or process) to setup optical cross connects
SSH is used to query source router of AS path to destination router
Each optical cross connect is treated as an object with names given by AS path
Recursive queries are made to objects to discover optical path, reserve and setup
NEXT_HOP at source router is modified through SSH
End result is a direct peer and intermediate ASs disappear
Requires all devices to be on same subnet
OBGP Optical Transit
 Wavelengths are under control of another entity who has temporarily allowed
them to be available for transit
 Viagenie – Marc Blanchet and Florent Parent
 Designed specifically for optical transit applications
 Uses MBGP and establishes new address family for OBGP
 Community tags are used to advertise availability of optical paths as part
of NLRI and COMMUNITY TAG
 Reservation and setup is done by advertising update NLRI message
 Exploring using CR-LDP & RSVP-TE with AS loose routing for path
reservation and setup
 Changcheng Huang
 The same NLRI message is sent back and forth and modified to indicate
first availability of wavelengths, reservation and setup
 Over rides loop back detection in RIBS for advertised NLRI messages
Target Market for OBGP
 University research and community networks who are deploying
condominium fiber networks who want to exchange traffic between members
of the community but who want to maintain customer control of the network
at the edge and avoid recreating the need for aggregating traffic via traditional
mechanisms
 E.g. Ottawa fiber build, Peel County, I-wire, SURAnet, G-Wire, CENIC DCP,
SURFnet, etc etc
 Next generation fiber companies who are building condominium fiber
networks for communities and school boards and who want to offer value
added fiber services but not traditional telcommunications service
 E.g. C2C, Universe2u, PF.net, Williams, QuebecTel, Videotron, etc
 Next generation collocation facilities to offer no-cost peering and wavelength
routing
 Metromedia, Equinix, LINX, PF.net, LayerOne, Westin, PAIX, Above.com,
Colo.com, etc etc
 Over 500 Ixs and carrier hotels worldwide
OBGP Peering
 Possible technique for allowing automatic peering at IXs between consenting
ISPs
 External routers are given control of specific ports on the OXC
 The router that controls switch can act as an optical route server notifying all
peers of any new consenting OBGP peers
 External routers signal to each other if they wish to setup direct optical
connection
 Choice of partner can be based on size of traffic flows
 Partners can be changed through a routing flap
 Only see each other’s customers routes – not the default core
OIX using OBGP
AS 200
170.10.10.0
Institution A
Switch Ports are part of
institution’s AS
Institution B
AS 100
160.10.10.0
Institution C
Institution D
Figure 10.0
AS 400
190.10.10.0
AS 300
180.10.10.0
Transport Architecture
 Heterogeneous transport architectures used on backbone links
 Type of transport architecture on each link determined by length of link
between O-ADMs, GbE-ADMs or OBGP switches, requirement for optical
repeaters or regenerators, etc
 Examples:
 8 or 16 channel GbE used on short haul links (up to 2000 km) between
OADMs or OBGPS; or
 OC-192 Ethernet over SONET with multiplexed 10 single GbE or
trunked 10GbE; or
 Proprietary 8 channel 2 x GbE multiplexed into OC48 optics with FEC
wrapper
 Repeaters:
 GbE or 10GbE 2R transceivers every 50-80 km combined with GbE or
10Gbe 3R switches every 200 – 400 km; or
 Traditional EDFAs at 1550nm every 50- 80 km with OC-192 regenerators
at every 200-400km; or
 All optical broadband: Counter rotating Raman amplifiers, multi band
EDFAs, EFFs, dispersion correction fiber, etc
Tributary Architecture
 Customer can connect through OADM,Gbe-ADM, direct to OBGP
switch or through CA*net 4 router
 Customer access link is either GbE or trunked 10GbE (I.e. 10
separate GbE channels
 In future customer will have a choice of protocols, but for now GbE
will be basic standard across the network
Switch Architecture
 Low speed MEMs or similar capacity switch
 Could also use non blocking GbE switch
 Switch can also be distributed across an optical network using GMPLS or
ODSI
 Each switch component can be controlled by a socket/port by any external
network element with appropriate security mechanisms
 If OXC used for traffic engineering or QoS then controlling router manipulates
both input and output ports
 If OXC used for distributed peering then participating AS only owns either
INPUT or OUTPUT ports
 Eventually switches can also support optical trunking of many optical paths
 Switch commands are kept very simple, leaving all complexity to OBGP
messages
 Switch does not know or care the direction of the wavelength – that is
established with OBGP protocol