Transcript UltraNet Engineering

DOE-UltraScience Net
(& network infrastructure) Update
JointTechs Meeting
February 15, 2005
W. R. Wing
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Motivation - DOE Needs Extreme
Networking
 160-200 Gbs throughput by 2008
 Probably only achievable by circuit bonding
 0.1% packet re-order and jitter control
 Probably only achievable at SONET layer
 Line rate, provably-secure, connections
 Only available at SONET or optical layer
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
The Well Known Bandwidth Problem
 DOE needs 160 - 200 Gbs by 2008
 Typical multi-stream throughput limited to
25+/- Gbs, and programming multi-steam
is hard - hero efforts yield ~30 Gbs
 Typical single-steam throughput to a
single application ~1Gbs, even on a tuned
network - hero effort yields 5-6+ Gbs
 Networks will soon offer 40 Gbs channels
 Firewalls throw it away anyway
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Even Without Firewalls We Have a Technology Problem
•
•
DOE needs 160-200 Gbs by 2008
This is on the “blue line”
•
This isn’t a problem routers are
going to fix - bonded IP channels
ONLY work with VERY large
numbers of streams (as Teragrid
has discovered)
Bonding at layer-2 (Ethernet)
is just as bad
Solution is circuit switching and
Layer-1 bonding
•
•
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Jitter Control
 Needed to deconstruct data sets, needed
for remote vis, and for remote instrument
control or steering
 Routers and switch-routers are path
deterministic, but not time-deterministic
 Only at layer-1 (or SONET) do you get
time-determinism
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Security
 Security requirements will get more and
more onerous
 Solutions involving IPSec, VPN’s, and
encryption have more and more trouble
running at line speed
 Alternate solution is to use optical/SONET
circuits with provable, known end-points
and inherent immunity to injected traffic
 Again - only available at layer-1
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
In Summary
 Only at layer-1 do you get:




Zero packet re-ordering
Zero jitter
Zero drops due to Congestion
Known (by definition) paths and end points
 SONET (or below) can do this
 Ethernet switches and switch/routers can’t
 MPLS can’t
 Layer-1 circuits can:
 Bypass firewalls
 Carry non-IP frames (e.g., Fiber-Channel over SONET)
 Easily (transparently) support parallel, bonded circuits
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
MPLS Jitter - ORNL / Atlanta
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
How do we get there from here?
 It requires a research network we control
at least down to the SONET layer
 It requires a research network with
significant span
 It requires a research network with at least
two lambdas
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
UltraScience Net Research Testbed
 Building an extended-regional lambda-switching
testbed
 Connect to NLR in Atlanta and Chicago
 Use asset-trading to extend reach (Sunnyvale and
East Coast)
 Provide an evolving matrix of switching
capabilities
 Separately fund research projects (e.g., highperformance protocols, control, visualization)
that will exercise the network and directly support
applications at the host institutions
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Atlanta
Acworth
Dalton
Chattanooga
Stevenson
Christiana
Nashville
Bowling Green
Upton
Louisville
Seymour
Indianapolis
Crawfordsville
Monon
Chicago
Schneider
In More Detail -
CD
F10
Nx
10GbE
OLA
2x10GbE ORNL TeraGRID
OLA - 2
T320
ORNL
CI
OLA -1
UT TeraGRID
Knoxville
ORNL TeraGRID
N x 10GbE, Ultranet
Memphis
TN.edu
GbE / 10GbE
GbE / 10GbE
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
20 x Gig-E???
OC192’s
T640
2x10GbE – UT TeraGRID
GbE /
10GbE
T640
2xOC192 – UT
TeraGRID
GbE /
10GbE
2x10GbE
CD bypass for
future 10GbE
CD switching
2xOC192 – ORNL TeraGRID
GbE /
10GbE
Nx
10GbE
2xOC192 – UT TeraGRID
CI
Connections to National Nets
ORNL
Internet2
NLR
ORNL
TeraGrid
ORNL-TVA
ESnet
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Research Projects
 A progression of switching approaches
 Study/compare SONET, MPLS(GMPLS), and all optical
 New all-optical technologies coming (e.g., laser tuning)
 Fast Local Storage
 Storage Depots
 Transport-optimized storage
 Progression of experimental point-to-point transport
technologies
 Fiber channel
 Infiniband
 LAN-PHY Ethernet initial transport
 Circuit-Switched backbone, Frame-Switched edge
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Digression: LAN-PHY vs. WAN-PHY
 Conventional Wisdom: LAN-PHY will win
 Cheaper, faster, etc. (based on POS experience)
 False cost model (extra processing)
 Ignore SONET advantages
 New Transponders are transparent
 Software selectable OC192 (WAN–PHY) or LAN–
PHY
 Costs now equal
 Advantage now WAN-PHY
 DCC and OAM non-trivial considerations
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
PoS vs. Ethernet
SONET
SONET
POS
IP
DATA
IP
PAD POS
POS
IP
DATA
IP
PAD POS
E
IP
DATA
IP
PAD E
IP
DATA
IP
E
IP
DATA
IP
PAD E
E
IP
DATA
IP
PAD E
IP
DATA
IP
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
SONET
SONET
Status and Schedule
 All contracts signed
 Fiber, Co-Lo space and power, equipment and
installation, smart-hands
 Equipment shipping, built-out started
 Chicago-Sunnyvale paced by NLR crossconnects in Chicago (Level(3) to Starlight)
 NLR supplying 10Gig-E (LAN-PHY) initially
 PNNL fiber schedule will pace their connection
 First Chicago-Sunnyvale traffic this month
 Details on following VuGraphs -
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Phase-1 (February)
Seattle
Chicago
1G Ethernet
10G LanPHY
Force10
E600
10G WanPHY
OC-192 SONET
OC-192 SONET (integrated optics)
Force10
E600
Sunnyvale
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Phase-2 (February-March)
Chicago
Seattle
1G Ethernet
10G LanPHY
Force10
E600
10G WanPHY
OC-192 SONET
OC-192 SONET (integrated optics)
Ciena
CD-CI
VL2020
N x 1Gig-E
Ciena
CD-CI
ORNL
M160
Turin
TR2000
Sycamre
SN16000
Ciena
CD-CI
Force10
E600
ORNL
Ciena
CD-CI
Sycamre
SN16000
SOX
M320
Atlanta
Sunnyvale
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Phase-3 (March - April)
Seattle
Chicago
1G Ethernet
Force10
E300
Force10
E600
10G LanPHY
10G WanPHY
OC-192 SONET
Ciena
CD-CI
OC-192 SONET (integrated optics)
VL2020
Ciena
CD-CI
Force10
E300
Turin
TR2000
Force10
E600
Sycamre
SN16000
Ciena
CD-CI
Atlanta
Sunnyvale
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Sycamre
SN16000
ORNL
Phase-4 (April - May)
Chicago
Seattle
1G Ethernet
Force 10
E300
Ciena
CD-CI
10G LanPHY
10G WanPHY
OC-192 SONET
OC-192 SONET (integrated optics)
Force 10
E300
Ciena
CD-CI
VL2020
PNNL
Ciena
CD-CI
Force 10
E300
Turin
TR2000
Force 10
E300
Sycamre
SN16000
Sycamre
SN16000
Ciena
CD-CI
Sunnyvale
Atlanta
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
ORNL
Thank You
http://www.csm.ornl.gov/ultranet
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY