Transcript ESnet-update

Recent and Planned Developments in
ESnet
Michael Ernst, BNL
Slides courtesy of Mike Bennett, ESnet
Dedicated Meeting with NII
May 14, 2013
Tokyo
Energy Sciences Network Overview
A national network, optimized for science:
•
connecting 40 labs, facilities with >100 networks
•
optimized for massive science data flows
•
offering capabilities not available commercially
•
$34.5M in FY12 (40 staff)
$62M ARRA grant funding:
•
world’s first 100G network at continental scale
•
optical fiber assets, access to spectrum
•
easier (and cheaper) growth
On the web:
•
www.es.net
•
fasterdata.es.net
•
my.es.net
3/31/2016
2
2
Not the Commercial Internet
 Engineered for massive traffic flows:




emphasis on lossless transport
bandwidth guarantees spanning multiple networks
high-speed (n x 100G) data transport
distributed performance monitoring platform
 Optimized for DOE science missions:
 extensive scientific outreach, requirements-gathering
 visualization tools for monitoring health of science data flows
 global leadership in network standards, middleware, architecture
 Integrated with scientific user support, advocacy, outreach:
 rapid diagnosis of data transport issues, around the world
 global advocacy for science-optimized network /security architectures
 partnership and outreach with facilities, VOs, peer organizations
3/31/2016
3
ESnet5 March 2013
100
10
PNNL
STAR
10
10
100 10
JGI
10010
LBNL
10
10
100
100
10
10
10
AMES
100
10
1
ANL
10
100
10
100
100
Salt Lake
10
SNLL
LLNL
100
100
10
JLAB
100 100
1
10
LOSA
10
10
BNL
PPPL
GFDL
PU Physics
10
100
100
SUNN
100
10
100
100
100
SNLA
100
100
100
10
10 100
LOSA
LBNL
LLNL
ESnet PoP/hub locations
ESnet managed 100G routers
ESnet managed 10G router
Site managed routers
ESnet optical node locations (only some are shown)
ESnet optical transport nodes (only some are shown)
commercial peering points
R&E network peering locations
Major Office of Science (SC) sites
Major non-SC DOE sites
100
Geography is
only representational
Routed IP 100 Gb/s
Routed IP 4 X 10 Gb/s
3rd party 10Gb/s
Express / metro 100 Gb/s
Express / metro 10G
Express multi path 10G
Lab supplied links
Other links
Tail circuits
4
The Trend Continues …
100 PB
10 PB
1 PB
100 TB
10 TB
1 TB
Bytes/month transferred by ESnet
100 GB
’90 ’91 ’92 ‘93 ’94 ’95 ’96 ’97 ’98 ’99 ’00 ’01 ’02 ’03 ’04 ’05 ’06 ’07 ’08 ’09 ’10 ’11 ’ 12 ’13
Year
5
Facilities now Coupled by Networks
•
•
•
Using free-electron laser (LCLS) at SLAC
to take ‘snapshots’ of catalytic reaction
in Photosystem II
X-ray diffraction
(structure)
Liquid-jet
Injection of
mm-size crystals
Data transported to HPC resource
(NERSC) for real-time computational
analysis
Optical
pump
One experiment triples NERSC’s
network utilization
Diffraction
Detector
Injector
X-ray emission spectroscopy
(Chemistry at the catalytic site)
Analyzer
Crystals
XES
Detector
X-ray Beam
Apertures
Source: Nicholas
Sauter, LBNL
• charge density/spin state
• ligand environment
Kern et al (2012) PNAS 109: 9721
Sierra et al (2012) Acta Cryst D68:
6 1584
Mori et al (2012) PNAS 109: 19103
SLAC / LCLS NERSC (http://my.es.net)
All NERSC
Traffic
Photosystem II
X-Ray Study
3/31/2016
7
Near Term Developments
Work Completed Recently
Link acceptance testing at:
o LBNL, ANL
100G production connections to:
o ORNL and ANL
100G production connections to exchange points & R&E peers:
o MANLAN, CENIC, I2 (Chicago), SINET (WIX)
Work in progress
100G production connections to:
o BNL, FNAL, LBNL, LLNL, and NERSC
o 100G production connections to exchange points & R&E peers:
o Starlight, PACWAVE
Diverse fiber laterals & diverse optical nodes at ANL & FNAL
Lots of cleanup & consolidation at the hubs, moving connections from the Juniper MX’s
to the ALUs
Normalize 100G Testbed infrastructure
8
A First Pass at Planning the Future
 Rough Approximation
o Look at bisection bandwidth
– Cut the network in half (vertically) and examine capacity
o Look at traffic coming into the network
– Most of the traffic coming into ESnet terminates there
» No transit services
o Try to forecast when we’ll need to make changes
– Use our traffic growth trend
– Try to forecast when component costs decrease
– Put some milestones on the plot
 This gives us lots to think about
o Large margin of error, but it’s a start
9
Backbone & Traffic Metrics
Backbone Bisection Bandwidth:
Sum of Links crossing the red line
 Average Accepted Traffic:
 Sum of inbound traffic crossing red
line from Sites & Peers, converted to Gb/s
Site
Peers
Peers
ESnet
Site
Peers
Site
Peers
Site
10
11
Internet2
contract
expires
Add routers,
optical chassis
incrementally
starting 2015
Optical system
full in 2020; 88
x 100G
routed net
exceeds
ESnet4
complexity
new waves starting
Early 2014
10x100G on all
routes by 2017;
start deploying
ESnet6
12
Things ESnet is considering
 How much ‘head-room’ is enough
o Need less ‘bisection bandwidth’ if traffic is local vs cross-country
o Need more for unpredictable users
o Need less for predictable steady-state users
– Could automated science work-flow systems be more
predictable, and sustain utilization >75%?
 Some changes that could affect growth rates
o Science DMZ adoption reducing impacts of poorly performing
firewalls & security infrastructure
o PerfSONAR adoption & deployment allowing people to discover & fix
broken infrastructure
o Data mobility tools enhancement & adoption
o Changes in computing models
o Adoption rate of 100GE will drive the price curve of 100G
components down
13
Summary
 Lots of changes in the last nine months and still lots of work to do
o Finish connecting sites at 100G
o Clean up and optimize hubs
o Add 100G peers
 The dust hasn’t settled on Esnet5 and they’re already thinking about
turning up new waves
– Start this all over again in a couple of years!
– Start on ESnet6 deployment as early as 2017
14
backup
15
perfSONAR: New Dashboard for users
Visualization of perfSONAR
throughput and packet loss data
Easy to build a table with a specific
community of interest
Results displayed as a grid of
green/yellow/red making it easy to
identify problems
Dashboards for a few communities
developed and available at
http://ps-dashboard.es.net/
16
World-Wide perfSONAR-PS Deployments:
536 bwctl nodes, 505 owamp nodes as of Jan 7
3/31/2016
17
17