Transcript ESnet

ESnet Network Requirements
ASCAC Networking Sub-committee Meeting
April 13, 2007
Eli Dart
ESnet Engineering Group
Lawrence Berkeley National Laboratory
1
Overview
Requirements are primary drivers for ESnet – science focused
Sources of Requirements
Office of Science (SC) Program Managers
Direct gathering through interaction with science users of the
network
 Example case studies (updated 2005/2006)
 Magnetic Fusion
 Large Hadron Collider (LHC)
 Climate Modeling
 Spallation Neutron Source
Observation of the network
Other requirements
Requirements aggregation
Convergence on a complete set of network requirements
2
 Requirements from SC Program Managers
• SC Program Offices have determined that ESnet future priorities must
address the requirements for:
– Large Hadron Collider (LHC), CERN
– Relativistic Heavy Ion Collider (RHIC), BNL, US
– Large-scale fusion (ITER), France
– High-speed connectivity to Asia-Pacific
• Climate and Fusion
– Other priorities and guidance from SC will come from upcoming perProgram Office requirements workshops, beginning this summer
• Modern science infrastructure is too large to be housed at any one
institution
– Structure of DOE science assumes the existence of a robust, highbandwidth, feature-rich network fabric that interconnects scientists,
instruments and facilities such that collaboration may flourish
3
 Direct Gathering Through Interaction with Stakeholders
•
SC selected a representative set of applications for the 2002 Workshop
•
Case studies were created for each application at the Workshop in order to
consistently characterize the requirements
•
The requirements collected from the case studies form the foundation for the current
ESnet4 architecture
– Bandwidth, Connectivity Scope / Footprint, Services
– We do not ask that our users become network experts in order to communicate their
requirements to us
– We ask what tools the researchers need to conduct their science, synthesize the
necessary networking capabilities, and pass that back to our constituents for evaluation
•
Per-Program Office workshops continue this process
– Workshops established as a result of ESnet baseline Lehman Review
– Workshop survey process extended to ESnet sites via Site Coordinators
•
ESnet has a much larger user base (~50k to 100k users) than a typical supercomputer
center (~3k users) and so has a more diffuse relationship with individual users
– Requirements gathering focused on key Principal Investigators, Program Managers,
Scientists, etc, rather than a broad survey of every computer user within DOE
– Laboratory CIOs and their designates also play a key role in requirements input
4
Case Studies For Requirements
• Advanced Scientific Computing
Research (ASCR)
– NERSC
– NLCF
• Basic Energy Sciences
– Advanced Light Source
• Macromolecular Crystallography
• Fusion Energy Sciences
– Magnetic Fusion Energy/ITER
• High Energy Physics
– LHC
• Nuclear Physics
– RHIC
– Chemistry/Combustion
– Spallation Neutron Source
• Biological and Environmental
– Bioinformatics/Genomics
– Climate Science
•
There is a high level of correlation between network requirements for large and small
scale science – the only difference is bandwidth
– Meeting the requirements of the large-scale stakeholders will cover the smaller ones,
provided the required services set is the same
5
Case Studies Requirements Gathering
• For all the science cases the following were identified
by examining the science environment
– Instruments and facilities
• Location and use of facilities, instruments, computational
resources, etc.
• Data movement and storage requirements
– Process of science
• Collaborations
• Network services requirements
• Noteworthy patterns of use (e.g. duty cycle of instruments)
– Near-term needs (now to 12 months)
– 5 year needs (relatively concrete)
– 5-10 year needs (more uncertainty)
6
Example Case Study Summary Matrix: Fusion
• Considers instrument and facility requirements, the process of science drivers
and resulting network requirements cross cut with timelines
Feature
Anticipated Requirements
Time
Frame
Science Instruments and
Facilities
Near-term
 Each experiment only gets a few
days per year - high productivity is
critical
 Experiment episodes (“shots”)
generate 2-3 Gbytes every
20 minutes, which has to be
delivered to the remote analysis sites
in two minutes in order to analyze
before next shot
 Highly collaborative experiment and
analysis environment
 Real-time data access and analysis
for experiment steering (the more
that you can analyze between shots
the more effective you can make the
next shot)
 Shared visualization capabilities
5 years
 10 Gbytes generated by experiment
every 20 minutes (time between
shots) to be delivered in two minutes
 Gbyte subsets of much larger
simulation datasets to be delivered
in two minutes for comparison with
experiment
 Simulation data scattered across
United States
 Transparent security
 Global directory and naming
services needed to anchor all of the
distributed metadata
 Support for “smooth” collaboration
in a high-stress environment
 Real-time data analysis for
experiment steering combined with
simulation interaction = big
productivity increase
 Real-time visualization and
interaction among collaborators
across United States
 Integrated simulation of the several
distinct regions of the reactor will
produce a much more realistic
model of the fusion process
 Network bandwidth and data
analysis computing capacity
guarantees (quality of service)
for inter-shot data analysis
 Gbits/sec for 20 seconds out of
20 minutes, guaranteed
 5 to 10 remote sites involved for
data analysis and visualization
 Parallel network I/O between simulations, data
archives, experiments, and visualization
 High quality, 7x24 PKI identity authentication
infrastructure
 End-to-end quality of service and quality of
service management
 Secure/authenticated transport to ease access
through firewalls
 Reliable data transfer
 Transient and transparent data replication for
real-time reliability
 Support for human collaboration tools
5+ years
 Simulations generate 100s of Tbytes
 ITER – Tbyte per shot, PB per year
 Real-time remote operation of the
experiment
 Comprehensive integrated
simulation
 Quality of service for network
latency and reliability, and for
co-scheduling computing
resources
 Management functions for network quality of
service that provides the request and access
mechanisms for the experiment run time,
periodic traffic noted above.
Process of Science
Network
Network Services and
Middleware
 PKI certificate authorities that enable strong
authentication of the community members and
the use of Grid security tools and services.
 Directory services that can be used to provide
the naming root and high-level (communitywide) indexing of shared, persistent data that
transforms into community information and
knowledge
 Efficient means to sift through large data
repositories to extract meaningful information
from unstructured data.
7
Requirements from Instruments and Facilities
• This is the ‘hardware infrastructure’ of DOE science – types of
requirements can be summarized as follows
– Bandwidth: Quantity of data produced, requirements for timely movement
– Connectivity: Geographic reach – location of instruments, facilities, and users
plus network infrastructure involved (e.g. ESnet, Internet2, GEANT)
– Services: Guaranteed bandwidth, traffic isolation, etc.; IP multicast
• Data rates and volumes from facilities and instruments – bandwidth,
connectivity, services
– Large supercomputer centers (NERSC, NLCF)
– Large-scale science instruments (e.g. LHC, RHIC)
– Other computational and data resources (clusters, data archives, etc.)
• Some instruments have special characteristics that must be
addressed (e.g. Fusion) – bandwidth, services
• Next generation of experiments and facilities, and upgrades to
existing facilities – bandwidth, connectivity, services
–
–
–
–
Addition of facilities increases bandwidth requirements
Existing facilities generate more data as they are upgraded
Reach of collaboration expands over time
New capabilities require advanced services
8
Requirements from Examining the Process of Science (1)
• The geographic extent and size of the user base of
scientific collaboration is continuously expanding
– DOE US and international collaborators rely on ESnet to reach
DOE facilities
– DOE Scientists rely on ESnet to reach non-DOE facilities
nationally and internationally (e.g. LHC, ITER)
– In the general case, the structure of modern scientific
collaboration assumes the existence of a robust, highperformance network infrastructure interconnecting collaborators
with each other and with the instruments and facilities they use
– Therefore, close collaboration with other networks is essential for
end-to-end service deployment, diagnostic transparency, etc.
• Robustness and stability (network reliability) are critical
– Large-scale investment in science facilities and experiments
makes network failure unacceptable when the experiments
depend on the network
– Dependence on the network is the general case
9
Requirements from Examining the Process of Science (2)
• Science requires several advanced network services for
different purposes
– Predictable latency, quality of service guarantees
• Remote real-time instrument control
• Computational steering
• Interactive visualization
– Bandwidth guarantees and traffic isolation
• Large data transfers (potentially using TCP-unfriendly protocols)
• Network support for deadline scheduling of data transfers
• Science requires other services as well – for example
– Federated Trust / Grid PKI for collaboration and middleware
• Grid Authentication credentials for DOE science (researchers, users,
scientists, etc.)
• Federation of international Grid PKIs
– Collaborations services such as audio and video conferencing
10
Science Network Requirements Aggregation Summary
Science
Drivers
End2End
Reliability
Connectivity
2006
End2End
Band
width
2010
End2End
Band
width
1 TB/day
5 TB/day
300 Mbps
1.5 Gbps
• DOE sites
• US Universities
625 Mbps
250 Gbps
Science Areas
/ Facilities
Advanced
Light Source
-
Bioinformatics
-
• DOE sites
• US Universities
• Industry
12.5
Gbps in
two years
Traffic
Characteristics
Network Services
• Bulk data
• Remote control
• Guaranteed bandwidth
• PKI / Grid
• Bulk data
• Remote control
• Point-to-multipoint
• Guaranteed bandwidth
• High-speed multicast
Chemistry /
Combustion
-
• DOE sites
• US Universities
• Industry
-
10s of
Gigabits
per
second
• Bulk data
• Guaranteed bandwidth
• PKI / Grid
Climate
Science
-
• DOE sites
• US Universities
• International
-
5 PB per
year
• Bulk data
• Remote control
• Guaranteed bandwidth
• PKI / Grid
• Bulk data
• Remote control
• Guaranteed bandwidth
• Traffic isolation
• PKI / Grid
High Energy
Physics (LHC)
99.95+%
(Less than
4 hrs/year)
• US Tier1 (DOE)
• US Tier2 (Universities)
• International (Europe,
Canada)
5 Gbps
10 Gbps
60 to 80
Gbps
(30-40
Gbps per
US Tier1)
11
Science Network Requirements Aggregation Summary
Science
Drivers
End2End
Reliability
Connectivity
2006
End2End
Band
width
2010
End2End
Band
width
• DOE sites
• US Universities
• Industry
200+
Mbps
1 Gbps
• Bulk data
• Remote control
• Guaranteed bandwidth
• Guaranteed QoS
• Deadline scheduling
• DOE sites
• US Universities
• Industry
• International
• DOE sites
• US Universities
• Industry
• International
• DOE sites
• US Universities
• International
10 Gbps
20 to 40
Gbps
• Bulk data
• Remote control
• Guaranteed bandwidth
• Guaranteed QoS
• Deadline Scheduling
• PKI / Grid
Backbone
Band
width
parity
Backbone
band
width
parity
• Bulk data
12 Gbps
70 Gbps
• Bulk data
• DOE sites
640 Mbps
2 Gbps
• Bulk data
Science Areas
/ Facilities
Magnetic
Fusion Energy
99.999%
(Impossible
without full
redundancy)
NERSC
-
NLCF
-
Nuclear
Physics
(RHIC)
-
Spallation
Neutron
Source
High
Traffic
Characteristics
Network Services
• Guaranteed bandwidth
• PKI / Grid
(24x7
operation)
12
 Example Case Studies
• By way of example, four of the cases are discussed here
Magnetic fusion
Large Hadron Collider
Climate Modeling
Spallation Neutron Source
• Categorization of case study information: quantitative vs.
qualitative
– Quantitative requirements from instruments, facilities, etc.
•
•
•
•
Bandwidth requirements
Storage requirements
Computational facilities
Other ‘hardware infrastructure’
– Qualitative requirements from the science process
• Bandwidth and service guarantees
• Usage patterns
13
Magnetic Fusion Energy
14
Magnetic Fusion Requirements – Instruments and Facilities
• Three large experimental facilities in US (General Atomics, MIT,
Princeton Plasma Physics Laboratory)
– 3 GB data set per pulse today, 10+ GB per pulse in 5 years
– 1 pulse every 20 minutes, 25-35 pulses per day
– Guaranteed bandwidth requirement: 200+ Mbps today, ~1 Gbps in 5
years (driven by science process)
• Computationally intensive theory/simulation component
– Simulation runs at supercomputer centers, post-simulation analysis at
~20 other sites
– Large data sets (1 TB+ in 3-5 years)
– 10’s of TB of data in distributed archives
• ITER
–
–
–
–
Located in France
Groundbreaking soon, production operations in 2015
1 TB of data per pulse, 1 pulse per hour
Petabytes of simulation data per year
15
Magnetic Fusion Requirements – Process of Science (1)
• Experiments today
– Interaction between large groups of local and remote users and
the instrument during experiments – highly collaborative
– Data from current pulse is analyzed to provide input parameters
for next pulse
– Requires guaranteed network and computational throughput on
short time scales
• Data transfer in 2 minutes
• Computational analysis in ~7 minutes
• Science analysis in ~10 minutes
• Experimental pulses are 20 minutes apart
• ~1 minute of slack – this amounts to 99.999% uptime requirement
– Network reliability is critical, since each experiment gets only a
few days of instrument time per year
16
Magnetic Fusion Requirements – Process of Science (2)
• Simulation
– Large, geographically dispersed data sets, more so in the future
– New long-term initiative (Fusion Simulation Project, FSP) –
integrated simulation suite
– FSP will increase the computational requirements significantly in
the future, resulting in increased bandwidth needs between
fusion users and the SC supercomputer centers
• Both experiments and simulations rely on middleware
that uses ESnet’s federated trust services to support
authentication
• ITER
– Scale will increase substantially
– Close collaboration with the Europeans is essential for DOE
science
17
Magnetic Fusion – Network Requirements
• Experiments
– Guaranteed bandwidth requirement: 200+ Mbps today, ~1 Gbps in 5
years (driven by science process)
– Reliability (99.999% uptime)
– Deadline scheduling
– Service guarantees for remote steering and visualization
• Simulation
– Bulk data movement (310 Mbps end2end to move 1 TB in 8
hours)
• Federated Trust / Grid PKI for authentication
• ITER
– Large guaranteed bandwidth requirement (pulsed operation and
science process as today, much larger data sets)
– Large bulk data movement for simulation data (Petabytes per year)
18
Large Hadron Collider at CERN
19
LHC Requirements – Instruments and Facilities
•
Large Hadron Collider at CERN
– Networking requirements of two experiments have been characterized – CMS
and Atlas
– Petabytes of data per year to be distributed
•
LHC networking and data volume requirements are unique to date
– First in a series of DOE science projects with requirements of unprecedented
scale
– Driving ESnet’s near-term bandwidth and architecture requirements
– These requirements are shared by other very-large-scale projects that are
coming on line soon (e.g. ITER)
•
Tiered data distribution model
– Tier0 center at CERN processes raw data into event data
– Tier1 centers receive event data from CERN
• FNAL is CMS Tier1 center for US
• BNL is Atlas Tier1 center for US
• CERN to US Tier1 data rates: 10 Gbps by 2007, 30-40 Gbps by 2010/11
– Tier2 and Tier3 sites receive data from Tier1 centers
• Tier2 and Tier3 sites are end user analysis facilities
• Analysis results are sent back to Tier1 and Tier0 centers
• Tier2 and Tier3 sites are largely universities in US and Europe
20
LHCNet Security Requirements
• Security for the LHC Tier0-Tier1 network is being defined
by CERN in the context of the LHC Network Operations
forum
• Security to be achieved by filtering packets at CERN and
the Tier1 sites to enforce routing policy (only approved
hosts may send traffic)
• In providing circuits for LHC, providers must make sure
that these policies cannot be circumvented
21
LHC Requirements – Process of Science
•
Strictly tiered data distribution model is only part of the picture
– Some Tier2 scientists will require data not available from their local Tier1 center
– This will generate additional traffic outside the strict tiered data distribution tree
– CMS Tier2 sites will fetch data from all Tier1 centers in the general case
• CMS traffic patterns will depend on data locality, which is currently unclear
•
Network reliability is critical for the LHC
– Data rates are so large that buffering capacity is limited
– If an outage is more than a few hours in duration, the analysis could fall
permanently behind
• Analysis capability is already maximized – little extra headroom
•
•
CMS/Atlas require DOE federated trust for credentials and federation with LCG
•
Several unknowns will require ESnet to be nimble and flexible
– Tier1 to Tier1,Tier2 to Tier1, and Tier2 to Tier0 data rates could add significant
additional requirements for international bandwidth
– Bandwidth will need to be added once requirements are clarified
– Drives architectural requirements for scalability, modularity
Service guarantees will play a key role
– Traffic isolation for unfriendly data transport protocols
– Bandwidth guarantees for deadline scheduling
22
LHC Ongoing Requirements Gathering Process
• ESnet has been an active participant in the LHC network
planning and operation
– Been an active participant in the LHC network operations
working group since its creation
– Jointly organized the US CMS Tier2 networking requirements
workshop with Internet2
– Participated in the US Atlas Tier2 networking requirements
workshop
– Participated in all 5 US Tier3 networking workshops
23
LHC Requirements Identified To Date
•
•
•
•
•
•
10 Gbps “light paths” from FNAL and BNL to CERN
– CERN / USLHCnet will provide10 Gbps circuits to Starlight, to 32 AoA, NYC
(MAN LAN), and between Starlight and NYC
– 10 Gbps each in near term, additional lambdas over time (3-4 lambdas each by
2010)
BNL must communicate with TRIUMF in Vancouver
– This is an example of Tier1 to Tier1 traffic – 1 Gbps in near term
– Circuit is currently being built
Additional bandwidth requirements between US Tier1s and European Tier2s
– To be served by USLHCnet circuit between New York and Amsterdam
Reliability
– 99.95%+ uptime (small number of hours per year)
– Secondary backup paths – SDN for the US and possibly GLIF (Global Lambda
Integrated Facility) for transatlantic links
– Tertiary backup paths – virtual circuits through ESnet, Internet2, and GEANT
production networks
Tier2 site connectivity
– Characteristics TBD, and is the focus of the Tier2 workshops
– At least 1 Gbps required (this is already known to be a significant underestimate
for large US Tier2 sites)
– Many large Tier2 sites require direct connections to the Tier1 sites – this drives
bandwidth and Virtual Circuit deployment (e.g. UCSD)
Ability to add bandwidth as additional requirements are clarified
24
Identified US Tier2 Sites
• Atlas (BNL Clients)
• CMS (FNAL Clients)
– Boston University
– Caltech
– Harvard University
– MIT
– Indiana University Bloomington
– Purdue University
– Langston University
– University of California San
Diego
– University of Chicago
– University of New Mexico Alb.
– University of Oklahoma
Norman
– University of Texas at Arlington
– University of Florida at
Gainesville
– University of Nebraska at
Lincoln
– University of Wisconsin at
Madison
• Calibration site
– University of Michigan
25
LHC Tier 0, 1, and 2 Connectivity Requirements Summary
CERN-1 CERN-2
CERN-3
TRIUMF
(Atlas T1,
Canada)
Vancouver
CANARIE
USLHCNet
Seattle
ESnet
SDN
Internet2 / Gigapop
Footprint
Toronto
BNL
(Atlas T1)
Virtual Circuits
Boise
Chicago
Denver
KC
ESnet
IP Core
FNAL
(CMS T1)
Wash DC
Albuq.
San Diego
Dallas
GÉANT
Atlanta
GÉANT-2
LA
GÉANT-1
Sunnyvale
New York
Jacksonville
USLHC nodes
• Direct connectivity T0-T1-T2
Internet2/GigaPoP nodes
ESnet IP core hubs
• USLHCNet to ESnet to Internet2
ESnet SDN/NLR hubs
Tier 1 Centers
Cross connects with Internet2
Tier 2 Sites
• Backup connectivity
• SDN, GLIF, VCs
26
LHC ATLAS Bandwidth Matrix as of April 2007
Site A
Site Z
ESnet A
ESnet Z
A-Z 2007
A-Z 2010
Bandwidth Bandwidth
CERN
BNL
AofA (NYC)
BNL
10Gbps
20-40Gbps
BNL
U. of Michigan
(Calibration)
BNL (LIMAN)
Starlight
(CHIMAN)
3Gbps
10Gbps
BNL
Boston University
Internet2 / NLR
Peerings
3Gbps
10Gbps
BNL
Harvard University
(Northeastern
Tier2 Center)
(Northeastern
Tier2 Center)
BNL
Indiana U. at
Bloomington
Internet2 / NLR
Peerings
3Gbps
10Gbps
(Midwestern
Tier2 Center)
(Midwestern
Tier2 Center)
3Gbps
10Gbps
BNL
U. of Chicago
BNL
Langston
University
BNL
U. Oklahoma
Norman
BNL
U. of Texas
Arlington
BNL
BNL
BNL (LIMAN)
BNL (LIMAN)
BNL (LIMAN)
Internet2 / NLR
Peerings
(Southwestern
Tier2 Center)
(Southwestern
Tier2 Center)
Tier3 Aggregate
BNL (LIMAN)
Internet2 / NLR
Peerings
5Gbps
20Gbps
TRIUMF (Canadian
ATLAS Tier1)
BNL (LIMAN)
Seattle
1Gbps
5Gbps
27
LHC CMS Bandwidth Matrix as of April 2007
Site A
Site Z
ESnet A
ESnet Z
A-Z 2007
Bandwidth
A-Z 2010
Bandwidth
CERN
FNAL
Starlight
(CHIMAN)
FNAL
(CHIMAN)
10Gbps
20-40Gbps
FNAL
U. of Michigan
(Calibration)
FNAL
(CHIMAN)
Starlight
(CHIMAN)
3Gbps
10Gbps
FNAL
Caltech
FNAL
(CHIMAN)
Starlight
(CHIMAN)
3Gbps
10Gbps
FNAL
MIT
FNAL
(CHIMAN)
AofA (NYC)/
Boston
3Gbps
10Gbps
FNAL
Purdue University
FNAL
(CHIMAN)
Starlight
(CHIMAN)
3Gbps
10Gbps
FNAL
U. of California at
San Diego
FNAL
(CHIMAN)
San Diego
3Gbps
10Gbps
FNAL
U. of Florida at
Gainesville
FNAL
(CHIMAN)
SOX
3Gbps
10Gbps
FNAL
U. of Nebraska at
Lincoln
FNAL
(CHIMAN)
Starlight
(CHIMAN)
3Gbps
10Gbps
FNAL
U. of Wisconsin at
Madison
FNAL
(CHIMAN)
Starlight
(CHIMAN)
3Gbps
10Gbps
FNAL
Tier3 Aggregate
FNAL
(CHIMAN)
Internet2 / NLR
Peerings
5Gbps
20Gbps
28
Estimated Aggregate Link Loadings, 2007-08
unlabeled links are 10 Gb/s
9
12.5
Seattle
13
Portland
Boise
13
9
Existing site
supplied
circuits
2.5
Boston
Chicago
Clev.
Sunnyvale
NYC
Denver
Philadelphia
KC
Salt
Lake
City
Pitts.
Wash DC
LA
Albuq.
San Diego
8.5
Raleigh
Tulsa
Nashville
OC48
(1(3))
(1)
Atlanta
6
6
Jacksonville
El Paso
ESnet IP switch/router hubs
ESnet IP switch only hubs
Houston
Baton
Rouge
ESnet SDN switch hubs
Layer 1 optical nodes at eventual ESnet Points of Presence
Layer 1 optical nodes not currently in ESnet plans
Committed bandwidth, Gb/s
2.5
Lab site
2.5
ESnet IP core (1)
ESnet Science Data Network core
ESnet SDN core, NLR links
Lab supplied link
LHC related link
MAN link
International IP Connections
29
ESnet4 2007-8 Estimated Bandwidth Commitments
Long Island MAN
600
W. Chicago
West
Chicago
MAN
unlabeled links are 10 Gb/s
CERN
5
Seattle
USLHCNet
BNL
(28)
Portland
CERN
32 AoA, NYC
Starlight
Boise
(29)
13
Sunnyvale
(32)
(23)
Bay Area MAN
LA
Chicago
10
(24)
SLAC
(19)
Philadelphia
KC
(15)
Wash DC
(30)
Raleigh
Tulsa
ANL
Nashville
OC48
(1(3))
(3)
(4)
Newport News - Elite
(2)
(20)
Jacksonville
(17)
(6)
LLNL
ESnet IP switch/router hubs
ESnet SDN switch hubs
(26)
Pitts.
Atlanta
NERSC
ESnet IP switch only hubsSNLL
(25)
(22)
Albuq.
El Paso
(10)
NYC
LBNL(1)
San Diego
Clev.
(21)
(0)
JGIFNAL
(11)
(13)
Denver
Salt
Lake
City
San Francisco
Boston
(9)
29
(total)
(7)
USLHCNet
10
Houston
(5)
All circuits are 10Gb/s.
Layer 1 optical nodes at eventual ESnet Points of Presence
Layer 1 optical nodes not currently in ESnet plans
Committed bandwidth, Gb/s
2.5
Lab site
Baton
MAX
Rouge
Wash.,
DC
MATP
JLab
ESnet IP core
ELITE
ESnet Science Data
Network core
ESnet SDN core, NLR links (existing)
Lab suppliedODU
link
LHC related link
MAN link
International IP Connections
30
Estimated Aggregate Link Loadings, 2010-11
unlabeled links are 10 Gb/s
labeled links are in Gb/s
30
Seattle
50
50
Boise
Boston
Sunnyvale
50
San Diego
50
Chicago
40
Philadelphia
50
KC
40
5
40
Wash. DC
5
30
Albuq.
Tulsa
5
40
Jacksonville
40
ESnet IP switch/router hubs
20
20
Atlanta
5
El Paso
OC48
30
30
10
Raleigh
50
Nashville
40
20
ESnet IP switch only hubs
50
50
Denver
4
40
Clev.
NYC
Salt
Lake
City
40
20
15
(>1 )
Portland
LA
45
Houston
ESnet IP core (1)
ESnet Science Data Network core
ESnet SDN core, NLR links (existing)
Lab supplied link
LHC related link
MAN link
International IP Connections
Baton
Rouge
ESnet SDN switch hubs
Layer 1 optical nodes at eventual ESnet Points of Presence
Layer 1 optical nodes not currently in ESnet plans
Committed bandwidth, Gb/s
2.5
Lab site
40
link capacity, Gb/s
31
ESnet4 2010-11 Estimated Bandwidth Commitments
unlabeled links are 10 Gb/s
600 W. Chicago
CERN
25
40
Seattle
BNL
(28)
Portland
15
(>1 )
32 AoA, NYC
CERN
5
(29)
Boise
Starlight
Sunnyvale
4
LA
(24)
4
(13)
Denver
Salt
Lake
City
FNAL
Albuq.
4
(20)
El Paso
10
5
(17)
Philadelphia
5 (26)
4
Wash. DC
(30)
5
Raleigh
5
Nashville
OC48
(4)
(3) 3
3
(19)
(25)
3
Tulsa
40 ANL
(1)
ESnet IP switch only hubs
5 (10)
(21)
(22)
(0)
ESnet IP switch/router hubs
Clev.
100
80
80
5
4
4
KC
(15)
5
(11)
Boston
(9)
NYC
5
USLHCNet
5
Chicago
(32)
(23)
20
65
(7)
4
San Diego
20
USLHCNet
25
10
Atlanta
(2)
5
4
Jacksonville
4
(6)
(5)
Houston
Baton
Rouge
ESnet SDN switch hubs
Layer 1 optical nodes at eventual ESnet Points of Presence
Layer 1 optical nodes not currently in ESnet plans
Committed bandwidth, Gb/s
2.5
Lab site
(20)
ESnet IP core (1)
ESnet Science Data Network core
ESnet SDN core, NLR links (existing)
Lab supplied link
LHC related link
MAN link
International IP Connections
Internet2 circuit number
32
Climate Modeling
33
Climate Modeling Requirements – Instruments and Facilities
• Climate Science is a large consumer of supercomputer
time
• Data produced in direct proportion to CPU allocation
– As supercomputers increase in capability and models become
more advanced, model resolution improves
– As model resolution improves, data sets increase in size
– CPU allocation may increase due to increased interest from
policymakers
– Significant data set growth is likely in the next 5 years, with
corresponding increase in network bandwidth requirement for
data movement (current data volume is ~200TB, 1.5PB/year
expected rate by 2010)
• Primary data repositories co-located with compute
resources
– Secondary analysis is often geographically distant from data
repositories, requiring data movement
34
Climate Modeling Requirements – Process of Science
• Climate models are run many times
– Analysis  improved model  analysis is typical cycle
– Repeated runs of models are required to generate sufficient data
for analysis and model improvement
• Current analysis is done by transferring model output
data sets to scientist’s home institution for local study
– Recent trend is to make data from many models widely available
– Less efficient use of network bandwidth, but huge scientific win
• PCMDI (Program for Climate Model Diagnosis and Intercomparison)
generated 200 papers in a year
• Wide sharing of data expected to continue
• PCMDI paradigm of wide sharing from central locations will require
significant bandwidth and excellent connectivity at those locations
– If trend of sharing data continues, more data repositories will be
opened, requiring more bandwidth resources
35
Climate Modeling Requirements
• Data movement
– Large data sets must be moved to remote analysis resources
– Central repositories collect and distribute large data volumes
• Hundreds of Terabytes today
• Petabytes by 2010
• Analysis cycle
– Steady growth in network usage as models improve
• Increased use of supercomputer resources
– As computational systems increase in capability, data set sizes
increase
– Increased demand from policymakers may result in increased
data production
36
Spallation Neutron Source (SNS) at ORNL
37
SNS Requirements – Instruments and Facilities
• SNS is latest instrument for Neutron Science
– Most intense pulsed neutron beams available for research
– Wide applicability to materials science, medicine, etc
– Users from DOE, Industry, Academia
– In process of coming into full production (full-power Accelerator
Readiness Review imminent as of April 2007)
• SNS detectors produce 160GB/day of data in production
– Operation schedule results in about 50TB/year
– Network requirements are 640Mbps peak
– This will increase to 10Gbps peak within 5 years
• Neutron science data repository is being considered
38
SNS Requirements – Process of Science
• Productivity is critical
– Scientists are expected to get just a few days per year of
instrument time
• Drives requirement for reliability
– Real-time analysis used to tune experiment in progress
• Linkage with remote computational resources
• 2Gbps network load for real-time remote visualization
• Most analysis of instrument data is expected to be done
using remote computational resources
– Data movement is necessary
– Workflow management software (possibly based on Grid tools)
will be necessary
• There is interest from the SNS community in ESnet’s
Federated Trust services for Grid applications
39
SNS Requirements
• Bandwidth
– 2Gbps today
– 10Gbps in 5 years
• Reliability
– Instrument time is a scarce resource
– Real-time instrument interaction
• Data movement
– Workflow tools
– Potential neutron science data repository
• Federated Trust
– User management
– Workflow tools
40
 Aggregation of Requirements from All Case Studies
•
Analysis of diverse programs and facilities yields dramatic convergence on a
well-defined set of requirements
– Reliability
•
•
•
•
Fusion – 1 minute of slack during an experiment (99.999%)
LHC – Small number of hours (99.95+%)
SNS – limited instrument time makes outages unacceptable
Drives requirement for redundancy, both in site connectivity and within ESnet
– Connectivity
• Geographic reach equivalent to that of scientific collaboration
• Multiple peerings to add reliability and bandwidth to interdomain connectivity
• Critical both within the US and internationally
– Bandwidth
•
•
•
•
•
10 Gbps site to site connectivity today
100 Gbps backbone by 2010
Multiple 10 Gbps R&E peerings
Ability to easily deploy additional 10 Gbps lambdas and peerings
Per-lambda bandwidth of 40 Gbps or 100 Gbps should be available by 2010
– Bandwidth and service guarantees
• All R&E networks must interoperate as one seamless fabric to enable end2end
service deployment
• Flexible rate bandwidth guarantees
– Collaboration support (federated trust, PKI, AV conferencing, etc.)
41
Additional Bandwidth Requirements Matrix – April 2007
Site A
Site Z
ESnet A
ESnet Z
A-Z 2007
Bandwidth
A-Z 2010
Bandwidth
ANL
(ALCF)
ORNL
ANL (CHIMAN)
ORNL (Atlanta
and Chicago)
10Gbps (2008)
20Gbps
ANL
(ALCF)
NERSC
ANL (CHIMAN)
NERSC (BAMAN)
10Gbps (2008)
20Gbps
BNL
(RHIC)
CC-J, RIKEN,
Japan
BNL (LIMAN)
NYC (MANLAN)
1Gbps
3Gbps
•
•
Argonne Leadership Computing Facility requirement is for large-scale
distributed filesystem linking ANL, NERSC and ORNL supercomputer centers
BNL to RIKEN traffic is a subset of total RHIC requirements, and is subject to
revision as the impact of RHIC detector upgrades becomes clearer
42
 Requirements Derivation from Network Observation
• ESnet observes several aspects of network traffic on an
ongoing basis
– Load
• Network traffic load continues to grow exponentially
– Flow endpoints
• Network flow analysis shows a clear trend toward the dominance of
large-scale science traffic and wide collaboration
– Traffic patterns
• Traffic pattern analysis indicates a trend toward circuit-like behaviors
in science flows
43
Total Esnet Traffic
(Showing Fraction of Top 100 AS-AS Traffic)
Network Observation – Bandwidth
1400
1200
top 100
sites to site
workflows
800
600
400
Jul, 06
Jan, 06
Jul, 05
Jan, 05
Jul, 04
Jan, 04
Jul, 03
Jan, 03
Jul, 02
Jan, 02
Jul, 01
Jan, 01
0
Jul, 00
200
Jan, 00
Terabytes / month
1000
ESnet Monthly Accepted Traffic, January, 2000 – June, 2006
• ESnet is currently transporting more than1 petabyte (1000 terabytes) per month
• More than 50% of the traffic is now generated by the top 100 sites — large-scale
science dominates all ESnet traffic
44
ESnet Traffic has Increased by
10X Every 47 Months, on Average, Since 1990
Apr., 2006
1 PBy/mo.
10000.0
Nov., 2001
100 TBy/mo.
53 months
1000.0
100.0
R2 = 0.9898
40 months
Oct., 1993
1 TBy/mo.
57 months
10.0
Aug., 1990
100 MBy/mo.
38 months
1.0
0.1
Log Plot of ESnet Monthly Accepted Traffic, January, 1990 – June, 2006
Jan, 06
Jan, 05
Jan, 04
Jan, 03
Jan, 02
Jan, 01
Jan, 00
Jan, 99
Jan, 98
Jan, 97
Jan, 96
Jan, 95
Jan, 94
Jan, 93
Jan, 92
Jan, 91
0.0
Jan, 90
Terabytes / month
Jul., 1998
10 TBy/mo.
45
Requirements from Network Utilization Observation
• In 4 years, we can expect a 10x increase in traffic over
current levels without the addition of production LHC
traffic
– Nominal average load on busiest backbone links is greater than
1 Gbps today
– In 4 years that figure will be over 10 Gbps if current trends
continue
• Measurements of this kind are science-agnostic
– It doesn’t matter who the users are, the traffic load is increasing
exponentially
• Bandwidth trends drive requirement for a new network
architecture
– New ESnet4 architecture designed with these drivers in mind
46
Requirements from Traffic Flow Observations
• Most ESnet science traffic has a source or sink outside of ESnet
– Drives requirement for high-bandwidth peering
– Reliability and bandwidth requirements demand that peering be
redundant
– Multiple 10 Gbps peerings today, must be able to add more flexibly and
cost-effectively
• Bandwidth and service guarantees must traverse R&E peerings
– “Seamless fabric”
– Collaboration with other R&E networks on a common framework is
critical
• Large-scale science is becoming the dominant user of the network
– Satisfying the demands of large-scale science traffic into the future will
require a purpose-built, scalable architecture
– Traffic patterns are different than commodity Internet
• Since large-scale science will be the dominant user going forward,
the network should be architected to serve large-scale science
47
Aggregation of Requirements from Network Observation
• Traffic load continues to increase exponentially
– 15-year trend indicates an increase of 10x in next 4 years
– This means backbone traffic load will exceed 10 Gbps within 4
years requiring increased backbone bandwidth
– Need new architecture – ESnet4
• Large science flows typically cross network
administrative boundaries, and are beginning to dominate
– Requirements such as bandwidth capacity, reliability, etc. apply
to peerings as well as ESnet itself
– Large-scale science is becoming the dominant network user
48
Other Networking Requirements
•
Production ISP Service for Lab Operations
– Captured in workshops, and in discussions with SLCCC (Lab
CIOs)
– Drivers are an enhanced set of standard business networking
requirements
– Traditional ISP service, plus enhancements (e.g. multicast)
– Reliable, cost-effective networking for business, technical, and
research operations
•
Collaboration tools for DOE science community
– Audio conferencing
– Video conferencing
49
Required Network Services Suite for DOE Science
•
We have collected requirements from diverse science programs, program
offices, and network analysis – the following summarizes the requirements:
– Reliability
• 99.95% to 99.999% reliability
• Redundancy is the only way to meet the reliability requirements
– Redundancy within ESnet
– Redundant peerings
– Redundant site connections where needed
– Connectivity
• Geographic reach equivalent to that of scientific collaboration
• Multiple peerings to add reliability and bandwidth to interdomain connectivity
• Critical both within the US and internationally
– Bandwidth
•
•
•
•
10 Gbps site to site connectivity today
100 Gbps backbone by 2010
Multiple 10+ Gbps R&E peerings
Ability to easily deploy additional lambdas and peerings
– Service guarantees
• All R&E networks must interoperate as one seamless fabric to enable end2end service deployment
• Guaranteed bandwidth, traffic isolation, quality of service
• Flexible rate bandwidth guarantees
– Collaboration support
• Federated trust, PKI (Grid, middleware)
• Audio and Video conferencing
– Production ISP service
50
Questions?
Thanks for listening!
51