Transcript ESnet Defined: Challenges and Overview Department of

ESnet4:
Networking for the Future of DOE
Science
ESnet R&D Roadmap Workshop, April 23-24, 2007
William E. Johnston
ESnet Department Head and Senior Scientist
Lawrence Berkeley National Laboratory
[email protected], www.es.net
1
 ESnet is an important, though somewhat specialized,
part of the US Research and Education infrastructure
•
The Office of Science (SC) is the single largest
supporter of basic research in the physical sciences
in the United States, … providing more than 40
percent of total funding … for the Nation’s research
programs in high-energy physics, nuclear physics,
and fusion energy sciences. (http://www.science.doe.gov)
•
SC will supports 25,500 PhDs, PostDocs, and
Graduate students, and half of the 21,500 users of
SC facilities come from universities
•
Almost 90% of ESnet’s 1+Petabyte/month of traffic
flows to and from the R&E community
2
DOE Office of Science and ESnet – the ESnet Mission
•
ESnet’s primary mission is to enable the largescale science that is the mission of the Office of
Science (SC) and that depends on:
–
–
–
–
–
Sharing of massive amounts of data
Supporting thousands of collaborators world-wide
Distributed data processing
Distributed data management
Distributed simulation, visualization, and computational
steering
– Collaboration with the US and International Research and
Education community
• ESnet provides network and collaboration services
to Office of Science laboratories and many other
DOE programs in order to accomplish its mission
3
 What ESnet Is
• A large-scale IP network built on a national circuit
infrastructure with high-speed connections to all major US and
international research and education (R&E) networks
• An organization of 30 professionals structured for the service
• An operating entity with an FY06 budget of $26.6M
• A tier 1 ISP (direct peerings will all major networks)
• The primary DOE network provider
– Provides production Internet service to all of the major DOE Labs* and
most other DOE sites
– Based on DOE Lab populations, it is estimated that between 50,000 100,000 users depend on ESnet for global Internet access
• additionally, each year more than 18,000 non-DOE researchers from
universities, other government agencies, and private industry use Office of
Science facilities
* PNNL supplements its ESnet service with commercial service
4
Office of Science US Community
Drives ESnet Design for Domestic Connectivity
Pacific Northwest
National Laboratory
Idaho National
Laboratory
Ames Laboratory
Argonne National
Laboratory
Fermi
National
Accelerator
Laboratory
Lawrence
Berkeley
National
Laboratory
Brookhaven
National
Laboratory
Stanford
Linear
Accelerator
Center
Princeton
Plasma
Physics
Laboratory
Lawrence
Livermore
National
Laboratory
Thomas Jefferson
National
Accelerator Facility
General
Atomics
Institutions supported by SC
Sandia
National
Major User Facilities
Laboratories
DOE Specific-Mission Laboratories
DOE Program-Dedicated Laboratories
DOE Multiprogram Laboratories
Oak Ridge
National
Laboratory
Los Alamos
National
Laboratory
National
Renewable Energy
Laboratory
Footprint of Largest SC Data Sharing Collaborators
Drives the International Footprint that ESnet Must Support
• Top 100 data flows generate 50% of all ESnet traffic (ESnet handles about 3x109 flows/mo.)
• 91 of the top 100 flows are from the Labs to other institutions (shown) (CY2005 data)
What Does ESnet Provide? - 1
•
An architecture tailored to accommodate DOE’s
large-scale science
– Move huge amounts of data between a small number of
sites that are scattered all over the world
•
Comprehensive connectivity
– High bandwidth access to DOE sites and DOE’s primary
science collaborators: Research and Education institutions
in the US, Europe, Asia Pacific, and elsewhere
•
Full access to the global Internet for DOE Labs
– ESnet is a tier 1 ISP managing a full complement of
Internet routes for global access
•
Highly reliable transit networking
– Fundamental goal is to deliver every packet that is received
to the “target” site
7
What Does ESnet Provide? - 2
•
A full suite of network services
– IPv4 and IPv6 routing and address space management
– IPv4 multicast (and soon IPv6 multicast)
– Primary DNS services
– Circuit services (layer 2 e.g. Ethernet VLANs), MPLS
overlay networks (e.g. SecureNet when it was ATM based)
– Scavenger service so that certain types of bulk traffic can
use all available bandwidth, but will give priority to any
other traffic when it shows up
– Prototype guaranteed bandwidth and virtual circuit services
8
What Does ESnet Provide? - 3
•
New network services
– Guaranteed bandwidth services
• Via a combination of QoS, MPLS overlay, and layer 2 VLANs
•
Collaboration services and Grid middleware
supporting collaborative science
– Federated trust services / PKI Certification Authorities with
science oriented policy
– Audio-video-data teleconferencing
•
Highly reliable and secure operation
– Extensive disaster recovery infrastructure
– Comprehensive internal security
– Cyberdefense for the WAN
9
What Does ESnet Provide? - 4
•
Comprehensive user support, including “owning” all
trouble tickets involving ESnet users (including
problems at the far end of an ESnet connection) until
they are resolved – 24x7x365 coverage
– ESnet’s mission is to enable the network based aspects of
OSC science, and that includes troubleshooting network
problems wherever they occur
•
A highly collaborative and interactive relationship
with the DOE Labs and scientists for planning,
configuration, and operation of the network
– ESnet and its services evolve continuously in direct
response to OSC science needs
– Engineering services for special requirements
10
ESnet History
ESnet0/MFENet
mid-1970s-1986
ESnet0/MFENet
56 Kbps microwave and
satellite links
ESnet1
1986-1995
ESnet formed to serve the
Office of Science
56 Kbps, X.25 to
45 Mbps T3
ESnet2
1995-2000
Partnered with Sprint to
build the first national
footprint ATM network
IP over 155 Mbps ATM
net
ESnet3
2000-2007
Partnered with Qwest to
build a national Packet over
SONET network and optical
channel Metropolitan Area
fiber
IP over 10Gbps SONET
Partner with Internet2 and
US Research& Education
community to build a
dedicated national optical
network
IP and virtual circuits on
a configurable optical
infrastructure with at
least 5-6 optical
channels of 10-100
Gbps each
11
transition
in progress
ESnet4
2007-2012
ESnet Science Data Network (SDN) core
ESnet3 Today Provides Global High-Speed Internet Connectivity for
DOE Facilities and Collaborators (Early 2007)
Japan (SINet)
Australia (AARNet)
Canada (CA*net4
Taiwan (TANet2)
Singaren
CA*net4
France
GLORIAD
(Russia, China)
Korea (Kreonet2
MREN
Netherlands
StarTap
Taiwan (TANet2,
ASCC)
GÉANT
- France, Germany,
Italy, UK, etc
SINet (Japan)
Russia (BINP)
CERN
(USLHCnet
DOE+CERN funded)
NSF/IRNC
funded
LIGO
PNNL
AU
MIT
JGI
TWC
LLNL
SNLL
LBNL
NERSC
SLAC
ESnet IP core:
Packet over SONET
Optical Ring and
Hubs
Lab DC
Offices
NASA
Ames
GA
AMPATH
Equinix
OSC GTN
NNSA
KCP
JLAB
OSTI
LANL
ARM
AU
PPPL
MAE-E
Equinix
PAIX-PA
Equinix, etc.
YUCCA MT
FNAL
ANL
AMES
SNLA
Allied
Signal
42 end user sites (S. America)
Office Of Science Sponsored (22)
NNSA Sponsored (13)
Joint Sponsored (3)
Other Sponsored (NSF LIGO, NOAA)
Laboratory Sponsored (6)
R&E
commercial peering points
networks Specific R&E network peers
Other R&E peering points
ESnet core hubs
high-speed peering points with Internet2/Abilene
BNL
ORNL
ORAU
NOAA
SRS
AMPATH
(S. America)
International (high speed)
10 Gb/s SDN core
10G/s IP core
2.5 Gb/s IP core
MAN rings (≥ 10 G/s)
Lab supplied links
OC12 ATM (622 Mb/s)
OC12 / GigEthernet
OC3 (155 Mb/s)
45 Mb/s and less
ESnet is a Highly Reliable Infrastructure
“5 nines” (>99.995%)
Dually connected sites
“4 nines” (>99.95%)
“3 nines”
Note: These availability measures are only for ESnet infrastructure, they do
not include site-related problems. Some sites, e.g. PNNL and LANL, provide
circuits from the site to an ESnet hub, and therefore the ESnet-site demarc
is at the ESnet hub (there is no ESnet equipment at the site. In this case,
circuit outages between the ESnet equipment and the site are considered
13
site issues and are not included in the ESnet availability metric.
ESnet is An Organization Structured for the Service
Network engineering, routing and network
services, WAN security
5.5
8.3
Deployment and WAN maintenance
Internal infrastructure, disaster recovery,
security
Applied R&D for new network services
(Circuit services and end-to-end monitoring)
7.4
1.5
Science collaboration services
(Public Key Infrastructure certification authorities,
AV conferencing, email lists, Web services)
5.5
Management, accounting, compliance
3.5
30.7 FTE (full-time staff) total
Network operations and user support
(24x7x365, end-to-end problem resolution
14
ESnet FY06 Budget is Approximately $26.6M
Approximate Budget Categories
SC Special
Projects:
$1.2M
SC R&D:
$0.5M
Carryover:
$1M
Other
DOE:
$3.8M
Target
Special projects
carryover:
(Chicago and LI MANs):
$1.0M
$1.2M
Management and
compliance: $0.7M
Collaboration
services: $1.6
Internal
infrastructure,
security, disaster
recovery: $3.4M
Circuits & hubs:
$12.7M
SC operating:
$20.1M
Operations:
$1.1M
Engineering &
research: $2.9M
WAN
equipment:
$2.0M
Total funds:
$26.6M
Total expenses:
$26.6M
15
Planning the Future Network - ESnet4
There are many stakeholders for ESnet
1. SC programs
–
–
–
–
–
–
–
Advanced Scientific Computing Research
Basic Energy Sciences
Biological and Environmental Research
Fusion Energy Sciences
High Energy Physics
Nuclear Physics
Office of Nuclear Energy
2. Major scientific facilities
– At DOE sites: large experiments, supercomputer centers, etc.
– Not at DOE sites: LHC, ITER
3. SC supported scientists not at the Labs
(mostly at US R&E institutions)
These account
for 85% of all
ESnet traffic
4. Other collaborating institutions
(mostly US, European, and AP R&E)
5. Other R&E networking organizations that support major collaborators
– Mostly US, European, and Asia Pacific networks
6. Lab operations (e.g. conduct of business) and general population
7. Lab networking organizations
16
Planning the Future Network - ESnet4
•
Requirements of the ESnet stakeholders are primarily
determined by
1) Data characteristics of instruments and facilities that
will be connected to ESnet
• What data will be generated by instruments coming on-line over the next
5-10 years?
• How and where will it be analyzed and used?
2) Examining the future process of science
• How will the processing of doing science change over 5-10 years?
• How do these changes drive demand for new network services?
3) Studying the evolution of ESnet traffic patterns
• What are the trends based on the use of the network in the past 2-5
years?
• How must the network change to accommodate the future traffic patterns
implied by the trends?
17
(1) Requirements from Instruments and Facilities
DOE SC Facilities that are, or will be, the top network users
•
Advanced Scientific Computing Research
– National Energy Research Scientific
Computing Center (NERSC) (LBNL)*
– National Leadership Computing Facility
(NLCF) (ORNL)*
– Argonne Leadership Class Facility (ALCF)
(ANL)*
•
•
– William R. Wiley Environmental Molecular
Sciences Laboratory (EMSL) (PNNL)*
– Joint Genome Institute (JGI)
– Structural Biology Center (SBC) (ANL)
•
– Alcator C-Mod (MIT)*
– National Spherical Torus Experiment (NSTX)
(PPPL)*
– ITER
– National Synchrotron Light Source (NSLS)
(BNL)
– Stanford Synchrotron Radiation Laboratory
(SSRL) (SLAC)
•
High Energy Physics
– Tevatron Collider (FNAL)
– B-Factory (SLAC)
– Large Hadron Collider (LHC, ATLAS, CMS)
(BNL, FNAL)*
– Spallation Neutron Source (ORNL)*
– National Center for Electron Microscopy
(NCEM) (LBNL)*
– Combustion Research Facility (CRF) (SNLL)*
Fusion Energy Sciences
– DIII-D Tokamak Facility (GA)*
Basic Energy Sciences
– Advanced Light Source (ALS) (LBNL)*
– Advanced Photon Source (APS) (ANL)
Biological and Environmental Research
•
Nuclear Physics
– Relativistic Heavy Ion Collider (RHIC) (BNL)*
– Continuous Electron Beam Accelerator
Facility (CEBAF) (JLab)*
*14 of 22 are characterized by current case studies
18
The Largest Facility: Large Hadron Collider at CERN
LHC CMS detector
15m X 15m X 22m,12,500 tons, $700M
human (for scale)
19
(2) Requirements from Examining
the Future Process of Science
• In a major workshop [1], and in subsequent updates [2],
requirements were generated by asking the science
community how their process of doing science will /
must change over the next 5 and next 10 years in
order to accomplish their scientific goals
• Computer science and networking experts then
assisted the science community in
– analyzing the future environments
– deriving middleware and networking requirements needed to
enable these environments
• These were complied as case studies that provide
specific 5 & 10 year network requirements for
bandwidth, footprint, and new services
20
Science Networking Requirements Aggregation Summary
Science
Drivers
End2End
Reliability
Connectivity
Science Areas
/ Facilities
Magnetic
Fusion Energy
NERSC and
ACLF
• DOE sites
(Impossible • US Universities
without full
• Industry
redundancy)
99.999%
-
• DOE sites
• US Universities
• International
• Other ASCR
Today
End2End
Band
width
5 years
End2End
Band
width
200+
Mbps
1 Gbps
10 Gbps
20 to 40
Gbps
Traffic
Characteristics
• Bulk data
• Remote control
• Bulk data
• Remote control
• Remote file
system sharing
supercomputers
NLCF
Nuclear
Physics (RHIC)
Spallation
Neutron Source
-
-
High
(24x7
operation)
• DOE sites
• US Universities
• Industry
• International
• DOE sites
• US Universities
• International
Backbone
Band
width
parity
Backbone
band width
parity
12 Gbps
70 Gbps
• DOE sites
640 Mbps
Network Services
• Guaranteed
bandwidth
• Guaranteed QoS
• Deadline scheduling
• Guaranteed
bandwidth
• Guaranteed QoS
• Deadline Scheduling
• PKI / Grid
• Bulk data
• Remote file
system sharing
• Bulk data
• Guaranteed
bandwidth
• PKI / Grid
2 Gbps
• Bulk data
Science Network Requirements Aggregation Summary
Science
Drivers
End2End
Reliability
Connectivity
Science Areas
/ Facilities
Advanced
Light Source
-
Bioinformatics
-
Chemistry /
Combustion
Climate
Science
-
-
• DOE sites
• US Universities
• Industry
• DOE sites
• US Universities
Today
End2End
Band
width
5 years
End2End
Band width
1 TB/day
5 TB/day
300 Mbps
1.5 Gbps
625 Mbps
250 Gbps
12.5
Gbps in
two years
• DOE sites
• US Universities
• Industry
-
• DOE sites
• US Universities
• International
-
Traffic
Characteristics
Network Services
• Bulk data
• Guaranteed
bandwidth
• Remote control
• PKI / Grid
• Bulk data
• Guaranteed
bandwidth
• Remote control
• High-speed
• Point-tomulticast
multipoint
10s of
Gigabits per
second
• Bulk data
5 PB per year
• Bulk data
• Guaranteed
• Remote control bandwidth
• PKI / Grid
5 Gbps
• Guaranteed
bandwidth
• PKI / Grid
Immediate Requirements and Drivers
High Energy
Physics (LHC)
99.95+%
(Less
than 4
hrs/year)
• US Tier1 (FNAL, BNL)
• US Tier2
(Universities)
• International (Europe,
Canada)
10 Gbps
60 to 80 Gbps
(30-40 Gbps
per US Tier1)
• Bulk data
• Coupled data
analysis
processes
• Guaranteed
bandwidth
• Traffic isolation
• PKI / Grid
3) These
Total Esnet Traffic
Fraction ofin
TopObserved
100 AS-AS Traffic)
Trends (Showing
are Seen
Evolution of
Historical ESnet Traffic Patterns
1400
1200
top 100
sites to site
workflows
800
600
400
Jul, 06
Jan, 06
Jul, 05
Jan, 05
Jul, 04
Jan, 04
Jul, 03
Jan, 03
Jul, 02
Jan, 02
Jul, 01
Jan, 01
0
Jul, 00
200
Jan, 00
Terabytes / month
1000
ESnet Monthly Accepted Traffic, January, 2000 – June, 2006
• ESnet is currently transporting more than1 petabyte (1000 terabytes) per month
• More than 50% of the traffic is now generated by the top 100 sites — large-scale
science dominates all ESnet traffic
23
ESnet Traffic has Increased by
10X Every 47 Months, on Average, Since 1990
Apr., 2006
1 PBy/mo.
10000.0
Nov., 2001
100 TBy/mo.
53 months
1000.0
100.0
R2 = 0.9898
40 months
Oct., 1993
1 TBy/mo.
57 months
10.0
Aug., 1990
100 MBy/mo.
38 months
1.0
0.1
Log Plot of ESnet Monthly Accepted Traffic, January, 1990 – June, 2006
Jan, 06
Jan, 05
Jan, 04
Jan, 03
Jan, 02
Jan, 01
Jan, 00
Jan, 99
Jan, 98
Jan, 97
Jan, 96
Jan, 95
Jan, 94
Jan, 93
Jan, 92
Jan, 91
0.0
Jan, 90
Terabytes / month
Jul., 1998
10 TBy/mo.
Requirements from Network Utilization Observation
•
In 4 years, we can expect a 10x increase in traffic over
current levels without the addition of production LHC traffic
– Nominal average load on busiest backbone links is ~1.5 Gbps today
– In 4 years that figure will be ~15 Gbps based on current trends
•
Measurements of this type are science-agnostic
– It doesn’t matter who the users are, the traffic load is increasing
exponentially
– Predictions based on this sort of forward projection tend to be
conservative estimates of future requirements because they cannot
predict new uses
•
Bandwidth trends drive requirement for a new network
architecture
– New architecture/approach must be scalable in a cost-effective way
25
0
FNAL -> CERN traffic is comparable to BNL -> CERN
but on layer 2 flows that are not yet monitored for traffic – soon)
NERSC (DOE Supercomputer) -> LBNL
Math. & Comp. (MICS)
Fermilab -> U. Florida
20
IN2P3 (France) -> Fermilab
LIGO (NSF)
Italy R&E -> SLAC
40
Argonne -> US Commodity
Fermilab -> U. Oklahoma
60
Fermilab -> UK R&E
80
UC San Diego -> Fermilab
100
BNL -> French R&E
Traffic Volume of the Top 30 AS-AS Flows, June 2006
(AS-AS = mostly Lab to R&E site, a few Lab to R&E
network, a few “other”)
Fermilab -> Swiss R&E
Fermilab -> Germany R&E
SLAC -> Karlsruhe (Germany)
Italy R&E -> Fermilab
PNNL -> Abilene (US R&E)
ESNET -> CalTech
Fermilab -> Belgium R&E
SLAC -> IN2P3 (France)
U. Neb.-Lincoln -> Fermilab
SLAC -> UK R&E
Abilene (US R&E) -> PNNL
Fermilab -> Germany R&E
RIKEN (Japan) -> BNL
Fermilab -> Estonia
SLAC -> Italy R&E
Fermilab -> DESY-Hamburg (Germany)
120
BNL -> RIKEN (Japan)
140
Fermilab -> Italy R&E
Fermilab -> MIT
Fermilab -> U. Neb.-Lincoln
CERN -> BNL
Terabytes
Top
30
AS-AS
flows,
June
2006
Large-Scale Flow Trends, June 2006
Subtitle: “Onslaught of the LHC”)
DOE Office of Science Program
LHC / High Energy
Physics - Tier 0-Tier1
LHC / HEP - T1-T2
HEP
Nuclear Physics
Lab - university
Lab - commodity
Traffic Patterns are Changing Dramatically
1/05
total traffic,
TBy
total traffic,
TBy
1200
1200
1000
1000
6/06
800
2 TB/month
800
600
600
400
400
2 TB/month
200
0
200
Jun. 06
Jan, 00
0
1200
1000
7/05
800
• While the total traffic is increasing
600
400
exponentially
200
0
Jul, 05
2 TB/month
– Peak flow – that is system-to-system
– bandwidth is decreasing
1200
1/06
– The number of large flows is
increasing
1000
800
600
400
200
0
Jan, 06
2 TB/month
27
The Onslaught of Grids
Question: Why is peak flow bandwidth decreasing while total traffic is increasing?
plateaus indicate the emergence of
parallel transfer systems (a lot of
systems transferring the same
amount of data at the same time)
Answer: Most large data transfers are now done by parallel / Grid data
movers
• In June, 2006 72% of the hosts generating the top 1000 flows were
involved in parallel data movers (Grid applications)
• This is the most significant traffic pattern change in the history of
ESnet
• This has implications for the network architecture that favor path
multiplicity and route diversity
28
What is the High-Level View of ESnet Traffic Patterns?
ESnet Inter-Sector Traffic Summary, Mar. 2006
48%
DOE sites
Inter-Lab
traffic
ESnet
~10%
7%
5%
12%
3%
R&E (mostly
universities)
Peering Points
58%
23%
43%
Traffic notes
• more than 90% of all traffic Office of Science
• less that 10% is inter-Lab
Commercial
International
(almost entirely
R&E sites)
Traffic coming into ESnet = Green
Traffic leaving ESnet = Blue
Traffic between ESnet sites
% = of total ingress or egress traffic
29
Requirements from Traffic Flow Observations
•
Most of ESnet science traffic has a source or sink outside of
ESnet
– Drives requirement for high-bandwidth peering
– Reliability and bandwidth requirements demand that peering be
redundant
– Multiple 10 Gbps peerings today, must be able to add more bandwidth
flexibly and cost-effectively
– Bandwidth and service guarantees must traverse R&E peerings
• Collaboration with other R&E networks on a common framework is critical
• Seamless fabric
• Large-scale science is now the dominant user of the network
– Satisfying the demands of large-scale science traffic into the future will
require a purpose-built, scalable architecture
– Traffic patterns are different than commodity Internet
30
Changing Science Environment  New Demands on Network
Requirements Summary
• Increased capacity
– Needed to accommodate a large and steadily increasing
amount of data that must traverse the network
• High network reliability
– Essential when interconnecting components of distributed
large-scale science
• High-speed, highly reliable connectivity between Labs
and US and international R&E institutions
– To support the inherently collaborative, global nature of largescale science
• New network services to provide bandwidth guarantees
– Provide for data transfer deadlines for
• remote data analysis, real-time interaction with instruments,
coupled computational simulations, etc.
31
 ESnet4 - The Response to the Requirements
I) A new network architecture and implementation strategy
• Rich and diverse network topology for flexible management and high
reliability
• Dual connectivity at every level for all large-scale science sources
and sinks
• A partnership with the US research and education community to
build a shared, large-scale, R&E managed optical infrastructure
•
a scalable approach to adding bandwidth to the network
•
dynamic allocation and management of optical circuits
II) Development and deployment of a virtual circuit service
• Develop the service cooperatively with the networks that are
intermediate between DOE Labs and major collaborators to ensure
and-to-end interoperability
32
 Next Generation ESnet: I) Architecture and Configuration
• Main architectural elements and the rationale for each element
1) A High-reliability IP core (e.g. the current ESnet core) to address
–
–
–
–
–
General science requirements
Lab operational requirements
Backup for the SDN core
Vehicle for science services
Full service IP routers
2) Metropolitan Area Network (MAN) rings to provide
–
–
–
–
Dual site connectivity for reliability
Much higher site-to-core bandwidth
Support for both production IP and circuit-based traffic
Multiply connecting the SDN and IP cores
2a) Loops off of the backbone rings to provide
– For dual site connections where MANs are not practical
3) A Science Data Network (SDN) core for
–
–
–
–
–
–
Provisioned, guaranteed bandwidth circuits to support large, high-speed science data flows
Very high total bandwidth
Multiply connecting MAN rings for protection against hub failure
Alternate path for production IP traffic
Less expensive router/switches
Initial configuration targeted at LHC, which is also the first step to the general configuration that will
address all SC requirements
– Can meet other unknown bandwidth requirements by adding lambdas
33
ESnet Target Architecture:
IP Core+Science Data Network Core+Metro Area Rings
international
connections
international
connections
international
connections
Loop off
Backbone
1625 miles / 2545 km
international
connections
Sunnyvale
SDN Core
New York
Denver
IP Core
Washington
DC
Metropolitan
Area Rings
LA
Albuquerque
San
Diego
IP core hubs
SDN hubs
international
connections
10-50 Gbps circuits
Production IP core
Science Data Network core
Metropolitan Area Networks
or backbone loops for Lab access
International connections
international
connections
Primary DOE Labs
Possible hubs
2700 miles / 4300 km
34
ESnet4
•
Internet2 has partnered with Level 3 Communications Co.
and Infinera Corp. for a dedicated optical fiber infrastructure
with a national footprint and a rich topology - the “Internet2
Network”
– The fiber will be provisioned with Infinera Dense Wave Division
Multiplexing equipment that uses an advanced, integrated opticalelectrical design
– Level 3 will maintain the fiber and the DWDM equipment
– The DWDM equipment will initially be provisioned to provide10 optical
circuits (lambdas - s) across the entire fiber footprint (80 s is max.)
•
ESnet has partnered with Internet2 to:
– Share the optical infrastructure
– Develop new circuit-oriented network services
– Explore mechanisms that could be used for the ESnet Network
Operations Center (NOC) and the Internet2/Indiana University NOC to
back each other up for disaster recovery purposes
35
ESnet4
•
ESnet will build its next generation IP network and
its new circuit-oriented Science Data Network
primarily on the Internet2 circuits (s) that are
dedicated to ESnet, together with a few National
Lambda Rail and other circuits
– ESnet will provision and operate its own routing and
switching hardware that is installed in various commercial
telecom hubs around the country, as it has done for the
past 20 years
– ESnet’s peering relationships with the commercial
Internet, various US research and education networks,
and numerous international networks will continue and
evolve as they have for the past 20 years
36
Internet2 and ESnet Optical Node
ESnet
IP
core
M320
ESnet
metro-area
networks
RON
T640
SDN
core
switch
support devices:
•measurement
•out-of-band access
•monitoring
•security
dynamically
allocated and
routed waves
(future)
grooming
Ciena device
CoreDirector
Direct Optical
Connections
to RONs
Network Testbeds
support devices:
•measurement
•out-of-band access
•monitoring
•…….
various equipment and
experimental control plane
management systems
Future access to control plane
fiber east
fiber west
Infinera DTN
fiber north/south
Internet2/Level3
National Optical Infrastructure
37
ESnet4
•
ESnet4 will also involve an expansion of the multi10Gb/s Metropolitan Area Rings in the
San Francisco Bay Area, Chicago, Long Island,
Newport News (VA/Washington, DC area), and
Atlanta
– provide multiple, independent connections for ESnet sites
to the ESnet core network
– expandable
•
Several 10Gb/s links provided by the Labs that will
be used to establish multiple, independent
connections to the ESnet core
– currently PNNL and ORNL
38
ESnet Metropolitan Area Network Ring Architecture for High Reliability Sites
US
LHCnet
switch
SDN
core
west
SDN
core
east
US
LHCnet
switch
SDN
core
switch
ESnet production IP core hub
IP core
west
SDN
core
switch
IP core
router
ESnet
IP core hub
ESnet SDN
core hub
MAN fiber ring: 2-4 x 10 Gbps channels provisioned initially,
with expansion capacity to 16-64
Large Science Site
ESnet MAN
switch
ESnet production
IP service
ESnet managed
λ / circuit services
Independent
port card
supporting
multiple 10 Gb/s
line interfaces
SDN circuits
to site systems
IP core
east
ESnet switch
Virtual
Circuit to
Site
Site
router
Site LAN
Site edge router
ESnet managed
virtual circuit services
tunneled through the
IP backbone
Site
Virtual Circuits
to Site
Site gateway router
39
ESnet4 Roll Out
ESnet4 IP + SDN Configuration, mid-September, 2007
All circuits are 10Gb/s, unless noted.
Seattle
(28)
Portland
(29)
Boise
Boston
(9)
(7)
Chicago
(11)
Clev.
(10)
Sunnyvale
NYC
(32)
Denver
Salt
Lake
City
San Diego
(30)
(22)
Raleigh
Tulsa
Nashville
OC48
(1(3))
(3)
(4)
(1)
(2)
(20)
(19)
El Paso
Jacksonville
(17)
(6)
ESnet IP switch/router hubs
ESnet IP switch only hubs
Houston
(5)
ESnet SDN switch hubs
Layer 1 optical nodes at eventual ESnet Points of Presence
Layer 1 optical nodes not currently in ESnet plans
Lab site
Wash DC
(21)
Albuq.
(24)
(26)
Pitts.
(0)
LA
Philadelphia
KC
(15)
(23)
(25)
(13)
Baton
Rouge
ESnet IP core
ESnet Science Data Network core
ESnet SDN core, NLR links
Lab supplied link
LHC related link
MAN link
International IP Connections
ESnet4 Metro Area Rings, 2007 Configurations
Long Island MAN
West Chicago MAN
600 W.
Chicago
Seattle
USLHCNet
32 AoA, NYC
Starlight
(28)
Portland
BNL
(29)
BoiseUSLHCNet
Boston
(9)
Sunnyvale
(7)
Chicago
FNAL
Denver
(32)
Salt
Lake
City
San Francisco
(23)
Bay Area MAN
LA
(24)
(19)
Philadelphia
KC
SNLL
ESnet IP switch only hubs
ESnet SDN switch hubs
(30)
Raleigh
Tulsa
Nashville
OC48
(1(3))
(3)
(4)
Newport News - Elite
Atlanta
(2)
(20)
Jacksonville
(17)
(6)
Atlanta MAN
ORNL
Nashville
All circuits are 10Gb/s. Wash., DC
56 Marietta
(SOX)
Layer 1 optical nodes at eventual ESnet Points of Presence
Layer 1 optical nodes not currently in ESnet plans
Lab site
Wash DC
(22)
NERSC
LLNL
ESnet IP switch/router hubs
(26)
Pitts.
(0)
El Paso
(25)
(21)
Albuq.
SLAC
(10)
(13)
LBNL(1)
San Diego
Clev.
NYC
ANL
(15)
JGI
(11)
180 Peachtree
Houston
Wash.,
DC
MATP
JLab
ESnet IP core
ELITE
ESnet Science Data
Network core
ESnet SDN core, NLR links (existing)
Lab suppliedODU
link
LHC related link
MAN link
International IP Connections
41
LHC Tier 0, 1, and 2 Connectivity Requirements Summary
CERN-1 CERN-2
CERN-3
TRIUMF
(Atlas T1,
Canada)
Vancouver
CANARIE
USLHCNet
Seattle
Toronto
ESnet
SDN
BNL
(Atlas T1)
Virtual Circuits
Boise
Chicago
Denver
KC
ESnet
IP Core
Internet2
Internet2 // RONs
RONs
FNAL
(CMS T1)
Wash DC
Albuq.
San Diego
Dallas
GÉANT
Atlanta
GÉANT-2
LA
GÉANT-1
Sunnyvale
New York
Jacksonville
USLHC nodes
• Direct connectivity T0-T1-T2
Internet2/GigaPoP nodes
ESnet IP core hubs
• USLHCNet to ESnet to Abilene
ESnet SDN/NLR hubs
Tier 1 Centers
Cross connects ESnet - Internet2
Tier 2 Sites
• Backup connectivity
• SDN, GLIF, VCs
42
ESnet4 2007-8 Estimated Bandwidth Commitments
Long Island MAN
600
W. Chicago
West
Chicago
MAN
CERN
CMS
5
Seattle
USLHCNet
BNL
(28)
Portland
CERN
32 AoA, NYC
Starlight
Boise
(29)
13
Sunnyvale
(32)
(23)
Bay Area MAN
LA
Chicago
10
(24)
SLAC
(19)
Philadelphia
KC
(15)
Wash DC
(30)
Raleigh
Tulsa
ANL
Nashville
OC48
(1(3))
(3)
(4)
Newport News - Elite
(2)
(20)
Jacksonville
(17)
(6)
LLNL
ESnet IP switch/router hubs
ESnet SDN switch hubs
(26)
Pitts.
Atlanta
NERSC
ESnet IP switch only hubsSNLL
(25)
(22)
Albuq.
El Paso
(10)
NYC
LBNL(1)
San Diego
Clev.
(21)
(0)
JGIFNAL
(11)
(13)
Denver
Salt
Lake
City
San Francisco
Boston
(9)
29
(total)
(7)
USLHCNet
10
Houston
(5)
All circuits are 10Gb/s.
Layer 1 optical nodes at eventual ESnet Points of Presence
Layer 1 optical nodes not currently in ESnet plans
Committed bandwidth, Gb/s
2.5
Lab site
Baton
MAX
Rouge
Wash.,
DC
MATP
JLab
ESnet IP core
ELITE
ESnet Science Data
Network core
ESnet SDN core, NLR links (existing)
Lab suppliedODU
link
LHC related link
MAN link
International IP Connections
43
Aggregate Estimated Link Loadings, 2007-08
9
12.5
Seattle
13
(28)
Portland
(29)
Boise
13
9
Existing site
supplied
circuits
2.5
Boston
(9)
(7)
Chicago
(11)
Clev.
(10)
Sunnyvale
NYC
(32)
Denver
Salt
Lake
City
San Diego
Pitts.
(30)
Nashville
OC48
(1(3))
(3)
El Paso
6
(4)
Atlanta
(2)
(20)
(19)
8.5
Raleigh
Tulsa
(1)
6
Jacksonville
(17)
(6)
ESnet IP switch/router hubs
ESnet IP switch only hubs
Wash DC
(22)
Albuq.
(24)
(26)
(21)
(0)
LA
Philadelphia
KC
(15)
(23)
(25)
(13)
Houston
(5)
Baton
Rouge
ESnet SDN switch hubs
Layer 1 optical nodes at eventual ESnet Points of Presence
Layer 1 optical nodes not currently in ESnet plans
Committed bandwidth, Gb/s
2.5
Lab site
2.5
ESnet IP core (1)
ESnet Science Data Network core
ESnet SDN core, NLR links
Lab supplied link
LHC related link
MAN link
International IP Connections
44
ESnet4 IP + SDN, 2008 Configuration
Seattle
(28)
Portland
(? )
(2)
(29)
Boise
(2)
(7)
(1)
Chicago
Sunnyvale
(2)
San Diego
(13)
Denver
(1)
Albuq.
(1)
(2)
(22)
(0)
(1)
(2)
Nashville
(30)
OC48
(2)
(1)
(1)
(4)
(3)
Atlanta
(2)
(20)
(19)
(1)
(17)
Jacksonville
(1)
ESnet IP switch/router hubs
ESnet IP switch only hubs
(6)
(5)
Houston
Baton
Rouge
ESnet SDN switch hubs
Layer 1 optical nodes at eventual ESnet Points of Presence
Layer 1 optical nodes not currently in ESnet plans
Lab site
Wash. DC
Raleigh
Tulsa
(1)
El Paso
Philadelphia
(2) (26)
(1)
(21)
(2)
(10)
(25)
(2)
KC
(15)
(1)
(24)
(2)
NYC
Salt
Lake
City
LA
Clev.
(2)
(32)
(23)
(11)
Boston
(9)
(20)
ESnet IP core
ESnet Science Data Network core
ESnet SDN core, NLR links (existing)
Lab supplied link
LHC related link
MAN link
International IP Connections
Internet2 circuit number
45
ESnet4 2009 Configuration
(Some of the circuits may be allocated dynamically from shared a pool.)
Seattle
(28)
Portland
(? )
3
(29)
Boise
Boston
(7)
2
3
Chicago
(11)
Sunnyvale
3
(13)
Denver
Salt
Lake
City
2
San Diego
3
3
2
Albuq.
Wash. DC
2
(22)
Tulsa
(30)
OC48
(4)
(3) 2
1
(1)
Atlanta
(2)
(20)
El Paso
2
(17)
Raleigh
3
Nashville
2
(19)
Jacksonville
2
ESnet IP switch/router hubs
ESnet IP switch only hubs
(6)
(5)
Houston
Baton
Rouge
ESnet SDN switch hubs
Layer 1 optical nodes at eventual ESnet Points of Presence
Layer 1 optical nodes not currently in ESnet plans
Lab site
Philadelphia
3 (26)
2
2
2
(25)
(21)
(0)
(24)
3 (10)
KC
(15)
(23)
LA
Clev.
NYC
3
(32)
(9)
(20)
ESnet IP core
ESnet Science Data Network core
ESnet SDN core, NLR links (existing)
Lab supplied link
LHC related link
MAN link
International IP Connections
Internet2 circuit number
Aggregate Estimated Link Loadings, 2010-11
30
45
Seattle
(28)
Portland
50
5
(29)
Boise
Sunnyvale
Boston
(7)
4
LA
(24)
San Diego
20
(13)
Denver
(25)
5
Philadelphia
5 (26)
(21)
5
4
Albuq.
Tulsa
5
(20)
El Paso
(17)
Raleigh
5
OC48
(4)
(3) 3
3
10
(30)
Nashville
4
(1)
Wash. DC
3
(22)
(0)
(19)
20
Atlanta
(2)
5
4
Jacksonville
4
ESnet IP switch/router hubs
20
(6)
(5)
Houston
Baton
Rouge
ESnet SDN switch hubs
Layer 1 optical nodes at eventual ESnet Points of Presence
Layer 1 optical nodes not currently in ESnet plans
Lab site
5 (10)
4
5
ESnet IP switch only hubs
Clev.
KC
(15)
4
4
(11)
(9)
NYC
5
Salt
Lake
City
(23)
5
Chicago
(32)
4
20
15
(>1 )
(20)
ESnet IP core (1)
ESnet Science Data Network core
ESnet SDN core, NLR links (existing)
Lab supplied link
LHC related link
MAN link
International IP Connections
Internet2 circuit number
47
ESnet4 2010-11 Estimated Bandwidth Commitments
600 W. Chicago
CMS
CERN
25
40
Seattle
BNL
(28)
Portland
15
(>1 )
32 AoA, NYC
CERN
5
(29)
Boise
Starlight
65
(7)
4
Sunnyvale
20
USLHCNet
25
LA
(24)
San Diego
20
4
(20)
El Paso
(17)
Wash. DC
(30)
5
OC48
(4)
(3) 3
10
Atlanta
(2)
5
4
Jacksonville
10
(6)
(5)
Houston
Baton
Rouge
ESnet SDN switch hubs
Layer 1 optical nodes at eventual ESnet Points of Presence
Layer 1 optical nodes not currently in ESnet plans
Lab site
Raleigh
5
Nashville
4
ESnet IP switch/router hubs
ESnet IP switch only hubs
4
3
(1)
(19)
Philadelphia
5 (26)
3
(22)
Tulsa
Albuq.
40 ANL
4
5
(21)
(0)
FNAL
(25)
100
80
80
5
4
4
5 (10)
KC
(15)
5
(23)
(13)
Denver
Salt
Lake
City
4
(11)
Clev.
NYC
5
(32)
USLHCNet
5
Chicago
Boston
(9)
(20)
ESnet IP core (1)
ESnet Science Data Network core
ESnet SDN core, NLR links (existing)
Lab supplied link
LHC related link
MAN link
International IP Connections
Internet2 circuit number
48
ESnet4 IP + SDN, 2011 Configuration
Seattle
(28)
Portland
(>1 )
5
(29)
Boise
(7)
4
Sunnyvale
Boston
(13)
Denver
Salt
Lake
City
San Diego
5
5
4
Albuq.
Wash. DC
3
(22)
Tulsa
(30)
OC48
(4)
(3) 3
3
(1)
Atlanta
(2)
(20)
El Paso
4
(17)
Raleigh
5
Nashville
4
(19)
Jacksonville
4
ESnet IP switch/router hubs
ESnet IP switch only hubs
(6)
(5)
Houston
Baton
Rouge
ESnet SDN switch hubs
Layer 1 optical nodes at eventual ESnet Points of Presence
Layer 1 optical nodes not currently in ESnet plans
Lab site
Philadelphia
5 (26)
4
4
4
(25)
(21)
(0)
(24)
5 (10)
KC
(15)
(23)
LA
(11)
Clev.
NYC
5
(32)
4
5
Chicago
(9)
(20)
ESnet IP core (1)
ESnet Science Data Network core
ESnet SDN core, NLR links (existing)
Lab supplied link
LHC related link
MAN link
International IP Connections
Internet2 circuit number
49
Typical ESnet4 Hub
To AoA,
NYC
To
Cleveland
GE
AOA
CL
t
o
AO V
AE
to
V
LE
2xT1
2xT1
10
MAX GE
L a mb
da
7609
10
1GE SX
WDC-CR1
ORAUDC
WDC-AR1
7206VXR
M320
1GE LX? GE to Eqx-ASH
OC3c
DS3
to
AT
L
1GE
SX
DS3
OC3 SM to DOE
T3 to DOE-RT1
T3 to NGA
WDC-PR1
GE
M7i
10
SD
N
JLAB
Foundry
10 GE
10 GE
E
G
MATP
7609
ITE
L
E
DClabs
LLNL-DC
T1
WDC-SDN1
MAX NGIX-E
Coillege Park
To
Atlanta
10
GE
10
C
to
GE
10
10 GE
10 G
E to
SD
N
N
SD
To GEANT
NGIX-E
6509
MAX
WDC L(3)
3 racks
10
E
G
to
L
AT
MAE-E DS3
50
The Evolution of ESnet Architecture
ESnet IP
core
ESnet to 2005:
ESnet IP
core
ESnet Science Data
Network (SDN) core
• A routed IP network with sites
singly attached to a national core
ring
ESnet sites
ESnet hubs / core network connection points
Metro area rings (MANs)
Other IP networks
Circuit connections to other science networks (e.g. USLHCNet)
independent
redundancy
(TBD)
ESnet from 2006-07:
• A routed IP network with sites
dually connected on metro area
rings or dually connected directly to
core ring
• A switched network providing
virtual circuit services for dataintensive science
• Rich topology offsets the lack of
dual, independent national cores
51
ESnet4 Planed Configuration
Core networks: 40-50 Gbps in 2009-2010, 160-400 Gbps in 2011-2012
Canada
Canada
Asia-Pacific
(CANARIE)
Asia Pacific
(CANARIE)
CERN (30 Gbps)
CERN (30 Gbps)
GLORIAD
Europe
(Russia and
China)
(GEANT)
Boston
Australia
1625 miles / 2545 km
Science Data
Network Core
Boise
IP Core
New York
Denver
Washington
DC
Australia
Tulsa
LA
Albuquerque
San Diego
South America
IP core hubs
(AMPATH)
SDN (switch) hubs
Primary DOE Labs
Core network fiber path is
High speed cross-connects
~ 14,000 miles / 24,000 km
with Ineternet2/Abilene
Possible hubs
2700 miles / 4300 km
South America
(AMPATH)
Jacksonville
Production IP core (10Gbps) ◄
SDN core (20-30-40Gbps) ◄
MANs (20-60 Gbps) or
backbone loops for site access
International connections
52
Next Generation ESnet: II) Virtual Circuits
• Traffic isolation and traffic engineering
– Provides for high-performance, non-standard transport mechanisms that
cannot co-exist with commodity TCP-based transport
– Enables the engineering of explicit paths to meet specific requirements
• e.g. bypass congested links, using lower bandwidth, lower latency paths
• Guaranteed bandwidth (Quality of Service (QoS))
– User specified bandwidth
– Addresses deadline scheduling
• Where fixed amounts of data have to reach sites on a fixed schedule,
so that the processing does not fall far enough behind that it could never
catch up – very important for experiment data analysis
• Reduces cost of handling high bandwidth data flows
– Highly capable routers are not necessary when every packet goes to the
same place
– Use lower cost (factor of 5x) switches to relatively route the packets
• Secure
– The circuits are “secure” to the edges of the network (the site boundary)
because they are managed by the control plane of the network which is
isolated from the general traffic
• Provides end-to-end connections between Labs and collaborator
institutions
53
Virtual Circuit Service Functional Requirements
• Support user/application VC reservation requests
– Source and destination of the VC
– Bandwidth, start time, and duration of the VC
– Traffic characteristics (e.g. flow specs) to identify traffic designated for the VC
• Manage allocations of scarce, shared resources
– Authentication to prevent unauthorized access to this service
– Authorization to enforce policy on reservation/provisioning
– Gathering of usage data for accounting
• Provide circuit setup and teardown mechanisms and security
– Widely adopted and standard protocols (such as MPLS and GMPLS) are well
understood within a single domain
– Cross domain interoperability is the subject of ongoing, collaborative
development
– secure and-to-end connection setup is provided by the network control plane
• Enable the claiming of reservations
– Traffic destined for the VC must be differentiated from “regular” traffic
• Enforce usage limits
– Per VC admission control polices usage, which in turn facilitates guaranteed
bandwidth
– Consistent per-hop QoS throughout the network for transport predictability
54
ESnet Virtual Circuit Service: OSCARS
(On-demand Secured Circuits and Advanced Reservation System)
Software Architecture (see Ref. 9)
•
•
•
•
Web-Based User Interface (WBUI) will prompt the user for a
username/password and forward it to the AAAS.
Authentication, Authorization, and Auditing Subsystem (AAAS) will
handle access, enforce policy, and generate usage records.
Bandwidth Scheduler Subsystem (BSS) will track reservations and map
the state of the network (present and future).
Path Setup Subsystem (PSS) will setup and teardown the on-demand
paths (LSPs).
User
request
via WBUI
User
Human
User
Web-Based
User Interface
Reservation Manager
Path Setup
Subsystem
User
feedback
User
Application
User app request via
AAAS
Authentication,
Authorization,
And Auditing
Subsystem
Bandwidth
Scheduler
Subsystem
Instructions to
routers and
switches to
setup/teardown
LSPs
The Mechanisms Underlying OSCARS
Based on Source and Sink IP addresses, route of LSP between ESnet border routers is determined
using topology information from OSPF-TE. Path of LSP can be explicitly directed to take SDN network.
On the SDN Ethernet switches all traffic is MPLS switched (layer 2.5), which stitches together VLANs
On ingress to ESnet,
packets matching
reservation profile are
filtered out (i.e. policy
based routing),
policed to reserved
bandwidth, and
injected into a LSP.
Source
VLAN 1
VLAN 2
VLAN 3
SDN
SDN
SDN
RSVP, MPLS
enabled on
internal interfaces
Label Switched Path
IP Link
IP
IP
Sink
IP
high-priority
queue
standard,
best-effort
queue
MPLS labels are attached onto packets from Source and
placed in separate queue to ensure guaranteed bandwidth.
Interface queues
Regular production traffic queue.
56
Environment of Science is Inherently Multi-Domain
•
End points will be at independent institutions – campuses or
research institutes - that are served by ESnet, Abilene,
GÉANT, and their regional networks
– Complex inter-domain issues – typical circuit will involve five or more
domains - of necessity this involves collaboration with other networks
– For example, a connection between FNAL and DESY involves five
domains, traverses four countries, and crosses seven time zones
FNAL (AS3152)
[US]
GEANT (AS20965)
[Europe]
ESnet (AS293)
[US]
DESY (AS1754)
[Germany]
DFN (AS680)
[Germany]
57
OSCARS: Guaranteed Bandwidth VC Service For SC Science
•
To ensure compatibility, the design and implementation is done in collaboration
with the other major science R&E networks and end sites
– Internet2: Bandwidth Reservation for User Work (BRUW)
• Development of common code base
– GEANT: Bandwidth on Demand (GN2-JRA3), Performance and Allocated Capacity for
End-users (SA3-PACE) and Advance Multi-domain Provisioning System (AMPS)
extends to NRENs
– BNL: TeraPaths - A QoS Enabled Collaborative Data Sharing Infrastructure for Petascale Computing Research
– GA: Network Quality of Service for Magnetic Fusion Research
– SLAC: Internet End-to-end Performance Monitoring (IEPM)
– USN: Experimental Ultra-Scale Network Testbed for Large-Scale Science
– DRAGON/HOPI: Optical testbed
•
In its current phase this effort is being funded as a research project by the Office
of Science, Mathematical, Information, and Computational Sciences (MICS)
Network R&D Program
•
A prototype service has been deployed as a proof of concept
– To date more then 20 accounts have been created for beta users, collaborators, and
developers
– More then 100 reservation requests have been processed
58
ESnet Virtual Circuit Service Roadmap
• Dedicated virtual circuits
• Dynamic virtual circuit allocation
• Generalized MPLS (GMPLS)
Initial production service
2005
2006
2007
2008
Full production service
• Interoperability between GMPLS circuits,
VLANs, and MPLS circuits (layer 1-3)
• Interoperability between VLANs and MPLS circuits
(layer 2 & 3)
• Dynamic provisioning of Multi-Protocol Label Switching
(MPLS) circuits in IP nets (layer 3) and in VLANs for
Ethernets (layer 2)
59
Federated Trust Services – Support for Large-Scale Collaboration
•
Remote, multi-institutional, identity authentication is critical
for distributed, collaborative science in order to permit
sharing widely distributed computing and data resources, and
other Grid services
•
Public Key Infrastructure (PKI) is used to formalize the
existing web of trust within science collaborations and to
extend that trust into cyber space
– The function, form, and policy of the ESnet trust services are driven
entirely by the requirements of the science community and by direct
input from the science community
• International scope trust agreements that encompass many
organizations are crucial for large-scale collaborations
– ESnet has lead in negotiating and managing the cross-site, crossorganization, and international trust relationships to provide policies
that are tailored for collaborative science
 This service, together with the associated ESnet PKI service, is the
basis of the routine sharing of HEP Grid-based computing resources
between US and Europe
60
DOEGrids CA (one of several CAs) Usage Statistics
20000
19000
18000
17000
16000
No.of certificates or requests
15000
14000
13000
User Certificates
12000
Service Certificates
11000
10000
Expired(+revoked) Certificates
9000
8000
Total Certificates Issued
7000
Total Cert Requests
6000
5000
4000
3000
2000
1000
0
Jan2003
Apr2003
Jul2003
Oct2003
Jan2004
Apr2004
Jul2004
Oct2004
Jan2005
Apr2005
Jul2005
Oct2005
Jan2006
Apr2006
Jul2006
Oct2006
Jan2007
Production service began in June 2003
User Certificates
4307 Total No. of Active Certificates
5344
Host & Service Certificates
8813 Total No. of Expired Certificates
7800
Total No. of Requests
ESnet SSL Server CA Certificates
FusionGRID CA certificates
16384 Total No. of Certificates Issued
13144
45
128
* Report as of Feb 4, 2007
61
DOEGrids CA Usage - Virtual Organization Breakdown
**OSG
10.2%
DOEGrids CA Statistics (5344)
*Others
ESG
19.6%
ESnet
ANL
0.77%
0.30%
2.40%
FusionGRID
0.71%
iVDGL
18.38%
LBNL
0.90%
NERSC
1.55%
LCG
1.53%
ORNL
0.56%
PNNL
0.02%
FNAL
31.72%
* DOE-NSF collab. & Auto renewals
PPDG
14.50%
** OSG Includes (BNL, CDF, CMS, DES, DOSAR, DZero, Fermilab, fMRI, GADU, geant4, GLOW, GRASE, GridEx,
GROW, i2u2, iVDGL, JLAB, LIGO, mariachi, MIS, nanoHUB, NWICG, OSG, OSGEDU, SDSS, SLAC, STAR & USATLAS)
62
DOEGrids CA (Active Certificates) Usage Statistics
6000
5500
5000
No.of certificates or requests
4500
4000
3500
Active User Certificates
3000
Active Service Certificates
2500
Total Active Certificates
2000
1500
1000
500
0
Jan-
Apr-
Jul-
Oct-
Jan-
Apr-
Jul-
Oct-
Jan-
Apr-
Jul-
Oct-
Jan-
Apr-
Jul-
Oct-
Jan-
2003
2003
2003
2003
2004
2004
2004
2004
2005
2005
2005
2005
2006
2006
2006
2006
2007
Production service began in June 2003
* Report as of Feb 4, 2007
63
 ESnet Conferencing Service (ECS)
•
A highly successful ESnet Science Service that provides
audio, video, and data teleconferencing service to support
human collaboration of DOE science
– Seamless voice, video, and data teleconferencing is important for
geographically dispersed scientific collaborators
– Provides the central scheduling essential for global collaborations
– ESnet serves more than a thousand DOE researchers and
collaborators worldwide
• H.323 (IP) videoconferences (4000 port hours per month and rising)
• audio conferencing (2500 port hours per month) (constant)
• data conferencing (150 port hours per month)
• Web-based, automated registration and scheduling for all of these
services
– Very cost effective (saves the Labs a lot of money)
64
ESnet Collaboration Services (ECS) 2007
Sycamore
Networks
DSU/CSU
PST
N
6 - T1s
Latitude Web
Collaboration
Server
(data)
Latitude
Audio Bridge
(144 ports)
6 - T1s
(144 x 64kb
voice channels)
(monitoring)
(monitoring)
1 - PRI
(23 x 64kb
ISDN channels)
Codian
ISDN to IP video
gateway
(to MCUs)
Codian MCU 1
Tandberg
Management
Suite
•
•
•
•
•
•
Codian MCU 2
Tandberg
Gatekeeper
(“DNS” for H.323)
Codian MCU 3
ESnet
High Quality videoconferencing over IP and ISDN
Reliable, appliance based architecture
Ad-Hoc H.323 and H.320 multipoint meeting creation
Web Streaming options on 3 Codian MCU’s using Quicktime or Real
Real-time audio and data collaboration including desktop and application sharing
Web-based registration and audio/data bridge scheduling
65
Summary
• ESnet is currently satisfying its mission by enabling SC
science that is dependant on networking and distributed,
large-scale collaboration:
“The performance of ESnet over the past year has been
excellent, with only minimal unscheduled down time. The
reliability of the core infrastructure is excellent.
Availability for users is also excellent” - DOE 2005 annual
review of LBL
• ESnet has put considerable effort into gathering
requirements from the DOE science community, and has
a forward-looking plan and expertise to meet the five-year
SC requirements
– A Lehman review of ESnet (Feb, 2006) has strongly endorsed the
plan presented here
66
References
1.
High Performance Network Planning Workshop, August 2002
–
2.
3.
http://www.doecollaboratory.org/meetings/hpnpw
Science Case Studies Update, 2006 (contact [email protected])
DOE Science Networking Roadmap Meeting, June 2003
–
4.
http://www.es.net/hypertext/welcome/pr/Roadmap/index.html
DOE Workshop on Ultra High-Speed Transport Protocols and Network Provisioning for
Large-Scale Science Applications, April 2003
–
5.
http://www.csm.ornl.gov/ghpn/wk2003
Science Case for Large Scale Simulation, June 2003
–
6.
http://www.pnl.gov/scales/
Workshop on the Road Map for the Revitalization of High End Computing, June 2003
–
–
7.
http://www.cra.org/Activities/workshops/nitrd
http://www.sc.doe.gov/ascr/20040510_hecrtf.pdf (public report)
ASCR Strategic Planning Workshop, July 2003
–
8.
http://www.fp-mcs.anl.gov/ascr-july03spw
Planning Workshops-Office of Science Data-Management Strategy, March & May 2004
–
9.
http://www-conf.slac.stanford.edu/dmw2004
For more information contact Chin Guok ([email protected]). Also see
-
http://www.es.net/oscars
67
ESnet Network Measurements
ESCC Feb 15 2007
Joe Metzger
[email protected]
68
Measurement Motivations
•
Users dependence on the network is increasing
– Distributed Applications
– Moving Larger Data Sets
– The network is becoming a critical part of large science experiments
•
•
The network is growing more complex
– 6 core devices in 05’,
25+ in 08’
– 6 core links in 05’,
40+ in 08’, 80+ by 2010?
Users continue to report performance problems
– ‘wizards gap’ issues
•
The community needs to better understand the network
– We need to be able to demonstrate that the network is good.
– We need to be able to detect and fix subtle network problems.
69
perfSONAR
•
perfSONAR is a global collaboration to design, implement
and deploy a network measurement framework.
– Web Services based Framework
•
•
•
•
•
Measurement Archives (MA)
Measurement Points (MP)
Lookup Service (LS)
Topology Service (TS)
Authentication Service (AS)
– Some of the currently Deployed Services
•
•
•
•
•
•
Utilization MA
Circuit Status MA & MP
Latency MA & MP
Bandwidth MA & MP
Looking Glass MP
Topology MA
– This is an Active Collaboration
• The basic framework is complete
• Protocols are being documented
• New Services are being developed and deployed.
70
perfSONAR Collaborators
•
•
•
•
•
•
•
•
•
•
•
•
ARNES
Belnet
CARnet
CESnet
Dante
University of Delaware
DFN
ESnet
FCCN
FNAL
GARR
GEANT2
•
•
•
•
•
•
•
•
•
•
•
•
Georga Tech
GRNET
Internet2
IST
POZNAN Supercomputing Center
Red IRIS
Renater
RNP
SLAC
SURFnet
SWITCH
Uninett
* Plus others who are contributing, but haven’t added
their names to the list on the WIKI.
71
perfSONAR Deployments
16+ different networks have deployed at least 1 perfSONAR
service (Jan 2007)
72
ESnet perfSONAR Progress
•
ESnet Deployed Services
– Link Utilization Measurement Archive
– Virtual Circuit Status
•
In Development
– Active Latency and Bandwidth Tests
– Topology Service
– Additional Visualization capabilities
•
perfSONAR visualization tools showing ESnet data
– Link Utilization
• perfSONARUI
– http://perfsonar.acad.bg/
• VisualPerfSONAR
– https://noc-mon.srce.hr/visual_perf
• Traceroute Visualizer
– https://performance.es.net/cgi-bin/level0/perfsonar-trace.cgi
– Virtual Circuit Status
• E2EMon (for LHCOPN Circuits)
– http://cnmdev.lrz-muenchen.de/e2e/lhc/G2_E2E_index.html
73
LHCOPN Monitoring
•
LHCOPN
– An Optical Private Network
connecting LHC Teir1
centers around the world to
CERN.
– The circuits to two of the
largest Tier1 centers,
FERMI & BNL cross ESnet
•
E2Emon
– An application developed by DFN for monitoring circuits using
perfSONAR protocols
• E2ECU
– End to End Coordination Unit that uses E2Emon to monitor
LHCOPN Circuits
– Run by the GEANT2 NOC
74
E2EMON and perfSONAR
•
E2Emon
– An application suite developed by DFN for monitoring circuits using perfSONAR
protocols
•
perfSONAR is a global collaboration to design, implement and deploy a network
measurement framework.
– Web Services based Framework
•
•
•
•
•
Measurement Archives (MA)
Measurement Points (MP)
Lookup Service (LS)
Topology Service (TS)
Authentication Service (AS)
– Some of the currently Deployed Services
•
•
•
•
•
•
Utilization MA
Circuit Status MA & MP
Latency MA & MP
Bandwidth MA & MP
Looking Glass MP
Topology MA
– This is an Active Collaboration
• The basic framework is complete
• Protocols are being documented
• New Services are being developed and deployed.
75
E2Emon Components
• Central Monitoring Software
– Uses perfSONAR protocols to retrieve current circuit status every minute or so
from MAs and MPs in all the different domains supporting the circuits.
– Provides a web site showing current end-to-end circuit status
– Generates SNMP traps that can be sent to other management systems when
circuits go down
• MA & MP Software
– Manages the perfSONAR communications with the central monitoring
software
– Requires an XML file describing current circuit status as input.
• Domain Specific Component
– Generates the XML input file for the MA or MP
– Multiple development efforts in progress, but no universal solutions
•
•
•
•
•
CERN developed one that interfaces to their abstraction of the Spectrum NMS DB
DANTE developed one that interfaces with the Acatel NMS
ESnet developed one that uses SNMP to directly poll router interfaces
FERMI developed one that uses SNMP to directly poll router interfaces
Others under development
76
E2Emon Central Monitoring Software
http://cnmdev.lrz-muenchen.de/e2e/lhc/G2_E2E_index.html
77
ESnet4 Hub Measurement Hardware
•
Latency
– 1U Server with one of:
• EndRun Praecis CT CDMA Clock
• Meinberg TCR167PCI IRIG Clock
• Symmetricom bc637PCI-U IRIG Clock
•
Bandwidth
– 4U dual Opteron server with one of:
• Myricom 10GE NIC
- 9.9 Gbps UDP streams
- ~6 Gbps TCP streams
- Consumes 100% of 1 CPU
• Chelsio S320 10GE NIC
– Should do 10G TCP & UDP with low CPU Utilization
– Has interesting shaping possibilities
– Still under testing…
78
Network Measurements ESnet is Collecting
•
SNMP Interface Utilization
– Collected every minute
• For MRTG & Monthly Reporting
•
Circuit Availability
– Currently based on SNMP Interface up/down status
– Limited to LHCOPN and Service Trial circuits for now
•
NetFlow Data
– Sampled on our boundaries
•
Latency
– OWAMP
79
ESnet Performance Center
• Web Interface to run Network Measurements
• Available to ESnet sites
• Supported Tests
–
–
–
–
Ping
Traceroute
IPERF
Pathload, Pathrate, Pipechar
• (Only on GE systems)
•
Test Hardware
– GE testers in Qwest hubs
• TCP iperf tests max at ~600 Mbps.
– 10GE testers are being deployed in ESnet4 hubs
• Deployed in locations where we have Cisco 6509 10GE Interfaces
• Available via Performance Center when not being used for other tests
• TCP iperf tests max at 6 Gbps.
80
ESnet Measurement Summary
• Standards / Collaborations
– PerfSONAR
• LHCOPN
– Circuit Status Monitoring
• Monitoring Hardware in ESnet 4 Hubs
– Bandwidth
– Latency
• Measurements
–
–
–
–
–
SNMP Interface Counters
Circuit Availability
Flow Data
One Way Delay
Achievable Bandwidth
• Visualizations
– PerfSONARUI
– VisualPerfSONAR
– NetInfo
81
References
1.
High Performance Network Planning Workshop, August 2002
–
2.
3.
http://www.doecollaboratory.org/meetings/hpnpw
Science Case Studies Update, 2006 (contact [email protected])
DOE Science Networking Roadmap Meeting, June 2003
–
4.
http://www.es.net/hypertext/welcome/pr/Roadmap/index.html
DOE Workshop on Ultra High-Speed Transport Protocols and Network Provisioning for
Large-Scale Science Applications, April 2003
–
5.
http://www.csm.ornl.gov/ghpn/wk2003
Science Case for Large Scale Simulation, June 2003
–
6.
http://www.pnl.gov/scales/
Workshop on the Road Map for the Revitalization of High End Computing, June 2003
–
–
7.
http://www.cra.org/Activities/workshops/nitrd
http://www.sc.doe.gov/ascr/20040510_hecrtf.pdf (public report)
ASCR Strategic Planning Workshop, July 2003
–
8.
http://www.fp-mcs.anl.gov/ascr-july03spw
Planning Workshops-Office of Science Data-Management Strategy, March & May 2004
–
9.
http://www-conf.slac.stanford.edu/dmw2004
For more information contact Chin Guok ([email protected]). Also see
-
http://www.es.net/oscars
ICFA SCIC “Networking for High Energy Physics.” International Committee for Future
Accelerators (ICFA), Standing Committee on Inter-Regional Connectivity (SCIC),
Professor Harvey Newman, Caltech, Chairperson.
-
http://monalisa.caltech.edu:8080/Slides/ICFASCIC2007/
82
Additional Information
83
Example Case Study Summary Matrix: Fusion
• Considers instrument and facility requirements, the process of science drivers
and resulting network requirements cross cut with timelines
Feature
Time
Frame
Anticipated Requirements
Science Instruments
and Facilities
Process of Science
Network
Network Services and
Middleware
Near-term
 Each experiment only gets a few
days per year - high productivity
is critical
 Experiment episodes (“shots”)
generate 2-3 Gbytes every
20 minutes, which has to be
delivered to the remote analysis
sites in two minutes in order to
analyze before next shot
 Highly collaborative experiment
and analysis environment
 Real-time data access and
analysis for experiment steering
(the more that you can analyze
between shots the more effective
you can make the next shot)
 Shared visualization capabilities
5 years
 10 Gbytes generated by
experiment every 20 minutes
(time between shots) to be
delivered in two minutes
 Gbyte subsets of much larger
simulation datasets to be delivered
in two minutes for comparison
with experiment
 Simulation data scattered across
United States
 Transparent security
 Global directory and naming
services needed to anchor all of
the distributed metadata
 Support for “smooth”
collaboration in a high-stress
environment
 Real-time data analysis for
experiment steering combined
with simulation interaction = big
productivity increase
 Real-time visualization and
interaction among collaborators
across United States
 Integrated simulation of the
several distinct regions of the
reactor will produce a much more
realistic model of the fusion
process
 Network bandwidth and data
analysis computing capacity
guarantees (quality of service)
for inter-shot data analysis
 Gbits/sec for 20 seconds out
of 20 minutes, guaranteed
 5 to 10 remote sites involved
for data analysis and
visualization
 Parallel network I/O between simulations,
data archives, experiments, and visualization
 High quality, 7x24 PKI identity
authentication infrastructure
 End-to-end quality of service and quality of
service management
 Secure/authenticated transport to ease access
through firewalls
 Reliable data transfer
 Transient and transparent data replication for
real-time reliability
 Support for human collaboration tools
5+ years
 Simulations generate 100s of
Tbytes
 ITER – Tbyte per shot, PB per
year
 Real-time remote operation of the
experiment
 Comprehensive integrated
simulation
 Quality of service for network
latency and reliability, and for
co-scheduling computing
resources
 Management functions for network quality
of service that provides the request and
access mechanisms for the experiment run
time, periodic traffic noted above.
 PKI certificate authorities that enable strong
authentication of the community members
and the use of Grid security tools and
services.
 Directory services that can be used to
provide the naming root and high-level
(community-wide) indexing of shared,
persistent data that transforms into
community information and knowledge
 Efficient means to sift through large data
repositories to extract meaningful
information from unstructured data.
84
Parallel Data Movers now Predominate
Look at the hosts involved in 2006-01-31–— the plateaus in the
host-host top 100 flows are all parallel transfers (thx. to Eli Dart for this observation)
A132023.N1.Vanderbilt.Edu
lstore1.fnal.gov
5.847
bbr-xfer07.slac.stanford.edu
babar2.fzk.de
2.113
A132021.N1.Vanderbilt.Edu
lstore1.fnal.gov
5.884
bbr-xfer05.slac.stanford.edu
babar.fzk.de
2.254
A132018.N1.Vanderbilt.Edu
lstore1.fnal.gov
6.048
bbr-xfer04.slac.stanford.edu
babar.fzk.de
2.294
A132022.N1.Vanderbilt.Edu
A132021.N1.Vanderbilt.Edu
lstore1.fnal.gov
lstore2.fnal.gov
6.39
6.771
bbr-xfer07.slac.stanford.edu
bbr-xfer04.slac.stanford.edu
babar.fzk.de
babar2.fzk.de
2.337
2.339
A132023.N1.Vanderbilt.Edu
lstore2.fnal.gov
6.825
bbr-xfer05.slac.stanford.edu
babar2.fzk.de
2.357
A132022.N1.Vanderbilt.Edu
lstore2.fnal.gov
6.86
bbr-xfer08.slac.stanford.edu
babar2.fzk.de
2.471
A132018.N1.Vanderbilt.Edu
lstore2.fnal.gov
7.286
A132017.N1.Vanderbilt.Edu
lstore1.fnal.gov
7.62
A132017.N1.Vanderbilt.Edu
lstore2.fnal.gov
9.299
A132023.N1.Vanderbilt.Edu
lstore4.fnal.gov
10.522
bbr-xfer08.slac.stanford.edu
bbr-xfer04.slac.stanford.edu
bbr-xfer05.slac.stanford.edu
bbr-xfer08.slac.stanford.edu
babar.fzk.de
babar3.fzk.de
babar3.fzk.de
babar3.fzk.de
2.627
3.234
3.271
3.276
A132021.N1.Vanderbilt.Edu
lstore4.fnal.gov
10.54
A132018.N1.Vanderbilt.Edu
lstore4.fnal.gov
10.597
A132018.N1.Vanderbilt.Edu
lstore3.fnal.gov
10.746
A132022.N1.Vanderbilt.Edu
lstore4.fnal.gov
11.097
bbr-xfer07.slac.stanford.edu
bbr-xfer05.slac.stanford.edu
bbr-xfer07.slac.stanford.edu
bbr-xfer04.slac.stanford.edu
bbr-xfer08.slac.stanford.edu
babar3.fzk.de
bbr-datamove10.cr.cnaf.infn.it
bbr-datamove10.cr.cnaf.infn.it
bbr-datamove10.cr.cnaf.infn.it
bbr-datamove10.cr.cnaf.infn.it
3.298
2.366
2.519
2.548
2.656
A132022.N1.Vanderbilt.Edu
lstore3.fnal.gov
11.097
A132021.N1.Vanderbilt.Edu
lstore3.fnal.gov
11.213
A132023.N1.Vanderbilt.Edu
lstore3.fnal.gov
11.331
bbr-xfer08.slac.stanford.edu
bbr-xfer05.slac.stanford.edu
bbr-xfer04.slac.stanford.edu
bbr-xfer07.slac.stanford.edu
bbr-datamove09.cr.cnaf.infn.it
bbr-datamove09.cr.cnaf.infn.it
bbr-datamove09.cr.cnaf.infn.it
bbr-datamove09.cr.cnaf.infn.it
3.927
3.94
4.011
4.177
A132017.N1.Vanderbilt.Edu
lstore4.fnal.gov
11.425
bbr-xfer04.slac.stanford.edu
csfmove01.rl.ac.uk
5.952
A132017.N1.Vanderbilt.Edu
babar.fzk.de
babar.fzk.de
lstore3.fnal.gov
bbr-xfer03.slac.stanford.edu
bbr-xfer02.slac.stanford.edu
11.489
2.772
2.901
bbr-xfer04.slac.stanford.edu
move03.gridpp.rl.ac.uk
5.959
babar2.fzk.de
babar.fzk.de
bbr-export01.pd.infn.it
bbr-xfer06.slac.stanford.edu
bbr-xfer04.slac.stanford.edu
bbr-xfer03.slac.stanford.edu
3.018
3.222
11.289
bbr-export02.pd.infn.it
bbr-xfer03.slac.stanford.edu
19.973
bbr-xfer05.slac.stanford.edu
bbr-xfer05.slac.stanford.edu
bbr-xfer07.slac.stanford.edu
bbr-xfer08.slac.stanford.edu
bbr-xfer08.slac.stanford.edu
bbr-xfer07.slac.stanford.edu
csfmove01.rl.ac.uk
move03.gridpp.rl.ac.uk
csfmove01.rl.ac.uk
move03.gridpp.rl.ac.uk
csfmove01.rl.ac.uk
move03.gridpp.rl.ac.uk
5.976
6.12
6.242
6.357
6.48
6.604
85