Transcript 20020319-HENP-McKee

The Internet2 HENP Working Group
Internet2 Virtual Briefing
March 19, 2002
Shawn McKee
University of Michigan
HENP Co-chair
Overview
• Mission, Membership and
Motivation
• HENP WG Goals
• Recent activities
• Where to from here?
Slide 2
March 19, 2002 April
2002
Internet2 HENP WG*
• Mission
• To help ensure that the required national and
international network infrastructures, monitoring
tools and facilities, and support for collaborative
systems, are deployed and developed on an ongoing
basis, in time to meet the requirements of the major
HENP experimental programs, as well as the HENP
community at-large .
• To encourage that the Group's targeted developments
are applied broadly, in other fields, within and beyond
the bounds of scientific research.
• The goals of this Working Group are synergistic with the Internet2
End-to-End Initiative, which has HENP as one of its focal disciplines.
Slide 3
Formed as an Internet2 WG on Oct. 26 2001
* Co-Chairs: S. McKee (Michigan), H. Newman (Caltech);
Secretary J. Williams (Indiana); With thanks to R. Gardner (Indiana)
March 19, 2002 April
2002
Current Membership
• We have 88 members signed up on our
mailing list at
http://lists.bnl.gov/mailman/listinfo/henp-net-l
• Our membership is a mix of scientists,
engineers, local and backbone network
experts and funding agency members
• We are looking to expand our membership,
especially internationally…
Slide 4
March 19, 2002 April
2002
Why HENP Networking?
• Since the early 1980’s physicists have depended
upon leading-edge networks to enable ever larger
international collaborations.
• Major HENP collaborations require rapid access to
event samples from massive data stores, not all of
which can be locally stored at each computational
site.
• Evolving integrated applications, i.e. Data Grids,
rely on seamless, transparent operation of the
underlying LANs and WANs.
• Networks are among the most basic Grid building
blocks.
Slide 5
March 19, 2002 April
2002
Hierarchical Computing Model
CERN/Outside Resource Ratio ~1:2
Tier0/( Tier1)/( Tier2)
~1:1:1
~PByte/sec
Online System
Tier 1
~2.5 Gbits/sec
France
Italy
UK
~2.5 Gbps
Institute Institute
~0.25TIPS
Physics data cache
Workstations
Slide 6
Offline Farm,
CERN Computer Ctr
~25 TIPS
Tier 0 +1
Tier 2
Tier 3
~100 MBytes/sec
Institute
Institute
100 - 1000
Mbits/sec
Tier 4
BNL Center
Tier2 Center
Tier2 Center
Tier2 Center
Tier2 Center
Tier2 Center
Physicists work on analysis “channels”
Each institute has ~10 physicists
working on one or more channels
Yearly data volume is ~10 Petabytes
March 19, 2002 April
2002
HENP WG Goals
• One of our first task upon forming our working
group was to define a set of goals
• These goals needed to address the range of
interests and concerns of our community
• We arrived at 9 goals, which are now part of our
charter.
• These are are the best definition of what our
working group is about…
Slide 7
March 19, 2002 April
2002
HENP WG Goal #1
1.
Support the development and deployment of toolkits,
documentation and guidelines for best practices so
that existing expert knowledge and tools for high
throughput data transfers, packet loss and
throughput-limit monitoring, and collaborative systems
are widely known.
Contact: Tom Hacker/Shawn McKee (U Mich)
Slide 8
March 19, 2002 April
2002
HENP WG Goal #2
2.
Support the deployment of testing and monitoring
tools and applications, link and site instrumentation,
and a standard methodology, in association with the
Internet2 End-to-End Initiative, so that all of HENP's
major network paths can be adequately monitored,
and used at full capability.
Contact: Iosif Legrand()/Les Cottrell (SLAC)*
Slide 9
March 19, 2002 April
2002
HENP WG Goal #3
3. Share information and provide advice on the
configuration of routers, switches, PCs and
network interfaces, and network testing and
problem resolution, to achieve high
performance over local and wide area
networks in production.
Contact: Sylvain Ravot(CERN)/<To be named>
Slide 10
March 19, 2002 April
2002
HENP WG Goal #4
4.
Work with the staffs of the international (Tier0),
national (Tier1), regional (Tier2) and local (Tier3) Grid
facilities as needed, to verify high throughput and
network responsiveness, in association with the Grid
projects, and for new collaborative interactive
systems.
Contact: Les Cottrell* (SLAC)/<To be named>
Slide 11
March 19, 2002 April
2002
HENP WG Goal #5
5. Work with the sites, expert networking teams
at national labs, universities, and
computational science centers, to develop and
expand on a knowledge base for high
performance networks. Develop a support
base of knowledgeable engineers, technicians
and scientists to assist with the above goals.
Contact: Rob Gardner*/Jim Williams (IndianaU)
Slide 12
March 19, 2002 April
2002
HENP WG Goal #6
6.
Work to identify and track the programmatic network
needs of the current and future generation HENP
program: quantitatively in terms of bandwidths,
throughputs and computer system capabilities
required to provide the needed overall performance;
and qualitatively in terms of the features and
characteristics of existing and new systems needed
for efficient distributed data access, processing and
analysis as well as remote collaboration.
Contact: Harvey Newman(CalTech)/<To be named>
Slide 13
March 19, 2002 April
2002
HENP WG Goal #7
7. Work with GriPhyN, PPDG, EU DataGrid,
iVDGL and other Grid projects to ensure that
the US and global network infrastructures are
satisfying the programmatic needs of scientists
using grids.
Contact: Dantong Yu(BNL)/Iosif Legrand()*
Slide 14
March 19, 2002 April
2002
HENP WG Goal #8
8. Work with the network engineering staffs of the
US and global research networks to help
define the requirements and operational
procedures for a Global Grid Operations
Center, and mission-specific Grid Operations
Centers as needed, so that the ensemble of
research networks is able to work efficiently
and provide the high level of capability
required.
Contact: Jim Williams/Rob Gardner* (IndianaU)
Slide 15
March 19, 2002 April
2002
HENP WG Goal #9
9. Investigate emerging new network
technologies, such as optical switching and
lambda-based network infrastructure, for
applicability and potential use in support of the
HENP program. If appropriate, develop a
strategic plan for coordinated deployment of
these technologies among HENP sites.
Contact: Phil Demar(FNAL)/Bill St. Arnaud(CANARIE)
Slide 16
March 19, 2002 April
2002
About These Goals…
• As you can see from our goals we have many network
related issues to pursue.
• These goals are generally not specific to HENP.
• We are striving to work with many related groups on
these issues: HENP-WG; ICFA-SCIC; I2 Engineering in
the US; the GGF Networking WG, etc.
• NOTE: Grids implicitly assume high, repeatable network
performance…
=> Both grid and networking groups MUST work together
to insure an optimal outcome!
Slide 17
March 19, 2002 April
2002
Recent HENP Talks
• QoS Implementation and
Testing
• Report from the PPDG
High Performance
Monitoring Group
Throughput
• Network-related Issues in a
Present and Future
Grid Environment
Networks: an HENP
• Remarks on HENP and the
Perspective
I2 End-to-end Performance
Performance of Parallel
Initiative
TCP Streams in a
• Numerous other
Production Network
discussions, perspectives,
reports and overviews
Grid Monitoring
• Maximizing End-to-end
Network Performance
•
•
•
•
Slide 18
March 19, 2002 April
2002
Some HENP Issues…
• The HENP WG has focused on high
performance because of the unprecedented data
volume we are anticipating.
• Some examples of recent interest:
• Local infrastructure
• Protocol issues for high performance
• Issues in achieving high performance
networking
Slide 19
March 19, 2002 April
2002
Local Networking Infrastructure
• LANs used to lead WANs in performance,
capabilities and stability, but this is no longer true.
• WANs are deploying 10 Gigabit technology
compared with 1 Gigabit on leading edge LANs.
• New protocols and services are appearing on
backbones (Diffserv, IPV6, multicast) (ESNet, I2).
• Insuring our HENP institutions have the required
LOCAL level of networking infrastructure to
effectively participate in the evolving hierarchical
grid computing model is a major challenge.
Slide 20
March 19, 2002 April
2002
TCP WAN Performance
Mathis, et. al., Computer
Communications Review v27,
3, July 1997, demonstrated
the dependence of bandwidth
on network parameters:
0.7  MSS
BW 
RTT PkLoss
BW - Bandwidth
MSS – Max. Segment Size
RTT – Round Trip Time
PkLoss – Packet loss rate
If you want to get 90 Mbps via TCP/IP on a
WAN link from LBL to UM you need a packet
loss < 1.8e-6 !! (~70 ms RTT).
Slide 21
This gets MUCH (~103) worse if you try to
achieve ~1 gigabit across the Atlantic Ocean!
March 19, 2002 April
2002
Testing at UM
• We wanted to test what kinds of performance
increase we could achieve across campus using
a gigabit connection compared to using the
campus network (100 Mbps bottleneck)
• One machine was setup in Physics and one at
Arbor Lakes machine room
• Iperf testing was conducted using both machines
alternately as client/server with 2 different MTU
sizes
• There are many issues which influence network
performance…
Slide 22
March 19, 2002 April
2002
Iperf Network Test Setup
Slide 23
March 19, 2002 April
2002
UM Network IPERF Results
Server(MTU)
Client(MTU)
UDP(Mbps) TCP(Mbps)
ATGRID(1500)
SPEEDY(1500)
737
469
ATGRID(9000)
SPEEDY(9000)
710
977
ATGRID(1500)
SPEEDY(1500)
92.5
64.2
SPEEDY(1500)
ATGRID(1500)
922
618
SPEEDY(9000)
ATGRID(9000)
784
990
SPEEDY(1500)
ATGRID(1500)
55.3
50.3
A gigabit path has enabled us to increase our bandwidth to
the edge of campus by a factor of 7-15 (Gig vs Campus)
Slide 24
March 19, 2002 April
2002
What did we tune?
• The best results we achieved on the previous table were
not from default settings…we had to tune some network
related parameters:
• Gigabit cards were located in 64 bit/66Mhz PCI slots
• MTU size was increased to 9000 bytes
• Linux TCP stack parameters were adjusted:
• echo 1048576 > /proc/sys/net/core/rmem_max
• echo 1048576 > /proc/sys/net/core/wmem_max
• On a Linux 2.4+ kernel add the following
•
echo "4096 87380 4194304" > /proc/sys/net/ipv4/tcp_rmem
•
echo "4096 65536 4194304" > /proc/sys/net/ipv4/tcp_wmem
Slide 25
March 19, 2002 April
2002
Achieving High Performance
Networking
• Server and Client CPU, I/O and NIC throughput sufficient
• Must consider firmware, hard disk interfaces, bus type/capacity
• Knowledge base of hardware: performance, tuning issues, examples
• TCP/IP stack configuration and tuning is Absolutely Required
• Large windows, multiple streams?
• No Local infrastructure bottlenecks
• Gigabit Ethernet “clear path” between selected host pairs
• To 10 Gbps Ethernet by ~2003
• Careful Router/Switch configuration and monitoring
• Enough router “Horsepower” (CPUs, Buffer Size, Backplane BW)
• Packet Loss must be ~Zero (well below 0.1%)
• i.e. No “Commodity” networks (need I2, ESNet type networks)
• End-to-end monitoring and tracking of performance
Slide 26
March 19, 2002 April
2002
Where Are We Headed?
• We have had two official HENP WG meetings
and our third is co-scheduled with the Internet2
Spring Meeting, May 6-8.
• Our goals have just been populated with one or
two contacts each.
• By our next meeting we hope to have compiled
goal related information on our WWW page:
http://www.internet2.edu/henp
• And longer term…(next slide)
Slide 27
March 19, 2002 April
2002
Some Desired HENP WG
Outcomes…
• Define local hardware, software and configuration
details necessary to achieve ~gigabit TCP connections
to the desktop
• Enable high-quality, interactive video-conferencing on
demand over the WAN
• Enable multi-gigabit site-to-site interconnects with
related network services to facilitate the tiered grid
computing model
• Provide a set of worldwide, transparent, highperforming networks, which allow interaction with
users and applications that have special requirements
Slide 28
March 19, 2002 April
2002
Thanks for your attention!
• Questions?
• See our WWW page for more information:
http://www.internet2.edu/henp
Slide 29
March 19, 2002 April
2002