Transcript Its the Network Interface, Stupid!

It’s the Interface, Stupid!
2002 CAC Panel
Intel
Shubu Mukherjee
VSSAD, Intel Corporation
Disclaimer: All opinions expressed in this presentation
are mine and only mine. Intel or any other corporation
is not liable for any of the material presented in these
slides.
CAC 2002 Panel
Page 1
CPU
Memory
CPU
Memory
Memory Bus
Memory Bus
Network Interface
I/O Bus
?
Network Interface
Intel
MPP Interconnect
Cluster Interconnect
• Interface = Software on CPU + Network Interface
• Interface is key performance bottleneck
CAC 2002 Panel
Page 2
Interconnect not a bottleneck

Interconnect not a bottleneck
MPP & Cluster interconnects have similar properties
Intel


Link bandwidth ~= 10s of gigabits/second
Link latency ~= nano- to micro-seconds
CAC 2002 Panel
Page 3
MPP Interconnect Bandwidth
not a bottleneck for 1-to-1 communication
Peak Link Bandwidth
(megabits/second)
Pentium4 System Bus
Alpha 21364
10000
Pentium System Bus
Cray T3D
1000
100
8085 Bus
10
TMC CM-2
2002
2000
1998
1996
1994
1992
1990
1988
1986
1984
1982
1980
1978
1
1976
Intel
100000
Year of Introduction
Memory Bus
CAC 2002 Panel
MPP Interconnect
Page 4
Cluster Interconnect Bandwidth
not a bottleneck for 1-to-1 communication
Peak Link Bandwidth
(megabits/second)
133 MHz / 64 bit PCI-X
Mellanox
Infiniband
Quadrics
QsNet
10000
32 MHz / 20 bit Sun Sbus
1000
Myricom
Myrinet
100
IBM PC
10
2002
2000
1998
1996
1994
1992
1990
1988
1986
1984
1982
1980
1978
1
1976
Intel
100000
Year of Introduction
I/O Bus
CAC 2002 Panel
Cluster Interconnect
Page 5
Interconnect Latency
not a bottleneck

Sample breakdown
+ Interconnect latency ~= 10s of nanoseconds
+ Network Interface ~= few microseconds
+ Software ~= 10s of microseconds to milliseconds
Intel

Example
+ Ed Felten’s Thesis, 1993
+ On Intel Delta (MPP, but clusters would be similar)
+ Hardware = 1 microsecond
+ Software = 67 microsecond
CAC 2002 Panel
Page 6
Where are the bottlenecks?

Software Interface to interconnect
– operating system intervention
– protocol stacks
• reliable delivery
• congestion control
Intel

Hardware Interface to interconnect
– Extra hop via I/O bus (MPPs usu. don’t have this problem)
– Side-effect prone hardware (e.g., uncached loads)
• network interface hardware designed accordingly
CAC 2002 Panel
Page 7
Winner’s Properties?
The Right Thing To Do

User-level access to network interface
+ Myricom Myrinet or Quadrics QsNet (from Meiko CS2)
+ Infiniband or cLAN’s (with VIA)
Intel

Streamlined Network Interface
+ Integrated I/O bus and Cluster Interconnect
+ Direct Memory Access
+ Treat Network Interface like Cacheable Memory
+ most I/O bridges already do this
- most network interfaces don’t support this yet
CAC 2002 Panel
Page 8
Tug-of-War: inertia vs. performance

Inertia to use existing software
+ Gigabit Ethernet with TCP/IP

Performance from Cluster Interconnects
+ User-level access and streamlined network interface

IDC Forecast, May 2001
(from D.K.Panda’s Infiniband Tutorial)
Intel
% of Infiniband-enabled servers
+ 2003 : 20% of all servers
+ 2004 : 60% of all servers
+ 2005 : 80% of all servers

And, the winner is …. ULTINET (NOT)
+ User-Level Thin Interface NETwork
CAC 2002 Panel
Page 9
Don’t Discount Inertia



Software exists and works for gigabit ethernet &
TCP/IP
Hardware is cheap and widely available
It is a price/performance/inertia issue
– not performance alone
Intel

Infiniband will probably be a temporary I/O
bus/switch/backplane
– 3GIO coming up (backward compatible with PCI)

Mellanox, Quadrics, Myricom
– in a niche market, which can be dangerous because the volume
may not be high enough
– generalizing and adapting other interfaces (e.g., ethernet) may
help their business model
CAC 2002 Panel
Page 10