Transcript Sonoma_2008_Mon_Sonoma_OFA_grun_040808_publish

Paul Grun
System Fabric Works
[email protected]
4/7/08
Channel I/O – an embarrassment of riches
Winning in the low latency space, marking
time in the commercial sector
Never bet against Ethernet
Moving our customers forward
Conclusion
2
It all started with VIA
VIA gave us the concept of stack bypass
This gave us the makings of a low latency
interconnect.
Perfectly suited for, among other things, clusters.
3
From VIA sprang InfiniBand
(after a few false starts)
•
•
•
•
•
•
•
An efficient transport
Network and phy layers
Virtual lanes
A mgmt infrastructure
Std methods for accessing
these services
…
In short, the ability to efficiently conduct multiple traffic flows over a
single wire
Interesting…the basics for a unified fabric
4
Equally interesting is this…
Beyond a low latency
unified fabric, we also
gained the ability to
directly connect virtual
address spaces located
in disjoint physical
addresses - RDMA
5
Tailor-made for virtualization
Server virtualization- Platform
resources are shared among the
‘containers’ within the platform
Datacenter virtualization applications connected to pools of
virtualized resources
V app
VM
M
M
…
app
VM
App App App
VM
VM
VM
VM
Better yet…more efficient use of resources improves the ‘green
footprint’…dramatically
6
High bandwidth, no packet dropping…all we would need
would be a storage protocol and we’d have the perfect
fabric.
Enter SRP, followed a little later by iSER.
Voila, IB is an ideal storage interconnect!
7
Eventually, we arrived at the crux of the issue.
What really matters is the way an application
communicates with other applications and storage.
This is the notion of Channel I/O.
OFA emerged as the keeper and developer of the
Channel Interface
8
Channel I/O is about creating pipes
Application
Application
transport
transport
network
network
switch
switch
phy
phy
Channel i/f
It’s very much an application-centric
view…what matters is how the
application communicates.
9
As the industry began to focus on Channel I/O, the
underlying wire seemed to become progressively less
important.
Someone noticed that it made sense to define RDMA
over TCP/IP/Ethernet…iWARP emerged and became a
member of the family of Channel I/O protocols. Now
we’re really cooking with gas.
10
This is looking pretty good!
So far, we’ve got:
 The world’s lowest latency standards-based interconnect
 An incredible foundation for virtualization
 both server and data center virtualization




a native storage solution based on standard SCSI classes
wire independence
a unified fabric
An open Channel Interface for both Linux and Windows
platforms
 It works for legacy apps, and for those plucky native apps
 And it’s cheap to boot! (no pun intended)
I mean, is this nirvana, or what???
11
With this embarrassment of riches, we can address
some of the world’s most vexing computer science
problems




The world’s fastest/cheapest supercomputers
The world’s lowest price/perf clusters
The world’s most flexible/agile datacenters
Highly energy efficient data centers
The world is our oyster, right?
12
So how come we’re not all rich?
13
The box score so far looks like:
IB
Supercomputers

HPC, clusters

HPC-like portions of the
commercial space

EVERYTHING Else
Ethernet

Naturally most of the gold is in that last bucket
14
 It’s interesting that both supercomputing and
HPC/clustering are primarily ‘single fabric’
environments…
 With a few exceptions, these installations are
purpose-built from the ground up, using a single
fabric
But what about the commercial spaces?
15
Commercial datacenters, OTOH
 …tend to be built on top of a combination of fabrics,
 …tend to include huge application investments,
 …tend to involve huge amounts of existing infrastructure
 …tend to rely on high volume, commodity OTS hardware
These are some pretty large rocks to roll uphill
16
(Of course, the commercial space isn’t monolithic…clearly,
there are environments where the calculus of channel I/O
produces a positive ROI.)
17
What about virtualization? Isn’t that a key driver for the
commercial space?
Server virtualization, which doesn’t depend on channel I/O, is
doing very nicely at driving up utilization, and helping reduce the
physical space requirements.
Great progress is being made here, with or without channel I/O.
18
How about the green datacenter?
There is some promise here, but compared to the cost
of migrating a massive investment to a greener
approach, the pain will have to get *alot* higher before
channel I/O grabs a foothold*
*yes, there are certainly spots where the pain threshold is indeed a lot higher.
19
Unified fabrics? Doesn’t OpEx/TCO conquer all?
Well…
20
“Nobody ever lost a bet on Ethernet”
Ethernet continues to chug along…
The Ethernet community is thinking about a lossless wire
 Congestion management
reducing the burden on the transport, thus reducing the traditional
achilles heel of TCP/IP
 Virtual lanes to support multiple streams on a single
fabric
This is starting to sound like a credible converged
fabric…
21
… a converged fabric which is aimed squarely at the
commercial space…i.e., those environments which
can easily harvest the TCO benefits of a converged
fabric.
22
Application
transport
It’s a completely pragmatic, ‘wire-centric’
approach…intended to make the wire sufficiently
capable such that it will support multiple streams of
traffic.
A single-minded focus on creating a unified wire.
network
switch
phy
23
Nobody believes that Ethernet will ever go away.
So we have a few choices.
1. Cede the converged fabric, commercial multi-protocol
space to Ethernet’s emerging enhancements.
2. Battle it out for ownership of the converged fabric.
And probably lose.
3. Look for niche enclaves in the enterprise where IB
dominates - the ‘toehold’ strategy.
4. Drive channel I/O in above the wire.
How?
24
1. Don’t depend on the emergence of a ‘pure IB’ environment,
it won’t happen (except in HPC). Instead…
Leverage the heavy lifting Ethernet is proposing to do by
reducing the impedance mismatch between networks
wherever possible. How about an IB transport on Ethernet?
LRH
GRH
BTH
MAC
IPv6
TCP
ETH
payload
payload
(not as simple as that of course, but maybe worth a look…)
25
2. Enable IB in every toehold that exists in the data center
Make sure an application can
use the channel I/O interface
to access system resources,
regardless of the protocol.
FCoIB, anyone?
filesystem
SCSI
FC-4
transport
switch
phy
26
3. Reduce impediments to accessing the features of the IB
wire
IB already has the features now being proposed by the
Ethernet community, in spades. Let’s reduce the
impediments to accessing those features.
Is there any good reason for continuing to support different
bit rates/encoding schemes?
27
Apply industry leadership to make sure our customers
aren’t swamped in a quagmire of I/O protocols, wire
protocols, wires.
Remember that a key attraction of a converged
fabric is its simplicity.
28
We need to ensure that the promise of channel
I/O can be delivered…no matter what the wire.
We need to lower the pain threshold of adopting
RDMA-based networks (aka channel I/O).
29
Conclusions
 Channel I/O is a simple and powerful concept
 The trend toward convergence in the ‘multi-fabric’ space
is accelerating; Ethernet is quietly but aggressively
attacking this space.
 Channel I/O is in danger of being relegated to the
supercomputer and HPC niches.
 Expanding RDMA networks into the ‘multi-fabric’ space
requires that we keep moving channel I/O forward
30
A paid advertisement…
System Fabric Works is dedicated to ensuring that its customers
derive maximum competitive advantage from their information
technology investment.
Our focus is on accelerating adoption of advanced data center
networks (“Fabric Computing”) as a foundation technology.
We provide:
1) strategic and architectural consulting services directly to end user
clients,
2) design and software engineering services to our vendor and
integrator partners,
3) custom integrated computing, networking and storage products.
31
Abstract:
•Channel I/O delivers an embarrassment of riches in terms of the features it provides.
•Despite that, significant inroads are only being made primarily in the supercomputer,
scientific and commercial HPC spaces…spaces characterized as being purpose-built on top
of a single primary fabric.
•The commercial space is characterized as being built on top of a combination of fabrics (as
opposed to the single-fabric character of the HPC space).
•This space, where most of the gold is, remains more-or-less elusive.
•There are a few emerging exceptions.
•Why is this?
•The values delivered by channel I/O do not yet out-weigh the costs of transitioning.
•Meanwhile, Ethernet is quietly addressing those markets by providing a path toward a
converged fabric.
•Call to action:
•Lower the hurdles preventing end users from harvesting the values of channel I/O
•Provide leadership in helping select customers through the “I/O quagmire”
32