Transcript SAN

Elements of SAN
capacity planning
Mark Friedman
VP, Storage Technology
[email protected]
(941) 261-8945
DataCore Software Corporation
Founded 1998 - Storage networking Software
170+ employees, private - Over $45M raised
 Top
Venture firms - NEA, OneLiberty
 Funds – VanWagoner, Bank of America, etc
 Intel Business and Technical collaboration agreement
Exec. Team
 Proven
Storage expertise
 Proven Software company experience
 Operating systems, high-availability, Caching, networking
 Enterprise level support and training
Worldwide: Ft. Lauderdale HQ, Silicon Valley, Canada
France, Germany, U.K., Japan
Overview
How do we take what we know about storage
processor performance and apply it to emerging
SAN technology?
What is a SAN?
Planning for SANs:
SAN
performance characteristics
Backup and replication performance
Evolution Of Disk Storage Subsystems
Cached Disk
Spindles
Strings & Farms
See: Dr. Alexandre Brandwajn,
“A study of cached RAID 5
I/O”
CMG Proceedings, 1994.
Storage
Processors
Write-thru
Cached
subsystems
What Is A SAN?
Storage Area Networks are designed to exploit
Fibre Channel plumbing
Approaches to simplified networked storage:
SAN
appliances
SAN Metadata Controllers (“out of band”)
SAN storage managers (“in band”)
The Difference Between NAS and SAN
Storage Area Network (SAN) designed to exploit
Fibre Channel plumbing require a new
infrastructure.
Network Attached Storage (NAS) devices plug
into the existing networking infrastructure.
Networked
file access protocols (NFS, SMB, CIFS)
TCP/IP stack
Application: HTTP, RPC
Host-to-Host: TCP, UDP
Internet Protocol: IP
Media Access: Ethernet, FDDI
Packet
Packet
Packet
Packet
The Difference Between NAS and SAN
NAS devices plug into
existing TCP/IP
networking support.
Performance
considerations:
Application Interfaces
RPC
DCOM
Winsock
NetBIOS
User Mode
Kernel
Named Pipes
Redirector
Server
NetBT
1500
byte Ethernet MTU
TCP requires
acknowledgement of each
packet, limiting
performance.
TDI
TCP
ICMP
UDP
IP
IGMP
IP Filtering
ARP
IP Forwarding
Packet Scheduler
NDIS
NDIS Wrapper
NDIS Miniport
NIC Device Driver
The Difference Between NAS and SAN
Performance
considerations:

e.g.,
1.5 KB Ethernet MTU
 Requires processing
80,000 Host interrupts/sec
@ 1 Gb/sec
 or Jumbo frames, which
also requires installing a
new infrastructure
Which
is why Fibre Channel
was designed the way it is!
Source: Alteon Computers, 1999.
Competing Network File System Protocols
Universal data sharing is developing ad hoc on
top of de facto industry standards designed for
network access.
Sun
NFS
HTTP, FTP
Microsoft CIFS (and DFS)


also known as SMB
CIFS-compatible is the the largest and fastest growing
category of data
CIFS Data Flow
Session-oriented: e.g., call backs
Client
MS Word
System
Cache
File
Server
Server
Redirector
Network
Interface
SMB Request
Network
Interface
SMB Request
System
Cache
What About Performance?
NFS Server
NFS Client
User Process
Remote
Procedure
Call (RPC)
Client Process
NFSD Daemon
Application: HTTP, RPC
Host-to-Host: TCP, UDP
TCP/IP Driver
Response
Data
TCP/IP Driver
Internet Protocol: IP
Media Access: Ethernet, FDDI
TCP/IP Network
What About Performance?
 Network-attached yields fraction of the performance of directattached drives when block size does not match frame size.
Client
MS Word
System
Cache
File
Server
Server
Redirector
Application: HTTP, RPC
Host-to-Host: TCP, UDP
Network
Interface
Network
Interface
Internet Protocol: IP
Media Access: Ethernet, FDDI
SMB Request
See ftp://ftp.research.microsoft.com/pub/tr/tr-2000-55.pdf
SMB Request
System
Cache
What about modeling?
Add a network delay
component to interconnect
two Central Server models
and iterate.
The Holy Grail!
Storage Area Networks
Uses low latency, high performance Fibre
Channel switching technology (plumbing)
100 MB/sec Full duplex serial protocol
over copper or fiber
Extended distance using fiber
Three topologies:
Point-to-Point
Arbitrated
Loop: 127 addresses, but can be
bridged
Fabric: 16 MB addresses
The Holy Grail!
Storage Area Networks
FC delivers SCSI commands, but Fibre
Channel exploitation requires new
infrastructure and driver support
Objectives:
Extended
addressing of shared storage pools
Dynamic, hot-plugable interfaces
Redundancy, replication & failover
Security administration
Storage resource virtualization
Distributed Storage & Centralized
Administration
Traditional tethered vs untethered SAN
storage
Untethered storage can (hopefully) be pooled for
centralized administration
Disk space pooling (virtualization)
Currently,
using LUN virtualization
In the future, implementing dynamic virtual:real address
mapping (e.g., the IBM Storage Tank)
Centralized back-up
SAN
LAN-free backup
Storage Area Networks
FC is packet-oriented (designed for routing).
FC pushes many networking functions into the
hardware layer.
 e.g.,
 Packet
fragmentation
 Routing
Upper Level Protocol
SCSI
IPI-3
HIPPI
IP
Fc4
Common Services
Fc3
Framing Protocol/Flow Control
Fc2
8B/10B Encode/Decode
Fc1
100MB/s Physical Layer
Fc0
Storage Area Networks
FC is designed to work with optical fiber and
lasers consistent with Gigabit Ethernet hardware
100
MB/sec interfaces
200 MB/sec interfaces
This creates a new class of hardware that you
must budget for: FC hubs and switches.
Storage Area Networks
Performance characteristics of FC switches:
Extremely
low latency ( 1sec), except when cascaded
switches require frame routing
Deliver dedicated 100 MB/sec point-to-point virtual circuit
bandwidth
Measured 80 MB/sec effective data transfer rates per
100 MB/sec Port
Storage Area Networks
When will IP and SCSI co-exist on
the same network fabric?
iSCSI
Nishan
Others?
Upper Level Protocol
SCSI
IPI-3
HIPPI
IP
Fc4
Common Services
Fc3
Framing Protocol/Flow Control
Fc2
8B/10B Encode/Decode
Fc1
100MB/s Physical Layer
Fc0
Storage Area Networks
FC zoning is used to control
access to resources (security)
Two approaches to SAN
management:
Management
functions must
migrate to the switch, storage
processor, or….
OS must be extended to support
FC topologies.
Approaches to building SANs
Fibre Channel-based Storage Area Networks
(SANs)
SAN
appliances
SAN Metadata Controllers
SAN Storage Managers
Architecture (and performance) considerations
Approaches to building SANs
Where does the logical device:physical device
mapping run?
Out-of-band:
on the client
In-band: inside the SAN appliance, transparent to the
client
Many industry analysts have focused on this
relatively unimportant distinction.
SAN appliances
Conventional storage processors with
Fibre Channel interfaces
Fibre Channel support
FC
Fabric
Zoning
LUN virtualization
SAN Appliance Performance
Same as before, except faster
Fibre Channel interfaces
Multiple Processors
Commodity
FC Interfaces
Cache Memory
FC Disks
Internal Bus
Host Interfaces
processors, internal
buses, disks, front-end and
back-end interfaces
Proprietary storage processor
architecture considerations
SAN appliances
SAN and NAS convergence?
Adding
Fibre Channel interfaces and Fibre Channel
support to a NAS box
SAN-NAS hybrids when SAN appliances are connected
via TCP/IP.
Current Issues:
Managing
multiple boxes
Proprietary management platforms
SAN Metadata Controller
SAN clients acquire
an access token from
the Metadata
Controller (out-ofband)
SAN Clients
1
Token
2
SAN clients then
access disks directly
using proprietary
distributed file system
3
Fibre Channel
SAN
Metadata
Controller
Pooled Storage Resources
SAN Metadata Controller
Performance considerations:
MDC
latency (low access rate assumed)
Additional latency to map client file system request to the
distributed file system
Other administrative considerations:
Requirement
for client-side software is a burden!
SAN Storage Manager
Requires all access to
pooled disks through the
SAN Storage Manager
(in-band)!
SAN Clients
Fibre Channel
Storage
Domain
Servers
Pooled Storage Resources
SAN Storage Manager
SAN Storage Manager
adds latency to every I/O
request
How much latency is
involved?
SAN Clients
Fibre Channel
Can this latency be
reduced using traditional
disk caching strategies?
Storag
e
Domai
n
Server
s
Pooled Storage Resources
Architecture of a Storage Domain Server
Runs on an ordinary
Win2K Intel server
The SDS intercepts
SAN I/O requests,
impersonating a SCSI
disk
Leverages:
 Native
Device drivers
 Disk management
 Security
 Native CIFS support
Client I/O
SANsymphony Storage Domain Server
Initiator/Target Emulation
FC Adaptor Polling Threads
Security
Fault Tolerance
Data Cache
Natives W2K I/O Manager
Disk Driver
Diskperf (measurement)
Fault Tolerance (Optional)
SCSI miniport Driver
Fibre Channel HBA Driver
Sizing the SAN Storage Manager server
In-band latency is a function of Intel
server front-end bandwidth:
Processor
speed
Number of processors
PCI bus bandwidth
Number of HBAs
and performance of the back-end Disk
configuration
SAN Storage Manager
Can SAN Storage Manager in-band latency be
reduced using traditional disk caching strategies?
Read
hits
Read misses
 Disk I/O + (2 * data transfer)
Fast
Writes to cache (with mirrored caches)
 2 * data transfer
 Write performance ultimately determined by the disk
configuration
SAN Storage Manager
Read hits (16 KB block):
Timings from an FC hardware monitor
1Gbit/s Interfaces
SCSI Read
Command
Length = 4000
Status
Frame
16x1024 Byte Data Frames
140sec
No bus arbitration delays!
27sec
Read vs. Write hits (16 KB block)
Fibre Channel Latency (16KB Blocks)
SCSI Command
Write Setup
Data Frames
SCSI Status
Decomposing SAN in-band Latency
How is time being spent inside the server?
PCI bus?
Host Bus adaptor?
Device polling?
Software stack?
SCSI Command Write Setup
Data Frames
SCSI Status
Benchmark Configuration
3 PCI buses
(528MB/s Total)
1, 4, or 8
QLogic 2200 HBAs
Memory Bus
32bit/33MHz PCI
of three
FC interface polling threads
32bit/33MHz PCI
Maximum
4x550MHz
XEON
Processors
64bit/33MHz PCI
4-way 550 MHz PC
Decomposing SAN in-band Latency
How is time being spent inside the SDS?
PCI bus?
Host Bus adaptor?
Device polling:
1
CPU is capable of 375,000 unproductive polls/sec
 2.66secs per poll
Software stack:
3
CPUs are capable of fielding 40,000 Read I/Os per
second from cache
 73secs per 512-byte I/O
Decomposing SAN in-band Latency
SANsymphony in-band Latency (16KB Blocks)
SDS
FC Interface
Data Transfer
Impact Of New Technologies
Front-end bandwidth:
Different
speed Processors
Different number of processors
Faster PCI Bus
Faster HBAs
e.g. Next Generation Server
2GHz
GHz Processors (4x Benchmark System)
200MB/sec FC interfaces (2x Benchmark System)
4x800MB/s PCI bus (6x Benchmark System)
...
Impact Of New Technologies
2GHz CPU, New HBAs,
2Gbit Switching
2GHz CPU & New
HBAs
Today
Sizing the SAN Storage Manager
Scalability
Processor
speed
Number of processors
PCI bus bandwidth




32bit/33MHz 132MB/sec
64bit/33MHz 267MB/sec
64bit/66MHz 528MB/sec
64bit/100MHz 800MB/s (PCI-X)
Infiniband
technology???
Number of HBAs
 200 MB/sec FC interfaces feature faster internal processors
Sizing the SAN Storage Manager
Entry level system:
Dual
Processor, single PCI bus, 1 GB RAM
Mid-level departmental system:
Dual
Processor, dual PCI bus, 2 GB RAM
Enterprise-class system:
Quad
Processor, triple PCI bus, 4 GB RAM
SAN Storage Manager PC scalability
50 ,0 0 0
M ax Read I/ Os
M ax W r it e I/ Os
M ax Read Thr ou ghp u t
M ax W r it e Thr ou ghp u t
30 0
M ax I Os p er seco n d
( @ 512 b y t es)
20 0
30 ,0 0 0
150
20 ,0 0 0
10 0
10 ,0 0 0
50
0
0
0
2
4
6
FC H BA s
8
10
M ax M B/ sec ( @ 16 K B)
250
4 0 ,0 0 0
M ax Read I/ Os
M ax W r it e I/ Os
50 ,0 0 0
M ax Read Thr ou ghp u t
M ax W r it e Thr ou ghp u t
4 0 ,0 0 0
Departmental SAN
30 0
250
20 0
Entry level
30 ,0 0 0
Enterprise class
150
20 ,0 0 0
10 0
10 ,0 0 0
50
0
0
0
2
4
6
FC H BA s
8
10
M ax M B/ sec ( @ 16 K B)
M ax I Os p er seco n d
( @ 512 b y t es)
SAN Storage Manager PC scalability
SANsymphony Performance
Conclusions
FC
switches provide virtually unlimited bandwidth with
exceptionally low latency so long as you do not cascade
switches
General purpose Intel PCs are a great source of
inexpensive MIPS.
In-band SAN management is not a CPU-bound process.
PCI bandwidth is the most significant bottleneck in the
Intel architecture.
FC Interface cards speeds and feeds are also very
significant
SAN Storage Manager – Next Steps
Cacheability of Unix and NT workloads
Domino,
MS Exchange
Oracle, SQL Server, Apache, IIS
Given mirrored writes, what is the effect of
different physical disk configurations?
JBOD
RAID
0 disk striping
RAID 5 write penalty
Asynchronous disk mirroring over long distances
Backup and Replication (snapshot)
Questions
www.datacore.com