Transcript Is SAN in the WAN possible

IP-Based Storage
Networking
陳文賢
中興大學資訊科學研究所
Outline

Introduction
iSCSI: Architecture and Standardization

Issues and Solutions of IP Storage







Performance
Security
Cost
Interoperability
Storage over WAN
Conclusion
Introduction





SAN: Storage Area Network
NAS: Network Attached Storage
PCI Bus speed
Gigabit and 10Gigabit Ethernet
Storage is no fun until networking
comes in.
Introduction




Standardization: The Internet Engineering
Task Force (IETF) has approved the iSCSI
standard since Feb. 2003.
A mapping of the SCSI remote procedure
invocation model on top of the TCP protocol.
A new “SCSI transport” as defined by the
SCSI SAM-2 document.
 Equivalent protocols include “SPI-2”,
“FCP-2”, …
To take compelling advantages from the
IP/Ethernet infrastructure.
SCSI-3 Architecture Roadmap
Common Access Method (CAM)
Architecture Model (SAM, SAM)
2
Reduced
Medium
Multi-Media Controller
Block
Stream
Block
Changer
Commands Commands
Commands
Commands
Commands
Commands (MMC, MMC-2,
(SBC, SBC-2)
(SSC, SSC-2)
(SCC, SCC-2)
(SMC, SMC-2) MMC-3, RMC)
(RBC)
Enclosure
Services
(SES)
Object-Based
Storage
Device
(OSD)
Primary Commands (SPC, SPC-2, SPC-3)
SCSI
Parallel
Interface
(SPI-2, SPI-3,
SPI-4)
(aka, Ultra2,
Ultra160,
Ultra320)
Serial Bus
Protocol – 2
(SBP-2)
Fibre
Channel
Protocol
(FCP, FCP-2)
SSA
SCSI-3
Protocol
(SSA-S3P)
SCSI
over ST
(SST)
SCSI
VI Protocol
(SVP)
iSCSI/TCP/IP
Scheduled
Transfer
(ST)
Virtual
Interface
(VI)
802.2/802.3
……
SSA-TL2
IEEE 1394
Fibre
Channel
(FC)
SSA-PH1 or
SSA-PH2
Layers and Sessions
Initiator
Target
Session
Session
SCSI layer (SAM-2)
SCSI CDB
SCSI layer (SAM-2)
iSCSI layer
iSCSI layer
iSCSI
layer
iSCSI
layer
iSCSI
layer
TCP/IP
TCP/IP
TCP/IP
iSCSI PDUs
TCP
packets
TCP Connection
TCP Connection
TCP Connection
iSCSI
layer
iSCSI
layer
iSCSI
layer
TCP/IP
TCP/IP
TCP/IP
Layer and Session (Cont.)

Conceptual Layering Model




SCSI layer – builds/receives SCSI
Command Data Blocks (cf. SCSI
Architecture Model - 2)
iSCSI layer – builds/receives iSCSI PUDs
TCP Connections – form an initiator-target
“session”
Session


A group of TCP connections linking an
initiator with a target.
Defined by a session ID
What customer problems does iSCSI
solve?

iSCSI provides a cost-effective transport for Storage
Area Network (SAN) when compared with Fibre
Channel.

iSCSI enables affordable storage consolidation
solutions—particularly in environments populated
with mid range servers.

Together with Storage Management Solution, iSCSI
also provides affordable disaster recovery, backup,
and secondary storage solutions.
Performance and Cost: Hardware versus
software-based solutions

Software iSCSI initiators provide the lowest cost
iSCSI solution. A software-only iSCSI initiator uses a
standard Ethernet NIC or a NIC with TCP offload
Engine (TOE) to process the iSCSI commands and
the TCP/IP protocol. For workstations/servers with 2
GHz CPUs, iSCSI protocol processing does not
impose a significant overhead for most customer
workloads. Example: Microsoft iSCSI initiator driver.

Hardware: With older CPUs and heavily loaded
servers, a hardware-assisted iSCSI initiator is
appropriate as the iSCSI initiator HBA can offload
the CPU. Example: Intel iSCSI HBA (Intel Pro 1000T)
Performance and CPU overhead (For
single Gbps connection)
iSCSI
HBA
iSCSI
driver
SAN/IP
Sequential Read
94MB/s
59MB/s
105MB/s
CPU
Utilization(Client)
17%
23%
35%
Note: With multiple connections, the performance can
be enhanced even more.
Testing Configuration
Scenarios
IPStor Server
GBE Switch
FalconStor’s IPStor Server (iSCSI target)
Win 2000 Server
Intel IOMeter
running on
Win2k Server
1. Intel iSCSI
HBA (Pro
1000 T)
2. Microsoft
iSCSI
initiator
(software)
3. SAN/IP
Client
iSCSI Security



Fibre Channel is perceived to be more
secure as it is a private network. However, it
is a Layer 2 protocol with no security
mechanism built in essentially.
The iSCSI spec, on the other hand, covers
initiator and target authentication (using
CHAP, SRP, Kerberos, and SPKM) to prevent
unauthorized access and permit only
trustworthy nodes. In addition, IPsec can
be used to provide privacy and prevents
eavesdropping.
The solutions are readily available today.
Security Configuration

IPsec




CHAP



Peers must authenticate each other before data
transfer
Data is encrypted on the wire
Operates at IP layer
One way authentication mechanism, but may be
done by both Initiator and Target
Operates at iSCSI protocol layer
iSCSI CHAP and IPsec rely upon the peer
knowing
a “secret” for authentication

Pre-shared or private key
iSCSI interoperability

Operating system and application vendors often have a
catalog of qualified hardware solutions. The Microsoft
Windows Catalog lists iSCSI hardware devices that have been
qualified. In late 2003, more than 14 leading storage vendors
had qualified their iSCSI hardware products under Microsoft
iSCSI Designed for Windows Logo Program.

Fibre Channel interoperability problems were primarily due to
two issues. First, the vendors implemented the SCSI3
command set differently. Secondly, Fibre Channel lacks built-in
networking capabilities.

In iSCSI’s case, the interoperability issues are greatly reduced.
In addition, SNIA, SNW, and other labs are continuously
working on the interoperability issues.
IDC’s prediction


IDC expects that iSCSI adoption will
commence in most countries in the Asia
Pacific region during 2003 with progressive
deployment expected in 2004. In many cases,
an iSCSI implementation will be
complementary to existing fibre channel SANs.
Overall, IDC believes the two most likely
places where iSCSI will be adopted are:


In smaller organizations that haven't networked
their storage, yet are familiar with TCP/IP.
Large organizations that will use iSCSI to link FC
SANs.
Graham Penn, Director, Asia Pacific Storage, IDC
Microsoft’s iSCSI initiator
The Microsoft iSCSI Software Initiator
version 1.0 package was released to the Web
June 25, 2003. The Microsoft iSCSI software
initiator allows a Windows-based computer
to serve as an iSCSI initiator to connect to
iSCSI targets on an Internet Protocol Storage
Area Network (IP SAN).
 All iSCSI devices appear in Windows as a
local disk and can be managed in Disk
Administrator as any other local disk.
 Download:
http://www.microsoftcom/downloads/details.asp
x?FamilyID=12cb3c1a-15d6-4585-b385befd1319f825&DisplayLang=en

Benefits of using SANs







Enhance applications performance by
freeing up enterprise network
Permits more desktop use of RAID
technology
Consolidated backups and archives
Disk mirroring, backups to disaster
recovery sites
High availability mission critical
databases
Distributed (logical) server clustering
Disk virtualisation
SAN in the WAN




Enhance applications performance by
freeing up enterprise network
Permits more desktop use of RAID
technology
Consolidated backups and archives
Disk mirroring, backups to disaster
recovery sites



High availability mission critical
databases
Distributed (logical) server clustering
Disk virtualisation
WAN
Is SAN in the WAN possible ?

Yes, and SAN traffic loads are
typically less than many people
think :



Very few disks or RAID systems can
stream at > 10 Mbytes/sec, although
peaks of 30 Mbyte/sec are common
Even high performance UNIX servers
can rarely exceed 20 Mbyte/sec
NT servers are much worse, typically
< 10 Mbytes /sec
Is SAN in the WAN possible ?

Tape Subsystems are quite slow:


Disk mirroring depends upon
application


4 to 10 Mbyte/sec streaming is normal.
Peak data of 20 Mbyte/sec maximum
per interface and drive
Transaction or database system often
below 1 Mbyte/sec
Backups may be faster, but are
limited by system (controller / drive)
performance
Is SAN in the WAN possible ?

Given that high bandwidth network
links are increasingly affordable:



T3 (45 Mbit/sec) is capable of around 5
MBytes /sec : easily enough to run a
remote DLT drive or to handle disk
mirroring
OC3 (155 Mbit/sec) is capable of about 17
Mbytes/Sec
OC-12 (622 Mbit/sec) is capable of about
65 Mbytes/sec
Is SAN in the WAN possible?

Yes, but only with pipelined data
transfers

WAN data takes about 5Secs to travel
1Km, or 5mSec for 1000 Km. For a
single disk reading or writing 64KByte
blocks at 10 Mbytes/sec, over 1000 Km
distance, non pipelined operation will
reduce the performance to about 40%
of the transfer speed. If 8 such blocks
are pipelined, performance will be 84%
of transfer speed.
Is SAN in the WAN possible?
64 Kbyte block
6.4 mSec
5 mSec
5 mSec
Disk
Ack
Total time taken to transmit data block and
return ack is 6.4 + 5 + 5 mSec = 16.4 mSec
Lost transmission time due to ack = 10 mSec
Lost efficiency due to ack = 10/16.4 = 60%
Is SAN in the WAN possible ?
Yes, but only with low latency WANs


Data must not be held in queues within
the WAN
1.000
OC-3 (Frame)
T3 (Frame)
mSec Delay
0.800
0.600
0.400
0.200
0.000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
WAN Delays Vs. Efficiency
100
80
128 Kbyte Block
Efficiency
256 Kbyte Block
60
512 Kbyte Block
1MByte Block
40
20
64 Kbyte Block
0
1
5
10
15
20
One way delay mSec
Efficiency at 10 Mbytes/sec for different block sizes
Machine room technology
Today's Storage Area Networks
belong in the Machine Room




Very high speed
Restricted
transmission distance
Unreliable protocols
not designed for
communications use
Primitive windowing
SAN
Storage network speeds
SCSI Type
SCSI-1
SCSI-2 (Narrow)
Speed
Clock
Mbytes/sec
5 MHz
5 Mbytes/sec
10 MHz 10 Mbytes/sec
SCSI-2 (Wide)
10 MHz
Ultra SCSI (Narrow) 20 MHz
Ultra SCSI (Wide) 20 MHz
Ultra 2 SCSI
40 MHz
Ultra 3 SCSI
40 MHz*
20 Mbytes/sec
20 Mbytes/sec
40 Mbytes/sec
80 Mbytes/sec
160 Mbytes/sec
* Ultra320: 320Mbytes/sec
Storage network speeds

Fibre Channel





Up to 100 Mbytes/sec
Runs at 1 Gbaud using 8B/10B
encoding, taken directly from FDDI
standard
Frame based technology based on
FDDI. Uses FDDI checksums
FC-AL shared between < 126 devices
2 Gbaud and 4 Gbaud Fibrechannel
coming
Transmission distances

SCSI



FibreChannel (100 Mbytes/sec, 1.06
Gbaud)



Low voltage differential = 25 metres
Single ended = 3 metres
Singlemode, 1300 nM < 10 Km
Multimode, 850 nM < 300 metres
FibreChannel (25 Mbytes/sec, 266
Mbaud)

Multimode 850 nM < 2 Km
Storage protocols

SCSI


defines a simple bus based transmission
scheme with limited reliability features
Fibre Channel

is conceived as a high speed carrier
mechanism capable of transporting any bit
stream reliably, but is really a local protocol:


Sequence retry is very inefficient (subsequent
sequences are repeated)
Networking layers are missing: FibreChannel is really
a layer 2 technology
Windowing


Performance at a distance requires
efficient windowing
SCSI (and SCSI over FibreChannel) does
not allow this



SCSI has no inherent windowing: commands
are acknowledged individually by the target
Command tag queuing is a solution, but is
not supported by many devices, and is a
higher level solution to a lower level problem
FibreChannel, as a transparent transport
mechanism, does NOT address this problem
SAN in the WAN

To build Storage Networks that operate
over WANs we need:



Realistic data speeds
Adaptation of SCSI or FCP (SCSI over
FibreChannel) to a networking protocol
Reliable stream transport


Disk Profiles operate with FibreChannel Class
3 service, an unacknowledged datagram
service
The only form of ACK is a sequence abort
Applications




Storage Consolidation through IP
SAN features such as storage virtualization,
Capacity-on-Demand mirroring,
TimeMark/TimeView (Disk Journaling),
Replication, Backup and Recovery, Storage
Vaulting, etc, can be carried out in a cost
effective manner.
Diskless Blades
Storage Infrastructure for On-Demand/Utility
Computing
Conclusion

Simply put, iSCSI provides network storage
connectivity at Ethernet prices: iSCSI brings along a
simple and cost-effective solution to storage
networking

Easy implementation for diskless servers,
workstations, blades, and utility/on-demand
computing.

Create opportunities to the traditional networking
and storage companies alike, as the networking
infrastructure can be leveraged.

iSCSI and IP Storage have arrived and will change
the perception of computing forever!

Storage Management is the key to success!