Transcript NeST: Network Storage - Computer Sciences Dept.

NeST: Network Storage
Flexible Commodity Storage Appliances
John Bent
Miron Livny, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau
Computer Sciences Department
University of Wisconsin-Madison
[email protected]
http://www.nestproject.com
Terms
› Appliance (Merriam-Webster)
b : an instrument or device
designed for a particular use;
specifically a household or
office device
› Storage appliance
Storage plus access methods
www.cs.wisc.edu/condor
What storage users want
› Reliability and availability
› Manageability
cost of management > cost of storage itself
“no futz” computing
› Scalability
› Performance
www.cs.wisc.edu/condor
What storage vendors have
› NetApp, EMC, etc.
› Manageable
Just plug it in and it works
Administrative web interface
› Reliable and available
Standard RAID techniques
› High performance
Specialized, thin OS focused on
serving files
www.cs.wisc.edu/condor
What storage vendors get,
annual revenues
NetApp
EMC
$800 million in 2000
$9 billion in 2000
www.cs.wisc.edu/condor
What’s the problem?
› False coupling between HW and SW
› “Playground syndrome”
› Myth of specialization
www.cs.wisc.edu/condor
H/W and S/W are bundled
› Hardware decisions are imposed
› Hard to ride commodity curve
Example:
• Netapp F720
– $35,000.00, 252 GB
– $139 / GB
• Linux server
– $18,000.00, 365 GB
– $49 / GB
www.cs.wisc.edu/condor
“Playground syndrome”
› “We have storage appliances . . .
if you use these protocols,
if you use these security mechanisms,
if you are comfortable with our data
semantics”
› Non-flexible software entity
www.cs.wisc.edu/condor
Myth of specialization
› Specialize for one protocol on one machine
› Specialization decreases over time as
Protocols are added
Product line expands
› Example: Netapp software
Generation 1 fit on a single floppy
Generation 2 took six
Generation 3?
www.cs.wisc.edu/condor
Alternatives?
› Appliance (MerriamWebster)
a : a piece of equipment for
adapting a tool or machine to
a special purpose
www.cs.wisc.edu/condor
Our game?
› Flexible, commodity based, softwareonly storage appliances
› Goal
Find a networked machine
“Drop” some software on it
Have a ready to use storage appliance
with flexible mechanisms
www.cs.wisc.edu/condor
New worlds, new problems
› Diverse hardware, software platforms
Netapp, EMC advantage
• fewer platforms, control over OS
Our approach
• Automate configuration to each host system
– Hardware example - use file system or self-manage
– Software example - use either read/write or mmap
› Cost of flexibility
› Key is design of the software
www.cs.wisc.edu/condor
Outline
› Introduction
› Building flexible storage modules
Big picture
Protocol layer
Concurrency architecture
Storage layer
› Motivations for flexible storage appliances
› Conclusion and current status
www.cs.wisc.edu/condor
NeST structure
› Cleanly separated modules for
communication, transfer and storage
Protocol layer
• Maps diverse protocols into common control flows
Concurrency architectures
• Different models to maximize system throughput
Storage layer
• Provides abstract interface to disks
www.cs.wisc.edu/condor
NeST structure
GFTP NeST WiND HTTP
NFS
Protocol Layer
Concurrency Architecture
Event
MultiMultidriven
process
threaded
Storage Layer
Raw disk
Local FS
RAID
www.cs.wisc.edu/condor
Control
Logic
NeST structure
GFTP NeST WiND HTTP
NFS
Transfer request
Protocol Layer
Concurrency Architecture
Event
MultiMultidriven
process
threaded
Storage Layer
Raw disk
Local FS
RAID
www.cs.wisc.edu/condor
Control
Logic
NeST structure
GFTP NeST WiND HTTP
NFS
Transfer request
Protocol Layer
Concurrency Architecture
Event
MultiMultidriven
process
threaded
Storage Layer
Raw disk
Local FS
RAID
www.cs.wisc.edu/condor
Control
Logic
NeST structure
GFTP NeST WiND HTTP
NFS
Transfer request
Protocol Layer
Concurrency Architecture
Event
MultiMultidriven
process
threaded
Storage Layer
Raw disk
Local FS
RAID
www.cs.wisc.edu/condor
Control
Logic
NeST structure
GFTP NeST WiND HTTP
NFS
Transfer request
Protocol Layer
Concurrency Architecture
Event
MultiMultidriven
process
threaded
Storage Layer
Raw disk
Local FS
RAID
www.cs.wisc.edu/condor
Control
Logic
NeST structure
GFTP NeST WiND HTTP
NFS
Transfer request
Protocol Layer
Concurrency Architecture
Event
MultiMultidriven
process
threaded
Storage Layer
Raw disk
Local FS
RAID
www.cs.wisc.edu/condor
Control
Logic
Protocol layer
A collection of servers is less than the sum of their parts.
GFTPd
HTTPd
Operating system
www.cs.wisc.edu/condor
Protocol layer
A collection of servers is less than the sum of their parts.
GFTPd
HTTPd
GFTPd
NFS
HTTPd
NeST
Operating system
Operating system
www.cs.wisc.edu/condor
Consolidate protocols
› Single point of control
Storage quotas and guarantees can be
supported across multiple protocols.
Bandwidth can be controlled and quality
of service can be guaranteed.
› Single administrative interface
Set policies
Manage user accounts
www.cs.wisc.edu/condor
Protocol layer implementation
› Each protocol listens on well-defined port
› Central control accepts connections
› Protocol layer reads from connection and
returns generic request object
› Like Linux V-nodes
Add new protocol by writing a couple of
methods
www.cs.wisc.edu/condor
Protocol layer example,
directory list request
“31: LIST”
“5”
FTP
Control Logic
NeST
speak
Storage
Layer
Protocol layer
www.cs.wisc.edu/condor
Protocol layer example,
directory list request
“31: LIST”
“5”
FTP
Directory list
Control Logic
NeST
speak
Storage
Layer
Protocol layer
www.cs.wisc.edu/condor
Protocol layer example,
directory list request
“31: LIST”
FTP
Directory list
Control Logic
Directory list
“5”
NeST
speak
Storage
Layer
Protocol layer
www.cs.wisc.edu/condor
Protocol layer example,
directory list request
“31: LIST”
FTP
Directory list
Control Logic
Directory list
“5”
NeST
speak
Protocol layer
Linked list
www.cs.wisc.edu/condor
Storage
Layer
Protocol layer example,
directory list request
“31: LIST”
FTP
Directory list
Control Logic
Directory list
“5”
NeST
speak
Linked list
Protocol layer
Linked list
www.cs.wisc.edu/condor
Storage
Layer
Protocol layer example,
directory list request
“31: LIST”
FTP
“ftp, ftp, ftp”
Directory list
Control Logic
Directory list
NeST
speak
“5”
“nest, nest”
Linked list
Protocol layer
Linked list
www.cs.wisc.edu/condor
Storage
Layer
Concurrency architecture
› Three difficult goals
Low latency
High bandwidth
Multiple simultaneous clients
› No single portable solution
Provide multiple models to provide solutions
on a range of different platforms
• Multi-threaded, multi-process, event driven
www.cs.wisc.edu/condor
Storage layer
› Three needed areas of flexiblity
File systems interfaces
• Example: read()/write() or mmap()
Abstract storage models
• RAID, JBOD, etc.
User account administration
• Creation and removal
• Quotas and guarantees for users and groups
www.cs.wisc.edu/condor
Outline
› Introduction
› Building flexible storage modules
› Motivations for flexible storage appliances
Communication protocols
Replacement costs
Data semantics
Security and authentication
Condor NeSTs
› Conclusion and current status
www.cs.wisc.edu/condor
Communication protocols
› The Esperanto problem
› Too many protocols to implement them all
› Too many clients use proprietary protocols
Storage must allow pluggable protocols.
www.cs.wisc.edu/condor
Replacement costs
› Infinite cost to replace first class data.
› Variable cost to replace cached data
depending on size and distance.
› Variable cost to replace job output files
depending on computation cost.
Cheap cached files
First class data
www.cs.wisc.edu/condor
Replacement costs
› Infinite cost to replace first class data.
› Variable cost to replace cached data
depending on size and distance.
› Variable cost to replace job output files
depending on computation cost.
Cheap cached files
First class data
Cost aware storage can effectively increase its own
capacity.
www.cs.wisc.edu/condor
Data semantics
› Must stored objects be protected from
read and write dependencies?
› Is transaction support desired?
› Acceptable replies to storage requests.
www.cs.wisc.edu/condor
Data semantics, example
› Problem
PFS on top of FTP fakes open
read may then return file not found
› Solution
Mechanisms are needed to support flexible
semantics independent of the transfer
protocol.
www.cs.wisc.edu/condor
Data semantics, example
› Problem
PFS on top of FTP fakes open
read may then return file not found
› Solution
Mechanisms are needed to support flexible
semantics independent of the transfer
protocol.
Divorce semantics from the protocol.
www.cs.wisc.edu/condor
Security and authentication
›
›
›
›
›
Ownership
Privacy
Encryption
Authentication
Access rights
www.cs.wisc.edu/condor
Who, when, how and how much?
› Who is allowed to use the storage?
› Promiscuity and monogamy are easy
› Polygamy is also easy
Promiscuous
Abstinent
www.cs.wisc.edu/condor
› Problem
Do I know you?
Migrant grid users may need temporary,
preferential storage access
› Solution
Provide mechanisms to
• advertise available storage
• create self-destructing user accounts
Matchmake applications with storage opportunities.
www.cs.wisc.edu/condor
Condor NeSTs
› Better, smarter checkpoint servers
› Checkpoints are just another data file
› NeST transparently replicates and
migrates data files
› Condor jobs access data files from
closest NeST
› Flexible policy support for managing
disk and network resources
www.cs.wisc.edu/condor
Condor NeSTs,example ReqEx
Tape Library
NeST
Compute cluster
ReqEx scheduler
NeST
NeST
www.cs.wisc.edu/condor
Compute cluster
Outline
›
›
›
›
Introduction
Building flexible storage solutions
Motivations for flexible storage appliances
Conclusion
Current status
Future work
Concluding remarks
www.cs.wisc.edu/condor
Current status
› Concurrency architectures are done
Gets, puts, reads and writes perform well
› Virtual protocol class interface is built
NeST speak is fully implemented
Grid ftp is partially implemented
› Simple first implementation of storage
reservations and remote quota
management is done
www.cs.wisc.edu/condor
Future work
› Discovery process of client storage
requirements
› Quality of service guarantees for
bandwidth and storage
› Support for transient and opportunistic
users
› Transparent inter-NeST cooperation
www.cs.wisc.edu/condor
Concluding remarks
› Return storage to the commodity
curve by creating software-only
storage appliances
› Allow greater storage flexibility for a
wide range of application needs
www.cs.wisc.edu/condor