Transcript Distributed File Systems

Distributed File Systems
NFS
AFS
SMB
http://en.wikipedia.org/wiki/Network_file_system
DFS INTRODUCTION
Distributed File Systems
(DFS)

Any computer file system that supports sharing
of files and other resources as persistent storage
over a computer network


First file servers were developed in the 1970s
Network File System (NFS)




Created at Sun Microsystems
1985
First widely used distributed file system
Other notable distributed file systems


Andrew File System (AFS)
Server Message Block SMB

AKA CIFS
DISTRIBUTION
Distribution

Distributed File System (DFS)

Network file system with:
Clients, servers, and storage devices
 Spread throughout the machines of a distributed
system or intranet


Service activity occurs across the network

System has multiple and independent storage
devices
 Some DFS servers run on dedicated machines
 Some machines can be both a server and a client

A DFS can be implemented:
As part of a distributed operating system
 By a software layer

 Managing the communication between conventional
operating systems and file systems
Distribution
 Distinctive
 System
 many
feature of a DFS:
with
- and  autonomous
 Clients
and Servers
Note:

NFS is a DFS for the Unix family




Network File System
Originated in the Unix world
Implementations for other OSes
NFS sometimes used as a synonym for
DFS

Be careful of the distinction
DFS ATTRIBUTES
Transparency

A DFS should appear to its users to be the
same as a conventional, centralized file
system

Multiplicity and dispersion of its servers and
storage devices should be invisible


Client interface used by programs should not
distinguish between local and remote files
DFS locates the files and arranges the
transport of data
Performance

Most important performance measurement of a
DFS:


Amount of time needed to satisfy service requests
Typical performance constraints:

Conventional file systems (local drives):



Disk-access time
Small amount of CPU-processing time
DFS:


Additional overhead due to the distributed structure
Includes:
 Time to deliver the request to a server
 Time to deliver the response to the client
 For each direction, a CPU overhead of running the
communication protocol software
Performance

Performance of DFS can be viewed as one
dimension of its transparency

Ideally, it would be comparable to that of a
conventional file system
Concurrent file updates

DFS should allow

Multiple client processes




On multiple machines
Access and update the same files
Updates to the file from one client should not
interfere with access and updates from other
clients
Concurrency control or locking may be:


Built into the file system
Provided by an add-on protocol
EXAMPLES
Sample Distributed File Systems

Client-server file systems





9P
Amazon S3
Coda
Secure File System (SFS)
Self-certifying File System (SFS)
Sample Distributed File Systems

Distributed file systems


Andrew File System (AFS), OpenAFS
DCE Distributed File System




Google File System

File system that is optimized for Google's core storage needs

Java software framework that supports distributed applications running
on large clusters of commodity computers
Hadoop
Kosmos FS





Remote file access protocol used with the Distributed Computing
Environment
High performance distributed file system. It is also integrated with
hadoop.
Lustre

Distributed file system for large scale cluster computing

Set of client and server services for organizing many distributed SMB
file shares into a distributed file system
Microsoft Distributed File System
GlusterFS

Platform for developing clustering applications

Parallel file system that focuses on high performance access to large
data sets
Parallel Virtual File System
Network File Systems
http://en.wikipedia.org/wiki/Network_File_System_%28proto
col%29
NFS PROTOCOL
Network File System Protocol

Network file system protocol:

Allows a user on a client computer to access files over a
network



Originally developed by Sun Microsystems in 1984
NFS builds on Open Network Computing Remote
Procedure Call system


As if the network devices were attached to its local disks
ONC RPC
Network File System protocol specified in:



RFC 1094
RFC 1813
RFC 3530 (which obsoletes RFC 3010)
VERSIONS AND VARIATIONS
Versions and variations

Version 1


Used only for Sun in-house experimental
purposes
Not released to public
Versions and variations

Version 2


Originally operated entirely over UDP
Defined in RFC 1094


March 1989
Designers meant to keep the protocol stateless

Locking (for example) implemented outside of the
core protocol
Versions and variations

Version 3 (RFC 1813, June 1995) added:

Support for 64-bit file sizes and offsets

Handle files larger than 4 gigabytes (GB)
 32 bit address limit

Support for asynchronous writes on the server


Additional file attributes in many replies


Avoid the need to re-fetch them
READDIRPLUS operation



Improve write performance
Get file handles and attributes
Along with file names when scanning a directory
Assorted other improvements
Versions and variations

At the introduction of Version 3


Vendor support for TCP as a transport-layer
protocol began increasing
Several vendors had already added support for
NFS Version 2 with TCP as a transport


Sun Microsystems added support for TCP as a
transport for NFS at the same time it added support
for Version 3
Using TCP as a transport made using NFS over
a WAN more feasible
Versions and variations

Version 4


First version developed with the IETF
After Sun Microsystems handed over the
development of the NFS protocols
Versions and variations

Version 4 includes:




Performance improvements
Mandates strong security
Introduces a stateful protocol
Defined by


RFC 3010, December 2000
Revised in RFC 3530, April 2003
Versions and variations

Various side-band protocols have become
associated with NFS, including:

Byte-range advisory Network Lock Manager
(NLM) protocol


Added to support UNIX System V file-locking APIs
Remote quota reporting (RQUOTAD) protocol

Allow NFS-users to view their data-storage quotas on
NFS servers
Versions and variations

WebNFS

Allows NFS to




Integrate more easily into Web-browsers
Enable operation through firewalls
Was an extension to Version 2 and Version 3
Sun Microsystems open sourced their
WebNFS implementation

via https://yanfs.dev.java.net/
PLATFORMS
Platforms
 NFS
 Most
common with UNIX like OSes
 Other software platforms:
 Mac
OS
 Microsoft Windows
 Novell NetWare
 IBM AS/400
Platforms

Alternative remote file access protocols include

Server Message Block (SMB) protocol





Also known as CIFS
Apple Filing Protocol (AFP)
NetWare Core Protocol (NCP)
OS/400 File Server file system (QFileSvr.400)
Note:



SMB and NetWare Core Protocol (NCP) are most common
on Microsoft Windows
AFP is most common on (old) Macintosh systems
QFileSvr.400 is most common on AS/400 systems
TYPICAL IMPLEMENTATION
Typical implementation
 Unix-style
 Client


scenario:
Requires access to data
Server

Serves data
Typical implementation

Server Side

Server



Server administrator


Determines what resources to make available
 Exporting names and parameters of directories
 Typically using the
 /etc/exports configuration file
 exportfs command
Server security-administration


NFS daemon (nfsd)
Make data generically available to clients
Recognizes and approves validated clients
Server network configuration

Lets appropriate clients can negotiate with it through any
firewall system
Typical implementation

Client Side

Client machine



Requests access to exported data
Issues a mount command
Users on the client machine


View and interact with mounted file systems on the server
Within the parameters permitted
 Permissions, etc.

Note: automating the NFS mounting process is
typical

E.g. using /etc/fstab and/or automounting facilities
NFS PROTOCOL DEVELOPMENT
VERSUS
COMPETING PROTOCOLS
NFS Protocol development versus
competing protocols

1980s




NFS and ONC figured prominently in the network-computing war between Sun
Microsystems and Apollo Computer, and later the UNIX wars (ca 1987-1996)
between AT&T and Sun on one side, and Digital Equipment, HP, and IBM on the
other.
During the development of the ONC protocol (called SunRPC at the time), only
Apollo's Network Computing System (NCS) offered comparable functionality.
Two competing groups developed over fundamental differences in the two
remote procedure call systems. Arguments focused on the method for dataencoding — ONC's External Data Representation (XDR) always rendered integers
in big-endian order, even if both peers of the connection had little-endian
machine-architectures, whereas NCS's method attempted to avoid byte-swap
whenever two peers shared a common endianness in their machinearchitectures. An industry-group called the Network Computing Forum formed
(March 1987) in an (ultimately unsuccessful) attempt to reconcile the two
network-computing environments.
Later, Sun and AT&T announced that the two firms would jointly develop AT&T's
next version of UNIX: System V Release 4. This caused many of AT&T's other
licensees of UNIX System V to become concerned that this would put Sun in an
advantaged position, and it ultimately led to Digital Equipment, HP, IBM, and
others forming the Open Software Foundation (OSF) in 1988. Ironically, Sun and
AT&T had previously competed over Sun's NFS versus AT&T's Remote File
System (RFS), and the quick adoption of NFS over RFS by Digital Equipment, HP,
IBM, and many other computer vendors tipped the majority of users in favor of
NFS.
OSF solicited the proposals for various technologies, including the remote
procedure call (RPC) system and the remote file access protocol. In the end,
proposals for these two requirements, called respectively, the Distributed
Computing Environment (DCE), and the Distributed File System (DFS) won over
Sun's proposed ONC and NFS. DCE derived from a suite of technologies,
including NCS and Kerberos. DFS used DCE as the RPC and derived from AFS.
Protocol development versus
competing protocols

1990s



Sun Microsystems and the Internet Society (ISOC) reached an agreement to
cede "change control" of ONC RPC so that ISOC's engineering-standards body,
the Internet Engineering Task Force (IETF), could publish standards documents
(RFCs) documenting the ONC RPC protocols and could extend ONC RPC. OSF
attempted to make DCE RPC an IETF standard, but ultimately proved unwilling
to give up change-control. Later, the IETF chose to extend ONC RPC by adding a
new authentication flavor, RPCSEC GSS, in order to meet IETF's requirements
that protocol standards have adequate security.
Later, Sun and ISOC reached a similar agreement to give ISOC change control
over NFS, although writing the contract carefully to exclude NFS version 2 and
version 3. Instead, ISOC gained the right to add new versions to the NFS
protocol, which resulted in IETF specifying NFS version 4 in 2003.
2000s

By the 21st century, neither DFS nor AFS had achieved any major commercial
success as compared to CIFS or NFS. IBM, which had previously acquired the
primary commercial vendor of DFS and AFS, Transarc, donated most of the AFS
source code to the free software community in 2000. The OpenAFS project lives
on. In early 2005, IBM announced end of sales for AFS and DFS.
Present

NFSv4.1 presents new opportunities for the NAS
community

Primary addition is Parallel NFS (pNFS)


Protocol defines a method of separating the meta-data of a file
system from the location of the file data




Holds the names of files and their data under the single umbrella
of the server
Products exist that are multi-node NFS servers


Meta-data: names and attributes
Goes beyond simple name/data separation
Defines method of striping the data amongst a set of data servers
Very different from the traditional NFS server


A method of introducing data access parallelism
Limited participation by the client in separation of meta-data and
data
NFSv4.1 client can be enabled to be a direct participant in the
exact location of file data

Avoid sole interaction with the single NFS server when moving
data
Present

NFSv4.1 pNFS server:

A collection or community of server resources
or components


Community members are controlled by the metadata server
pNFS client still accesses a single metadata server for traversal or interaction
with the namespace

Client moves data to and from the server

Directly interact with the set of data servers
belonging to the pNFS server community
Present

NFSv4.1 provides:




Sessions
Directory Delegation and Notifications
Multi-server Namespace
ACL/SACL/DACL





access control lists
system access control list
discretionary access control list
Retention Attributions
SECINFO_NO_NAME
Future

NFSv4.2


Under development
Some features are currently useable
AFS
http://en.wikipedia.org/wiki/Andrew_File
_System
Andrew File System
(AFS)

Distributed networked file system

Developed by Carnegie Mellon University


Named for



Part of the Andrew Project
Andrew Carnegie
Andrew Mellon
Primary used in distributed computing
FEATURES
Features

AFS has several benefits over traditional
networked file systems



Particularly in the areas of security and scalability
Not uncommon for enterprise AFS cells to exceed 50,000
clients
Uses Kerberos for authentication


Implements access control lists on directories for users and
groups
Each client caches files on the local file system for
increased speed on subsequent requests for the same file

Allows limited file system access in the event of a server
crash or a network outage
Features

Read and write operations on an open file are
directed only to the locally cached copy

When a modified file is closed



Cache consistency maintained by callback
When a file is cached



Server makes a note of this
Will inform the client if the file is updated by someone else
After any client, server, or network failure (including a
time-out)


Only the changed portions copied back to the file server
Callbacks are discarded and must be re-established
Re-establishing a callback involves:


Status check
Does not require re-reading the file itself
Features

Consequence of the whole file locking strategy


AFS does not support large shared databases or record
updating within files shared between client systems
Deliberate design decision


Based on the perceived needs of the university computing
environment
Leads to the use of a single file per message in the
original email system for the Andrew Project, the
Andrew Message System, rather than the more
customary single file per mailbox
Features

A significant feature of AFS is the volume



Tree of files
Sub-directories
AFS mountpoints


Volumes are created by administrators



Links to other AFS volumes
Linked at a specific named path in an AFS cell
Once created, users of the filesystem may create directories
and files as usual without concern for the physical location of
the volume
A volume may have a quota assigned to it in order to limit
the amount of space consumed.

As needed, AFS administrators can move that volume to
another server and disk location without the need to notify
users

Moving can occur while files in that volume are even being used
Features

AFS volumes can be replicated to read-only
cloned copies


When accessing files in a read-only volume, a client
system will retrieve data from any read-only copy
If a copy becomes unavailable, clients will look for any
remaining copies



Users of that data are unaware of the location of the readonly copy
Administrators can create and relocate copies as needed
AFS command suite guarantees that all read-only
volumes contain exact copies of the original read-write
volume at the time the read-only copy was created
Features

File name space on an Andrew workstation
is partitioned into a shared and local name
space

Shared name space identical on all
workstations


Usually mounted as /afs on the Unix file system
Local name space unique to each workstation

Contains only temporary files needed for workstation
initialization and symbolic links to files in the shared
name space
Features

Andrew File System


Heavily influenced Version 4 of Sun
Microsystems' NFS
Variant of AFS adopted by the Open Software
Foundation


Distributed File System (DFS)
Part of their Distributed computing environment
IMPLEMENTATIONS
Implementations

There were three major implementations

Transarc (IBM)





OpenAFS
Arla
Transarc software is losing support and is
deprecated
AFS (version two)


Released as OpenAFS and retired
Predecessor of the Coda file system
Fourth implementation in the Linux kernel source
code since at least version 2.6.10

Committed by Red Hat



A fairly simple implementation
Still in its early stages of development
Incomplete
AFS FILE PERMISSIONS
Available permissions

The following Access Control List (ACL)
permissions can be granted:

Lookup (l)

Allows users to:
 List the contents of the AFS directory
 Examine the ACL associated with the directory
 Access subdirectories

Insert (i)


Delete (d)


Allows users to add new files or subdirectories to the
directory
Allows users to remove files and subdirectories from the
directory
Administer (a)


Allows users to change the ACL for the directory
Users always have this right on their home directory
 Even if they accidentally remove themselves from the ACL
Available permissions

Permissions that affect files and subdirectories
include:

Read (r)



Write (w)



Allows users to look at the contents of files in a directory
and list files in subdirectories
Files that are to be granted read access to any user,
including the owner, need to have the standard UNIX
"owner read" permission set
Allows users to modify files in a directory
Files that are to be granted write access to any user,
including the owner, need to have the standard UNIX
"owner write" permission set
Lock (k)


Allows the processor to run programs that need to "flock"
files in the directory
See the UNIX man page for "flock" for more details
Available permissions

AFS includes Application ACLs (A)-(H)
which have no effect on access to files



ACL – Access Control Lists
Affects directories only
Overrides Unix permissions
SMB
http://en.wikipedia.org/wiki/Server_Message_Block
Server Message Block
(SMB)

Operates as an application-level network protocol

Applied to shared access to







Files
Printers
Serial ports
Miscellaneous communications
Between nodes on a network
Provides an authenticated Inter-process
communication mechanism
Most usage of SMB involves computers running
Microsoft Windows:

Microsoft environments users often know it simply as
"Microsoft Windows Network"
Server Message Block
(SMB)

When discussing SMB, one should
distinguish:





SMB the protocol
SMB services that run on the protocol
NetBIOS
DCE/RPC services that use SMB as an
authenticated Inter-process communication
channel (over named pipes)
"Network Neighborhood" protocols which
primarily (but not exclusively) run as datagram
services directly on the NetBIOS transport
HISTORY
History

Barry Feigenbaum originally invented SMB at IBM


Turned DOS "Interrupt 33" (21h) local file-access into a
networked file-system
Microsoft made considerable modifications to the
most commonly used version

Merged the SMB protocol with the LAN Manager


Developing with 3Com (circa 1990)
Continued to add features to the protocol


Windows for Workgroups (circa 1992)
Later versions of Windows
History

Original design of SMB envisaged it
running on top of the NetBIOS and
NetBEUI APIs

Typically implemented with




NBF (NetBIOS Frame)
NetBIOS over IPX/SPX
NBT (NetBIOS TCP/IP)
SMB can also run directly on the TCP/IP
protocols

Introduced with Windows 2000
History

Circa Sun Microsystems announcing WebNFS


Microsoft launched an initiative in 1996 to rename SMB to
Common Internet File System (CIFS)
Added more features, including

Support for:






Symbolic links
Hard links
Larger file sizes
Attempt at supporting direct connections without the NetBIOS
Largely experimental effort that required further refinement
Microsoft submitted some partial specifications as
Internet-Drafts to the IETF

Submissions have expired
History

Windows Vista

introduced Server Message Block 2.0
History

Side issue:

Samba project originated aim of



Reverse engineer SMB
Provide a free implementation of a compatible SMBclient and server for use with non-Microsoft operating
system
Driven by the importance of the SMB protocol
in interacting with the widespread Microsoft
Windows platform
Resume 3/27
IMPLEMENTATION
Implementation

Client-server approach

SMB works through a peer-to-peer approach



One section of the SMB protocol specifically deals with
access to file systems



Client makes specific requests
Server responds accordingly
Clients may make requests to a file server
Other sections of the SMB protocol specialize in interprocess communication (IPC)
Developers have optimized the SMB protocol for local
subnet usage

Users have also put SMB to work to access different subnets
across the Internet
 Involving file-sharing or print-sharing in MS Windows
environments
Implementation

Client-server approach

SMB servers make their file systems and other resources
available to clients on the network

Client computers may want access to the shared file
systems and printers on the server
 For this primary function, SMB is best-known and most
heavily used

SMB file-server aspect would count for little without the NT
domains suite of protocols
 Provide NT-style domain-based authentication

NT Domains protocols offer MSRPC services available
almost exclusively available on SMB IPC "named pipe“
 Almost all implementations of SMB servers use NT Domain
authentication to validate user-access to resources
Implementation

Microsoft's modifications

Microsoft has added several extensions to its
own SMB implementation.

Added NT LAN Manager (NTLM) Version 2
 NTLM version 1 used DES in a flawed manner
 Derived from the original legacy SMB specification's
requirement to use IBM "LanManager" passwords

NT 4.0 Domain Logon protocols use 40-bit encryption
outside of the United States of America
 Does not conform with modern security standards
Implementation

Points of interest

SMB's "Inter-Process Communication"
mechanism deserves a specific mention

SMB "IPC" system provides named pipes
 Provides one of the first few inter-process mechanisms
commonly available to programmers
 Provides a means for services to inherit the
authentication carried out when a client first
connected to an SMB server
 Inherited authentication in named pipes has become so
ubiquitous and transparent that both Windows-users
and programmers who use the Windows API often
simply take it for granted
Implementation

Points of interest

Some services that operate over named pipes
also allow MSRPC client programs to perform
authentication


Over-rides the authorization provided by the SMB
server
Only in the context of the MSRPC client program that
successfully makes the additional authentication
Implementation

Points of interest

SMB uses opportunistic locking



Special type of locking-mechanism
Improves performance
SMB serves as the basis for Microsoft's
Distributed File System implementation
VERSIONS AND
IMPLEMENTATIONS
Versions and implementations

The following list explicitly refers to "SMB" as
including an SMB client or an SMB server



Including the various protocols that extend SMB
For simplicity and conciseness and vagueness,
however, the list omits mention of the extent or
completeness of the reimplementation or porting
status for any of these implementations
Lumps them all together simply as "SMB"
Versions and implementations

List…

Samba includes an SMB server and a command-line SMB client



Samba TNG: a fork of Samba
The Linux kernel includes two SMB client implementations that use the
Linux VFS



smbfs
originally derived from the FreeBSD smbfs; they use the NetBSD and OS X VFS
Solaris has a project called CIFS client for Solaris


smbfs that uses its VFS
NetBSD and Mac OS X include SMB client implementations:


Provides access to NetWare volumes for Microsoft Network clients
FreeBSD includes an SMB client implementation


users can access the same data through both protocols
Novell NetWare version 6 and later has a CIFS server implementation


Provides access to files on an SMB server through the standard file system API: smbfs and cifs
ONStor Inc. offers an SMB implementation that also supports NFS protocol


Free re-implementation of the SMB protocol and of the Microsoft extensions to it
Based on the Mac OS X smbfs
Sun Microsystems added in-kernel CIFS support to Solaris in October 2007
Sun Microsystems Cascade (aka PC-Netlink) represents a port of Advanced
Server for Unix

Sun took over two years making the code useful, due to the poor quality of the original port
Versions and implementations

Continued…

FreeNAS runs FreeBSD for Network-attached storage (NAS) services


Advanced Server for Unix (AS/U) comprises a port of Windows NT 3.51's
SMB server code to Unix











Microsoft licensed the code to AT&T, which then licensed it to major Unix vendors.
The poor quality of the original port (allegedly carried out by Microsoft itself) has caused any vendor sublicensing it significant grief
VERITAS Software has an implementation of SMB
SCO has a port of Advanced Server for Unix
SCO has VisionFS, a Microsoft-independent re-implementation of SMB
developed through reference to Samba source code
Network Appliance has an SMB server implementation
Objective Development's Sharity provides an SMB file-system client for Unix
The Alfresco content-management system includes a Java implementation of
SMB
JCIFS offers an implementation of SMB in Java
RTSMB, a CIFS/SMB implementation written in ANSI C.


Supports protocols including CIFS/Samba
EBS designed RTSMB from scratch, independently of MS or SAMBA design reference, to run in embedded
devices
Visuality Systems NQ CIFS, a CIFS (SMB) server and client solution for
embedded devices — ported to many popular real-time operating systems
(RTOSs)
Thursby Software Systems offers a commercial implementation of SMB/CIFS
for Mac OS
NFS is most common on
16%
11%
s
in
do
w
ic r
os
of
tW
M
M
ac
OS
x
0%
ni
x
4.
L in
ux
/U
3.
74%
M
VS
2.
IBM MVS
Linux/Unix
Mac OS x
Microsoft
Windows
IB
M
1.
SMB is most common on
11%
6%
M
s
in
do
w
ic r
os
of
tW
OS
x
0%
M
ac
4.
L in
ux
3.
83%
M
VS
2.
IBM MVS
Linux
Mac OS x
Microsoft
Windows
IB
M
1.