Transcript Technologies for the Data Grid

The Globus Striped GridFTP
Framework and Server
Bill Allcock1 (presenting) John Bresnahan1
Raj Kettimuthu1 Mike Link2
Catalin Dumitrescu2 Ioan Raicu2 Ian Foster1,2
1 Math & Computer Science Division, Argonne National
Laboratory, Argonne, IL 60439, U.S.A.
2 Dept of Computer Science, University of Chicago, Chicago,
IL 60615, U.S.A.
Introduction

Problem we are addressing

A brief discussion of the GridFTP Protocol

Design / Architecture of our implementation

Performance results
Technology Drivers

Internet revolution: 100M+ hosts


Universal Moore’s law: x103/10 yrs


Sensors as well as computers
Petascale data tsunami


Collaboration & sharing the norm
Gating step is analysis
& our old infrastructure?
You are here
114 genomes
735 in progress
What issues are we addressing?

Striping


Collective Operations


storage systems are often clusters, and we need
to be able to utilize all of that parallelism
essentially, the striping should be invisible to
the outside world
Uniform interface

Ideally, any data source can be treated the
same way
What issues are we addressing?

Network Protocol Independence



Need to be able to take advantage of
aggressive protocols on circuits.
Diverse Failure Modes


TCP has well known issues with high
Bandwidth-Delay Product networks
Much happening under the covers, so must be
resilient to failures
End-to-End Performance

We need to be able to manage performance for
a wide range of resources
The GridFTP Protocol
What is GridFTP?


A secure, robust, fast, efficient, standards based,
widely accepted data transfer protocol
A Protocol

Multiple Independent implementation can interoperate



This works. Fermi Lab has an implementation with their
DCache system and U. Virginia has a .Net implementation
that work with ours.
Lots of people have developed clients independent of the
Globus Project.
The Globus Toolkit supplies a reference
implementation:



Server
Client tools (globus-url-copy)
Development Libraries
GridFTP: The Protocol

Existing standards

RFC 959: File Transfer Protocol

RFC 2228: FTP Security Extensions

RFC 2389: Feature Negotiation for the File
Transfer Protocol

Draft: FTP Extensions

GridFTP: Protocol Extensions to FTP for the Grid

Grid Forum Recommendation

GFD.20

http://www.ggf.org/documents/GWD-R/GFD-R.020.pdf
What did the GridFTP protocol add?

Extended Block Mode



allows out-of-order reception of packets
Restart and Performance Markers


data is sent in “packets” with a header
containing a 64 bit offset and length
allows for robust restart and perf monitoring
SPAS/SPOR

striped PASV and striped PORT

allows a list of IP/ports to be returned
What did the GridFTP Protocol add?

Data Channel Authentication


ESTO/ERET



allows for additional processing on the data prior
to storage/transmission
We use this for partial file transfers
SBUF/ABUF


Needed since in third party transfer, you don’t
know who will connect to the listener.
manual and automatic TCP buffer tuning
Options to set parallelism/striping parameters
Client/Server vs 3rd Party
Server
Server
1
2
Establish
control
connection
Establish
data
connection
Client
Client Server Model
3
Server
Establish data connection
1
2
Establish
control
connection
Establish
control
connection
Client
3rd Party Transfer Model
Parallelism vs Striping
Architecture / Design of our
Implementation
Overall Architecture
Info on transfer: restart markers,
performance markers, etc. Server
PI optionally processes these,
then sends them to the client PI
Client PI
Control
Channels
Server PI
Server PI
Internal IPC API
DTP
Internal IPC API
Data Channel
Description of transfer: completely
server-internal communication.
Protocol is unspecified and left up to
the implementation.
DTP
Possible Configurations
Typical Installation
Control
Data
Striped Server
Control
Data
Separate Processes
Control
Data
Striped Server (future)
Control
Data
Data Transfer Processor
Data
source
or sink
Data Storage
Interface
Data Processing
Module
Data Channel
Protocol Module
Data
channel
Data Storage Interface




This is a very powerful abstraction
Several can be available and loaded
dynamically via the ERET/ESTO commands
Anything that can implement the interface
can be accessed via the GridFTP protocol
We have implemented

POSIX file (used for performance testing)

HPSS (tape system; IBM)

Storage Resource Broker (SRB; SDSC)

NeST (disk space reservation; UWis/Condor)
Extensible IO (XIO) system







Provides a framework that implements a
Read/Write/Open/Close Abstraction
Drivers are written that implement the functionality (file,
TCP, UDP, GSI, etc.)
Different functionality is achieved by building protocol
stacks
GridFTP drivers will allow 3rd party applications to easily
access files stored under a GridFTP server
Other drivers could be written to allow access to other
data stores.
Changing drivers requires minimal change to the
application code.
Ported GridFTP to use UDT in less than a day

AFTER the UDT driver was written
Globus XIO Approach
Application
Globus XIO
Driver
Network
Network
Protocol
Network
Protocol
Protocol
Driver
Disk
Driver
Special Device
Globus XIO Framework






Asynchronous support.
Close and EOF Barriers.
Error checking
Internal API for passing
operations down the stack.
User API
Driver Stack
Framework

Moves the data from user to
driver stack.
Manages the interactions
between drivers.
Assist in the creation of
drivers.
Transform
Transform
Transport
What issues are we addressing?

Striping


Collective Operations


storage systems are often clusters, and we need
to be able to utilize all of that parallelism
essentially, the striping should be invisible to
the outside world
Uniform interface

Ideally, any data source can be treated the
same way
What issues are we addressing?

Network Protocol Independence



Need to be able to take advantage of
aggressive protocols on circuits.
Diverse Failure Modes


TCP has well known issues with high
Bandwidth-Delay Product networks
Much happening under the covers, so must be
resilient to failures
End-to-End Performance

We need to be able to manage performance for
a wide range of resources
Performance Results
Comparison in Stream Mode
Throughput comparison on a WAN (60 ms RTT and 1 Gbps
bandwidth)
160
180
140
160
120
140
100
80
NCFTP
60
ZEBRA
40
WUFTP
20
Throughput in mbps
Throughput in mbps
Throughput comparison on a LAN (0.2 ms RTT and 612 Mbps
bandwidth)
120
100
80
NCFTP
60
ZEBRA
40
WUFTP
20
0
0
1M
10M
100M
File size
1000M
1M
10M
100M
File size
1000M
Parallel Stream Performance
Throughput on DOT cluster (2.2 ms RTT and 1 Gbps bandwidth)
Throughput on a LAN (0.2 ms RTT and 612 Mbps bandwidth)
1000
700
900
800
500
iperf
400
zebra mem
zebra disk
300
bonnie
200
Throughput in mbps
Throughput in mbps
600
700
iperf
600
zebra mem
500
zebra disk
400
bonnie
300
200
100
100
0
0
0
5
10
15
20
25
30
35
0
5
10
Number of Streams
15
20
25
30
35
Number of Streams
)Iperf (memory-memory
Bandwidth vs Streams
4500
)Zebra (memory-memory
)Zebra (disk-disk
4000
1400
3500
Bonnie
1200
Bandwidth (in Mbps)
Seek count
3000
2500
Zebra (disk-disk)
2000
1500
1000
800
600
400
1000
200
500
0
0
0
5
10
15
20
Stream s
25
30
35
0
5
10
15
20
Streams
25
30
35
Memory to Memory
Striping Performance
BANDWIDTH Vs STRIPING
30000
25000
Bandwidth (Mbps)
20000
15000
10000
5000
0
0
10
20
30
40
50
60
# Stream = 16
# Stream = 32
Degree of Striping
# Stream = 1
# Stream = 2
# Stream = 4
# Stream = 8
70
Disk to Disk Striping Performance
BANDWIDTH Vs STRIPING
20000
18000
16000
14000
Bandwidth (Mbps)
12000
10000
8000
6000
4000
2000
0
0
10
20
30
40
50
60
Degree of Striping
# Stream = 1
# Stream = 2
# Stream = 4
# Stream = 8
# Stream = 16
# Stream = 32
70
Storage Performance SDSC
1800
1600
1400
Disk throughput (in Mbps)
1200
1000
Write
Read
800
600
400
200
0
0
1
2
3
4
5
Number of nodes
6
7
8
9
Storage Performance NCSA
1800
1600
1400
Disk throughput (in Mbps)
1200
1000
Write
Read
800
600
400
200
0
0
1
2
3
4
5
Number of nodes
6
7
8
9
Scalability Results
GridFTP Server Performance
Upload 10MB file from 1800 clients to ned-6.isi.edu:/dev/null
2000
100
1900
Load
1800
90
1700
Memory Used
1600
80
1500
70
CPU %
1300
1200
60
1100
1000
50
Throughput
900
800
40
700
600
30
Response Time
500
400
20
300
200
10
100
0
0
0
500
1000
1500
2000
Time (sec)
2500
3000
3500
CPU %
/ Throughput (MB/s))
# of Concurent Machines
/ Response Time (sec)
1400
Summary



The GridFTP protocol provides a good set of
features for data movement requirements in the
Grid.
The Globus Striped Server (Zebra)
implementation of this protocol provides a
flexible design / architecture for integrating with
different communities, storage systems, and
protocols.
Our implementation is robust and performant
over a range of environments.
Questions?