Transcript Delay Measurement Rev. 1 - Network Computing Lab

The Kangaroo Approach
to Data Movement on
the Grid
Author: D. Thain, J. Basney, S.-C. Son, and M.
Livny
From: HPDC 2001
Presenter: NClab, KAIST, Hyonik Lee
Motivation

In many Grid applications, client usually gets data from
data server, processes them and sends the results back
to data server.
Problem

While data are being transferred to destination server,
error or latency can occur due to
Server crash
 Performance variation of server
 Exhausted resources
-> not resilient


The throughput of grid application is not good because
the client should be blocked until the data are
transferred in such a hostile environment.
Solution Approach

Use intermediary memory or disk buffer.


Background processes move data and handling errors.
Interface




Get
Put
Commit
push
Advantages

One-hop Kangaroo

Insulates the client form many difficulties




Network failure, destination machine crash
Exhausted resources (e.g. BW)
Read operation can be satisfied from cached data.
Multi-hop Kangaroo


Multiple hops help avoid the need to co-allocate network
resources along all hops.
Multiple hops increase the available spooling space.
Architecture
Application
open, read, write, close, fsync
File
System
Adaptation
get, put, push, commit
Consistency
get,
commit,
put,
push
open, read, write, close, fsync
Consistency
Consistency
get, put,
ack
ack
push,commit
Kangaroo
Transport
Kangaroo
Transport
Kangaroo
Transport
TCP/IP
TCP/IP
TCP/IP
Adaptation Layer

POSIX

It is unreasonable to expect programmers to convert existing
applications to work with Kangaroo.
Adaptation Layer converts standard POSIX operations into Kangaroo
operations.
A number of operations are missing in the Kangaroo interface, but it
is sufficient to admit many grid applications that simply must read
and write data.
read
write
fsync
exit
open without create
open with create
lseek
close
get
put
push
commit
Fd
Kind
0
Unix
1
Kang
2
Kang
3
Unix
4
Kang
…
…
Kangaroo

Just update table
Host
Path
offset
/dev/null
0
Coral
/tmp/out.2
1056
Coral
/tmp/err.2
3122
/etc/hosts
785
dbhost
/data/db
59687
…
…
…
Consistency Layer


Commit causes the caller to block until all outstanding
changes have been written to some stable storage.
Push causes the caller to block until all outstanding
changes have been delivered to their respective
destinations.
Kangaroo Transport Layer

Performance


Uses all available resources (net, memory, disk) to maximize
throughput
Error management


Retry
Delay
Experiments
Blocking procedure call
Non-blocking message