Transcript BDTS and Its Evaluation on IGTMD link

BDTS and Its Evaluation on
IGTMD link
C. Chen, S. Soudan, M. Pasin, B. Chen, D. Divakaran,
P. Primet
CC-IN2P3, LIP ENS-Lyon
2008-06-30
Outline
Bulk Data Transfer Service
 A framework for Service Differentiation in
Grid

Motivations

Data transmission in Grid


Large volume, Long term
Between countable sites




Over private shared network
End-to-End quality of service.



Between storage systems
Between computation nodes and local storage system
Strict deadline
High reliability
Best-Effort with TCP



“Fair sharing” VS. Throughput intensive
ASAP VS. Strict deadline
Burst traffic
BDTS

Bulk Data Transfer Service
A centralized bandwidth allocation system
 Provides deadline concerned data
movement
 End-to-End traffic control
 Smooth traffic
 Optimizes link utilization

BDTS Architecture



User Interface
 Submit Transfer Job (t-job)
Network IS
 Provide network
information (topology,
bandwidth, etc.)
Job Management
Optimization (BDTSh)
Flow control (FLOC)
 Control the actual transfer
according to the profile
from JM


Key Concepts

Network Model


Static layer 2 topology
and characters
T-job



Volume: application
layer data size
Time: Start time, End
time
Source Destination pair



Path from NIS
Max-rate: Limitation
due to end systems
Profile

Time-rate: layer 2 rate.
Optimization

Minimize the Congestion Factor (allocated-throughput
/ Link-bandwidth)


Guarantee full usage of link bandwidth
Reduce the average package delay experienced by the
coexisting interactive traffic
B. Chen and P. Primet. Schedulling deadline-constrained bulk data transfer to
minimize network congestion. In CCGRID’07, Pages 410-417. May 2007
Flow Control

TCP + Precise Software Pacer (PSPacer)
 PSP

A module for iproute2

Precise network bandwidth control
Traffic Shaper


TCP with congestion avoid mechanism disabled


Optimized in restramssion and memory management
Collaborating with existing TCP based application
The Estimation


No Ideal Transport Protocol: Layer 2 profile VS. Layer 7 Date
Transfer
 Protocol overhead (L7 protocol, TCP/IP header)
 Synchronization between the profile and implementation
 TCP’s re-act to the network
 …
Linear estimation: Vs = a * Vr + b
Collaboration
The Project
Job Management (Java, c++)
 User Interface (Java)
 Floc / API (C, Linux Kernel, PSP)
 Gridftp client (c++, globus)

Testbed
On Grid5000

GridFTP
Security
 File transfer

Validation of System (Resource
allocation)
Validation of System (Stream
Isolation)
IGTMD

Long latency


Average 106ms
Background Traffic
Conclusions

Conclusions



Bandwidth resource management based on End-toEnd traffic control
Provide a deadline file transfer based on IP
Open problems



The effect of arriving time of t-jobs to the
optimization
The efficiency of Flow control
The deployment of BDTS and network resource
management in Grid
Outline
Bulk Data Transfer Service
 A framework for the Service
Differentiation in Grid

Grid-Managed Network Resources

Why the network (bandwidth/Qos) need to be
managed in a Grid running on a dedicated private
Giga/Tera network?



More bandwidth, more bandwidth consuming applications
each member will have its own demand into the network.
Grid network

Applications sharing common infrastructure



Middle-ware service
Hosts, storage systems, networks,
Network is transparent for Applications

Introduce differentiation of service to application level objects?
How?

Static



Different quality of
service is provided by
different instances
Applications make a
choice
Dynamic



Service identify
application
Assign SLA to instance
of applications
Knowledge of both its
users and its underlying
service
BDTS and GMNR
A “ulimit” like network resource
management interface

Adapt networking policy to traffic according to the process it belongs to

Identify






Policy




Traffics from a process and its children
Traffics from a user
Traffics from a VO
Traffics from a certain application
Traffics to certain destinations
Output Bandwidth
Diffserv QoS
…
Running time adjusting

Inner adjusting:


Outer adjusting


Grid-Job Wrapper: Deployment-User application-Uploading Result
Wrapper
Grid network Resource Management
Thank you!

Comments and Questions?