Transcript Collective Communications

Enhanced Replica Selection
Technique for Binding Replica Sites
in Data Grids
Authors:
Mahdi S. Al-Mhanna,
Rafah M. Almuttairi
University of Babylon, Babylon, Iraq
University of Babylon
Outlines
What is the DataGrid?
Replication Overview.
Replica Optimizer.
What is Association Rules?
New Replica Selection Technique for Binding
Cheapest Replica Sites in Data Grids.
Comparison with traditional methods.
Related Work.
Conclusion.
University of Babylon
What is Data Grid?
University of Babylon
What is Data Grid ?
 An infrastructure that enables replicating the
required data sets in different distributed sites and
then selecting the best replica matching user’s
requirements.
Data Grid job: It contains a set of files that must be
available at the time before or during execution time.
Job 1:
File 1;
File10;
File 11;
File12;
………
University of Babylon
Why do we need it?
Worldwide LHC Computing Grid
The Large Hadron Collider will produce roughly 15
petabytes (15 million gigabytes) of data annually –
enough to fill more than 1.7 million dual-layer DVDs a
year!
1 Megabyte (1MB)
A digital photo
1 Gigabyte (1GB)
= 1000MB
A DVD movie
1 Terabyte (1TB)
= 1000GB
World annual
book production
1 Petabyte (1PB)
= 1000TB
Annual production of
one LHC experiment
1 Exabyte (1EB)
= 1000 PB
World annual
information production
CMS
LHCb
ATLAS
University of Babylon
ALICE
LHC data
Balloon
(30 Km)
CD stack with
1 year LHC data!
(~ 20 Km)
LHC data correspond to about
20 million CDs each year!
Where will the experiments
store all of these data?
Concorde
(15 Km)
Data for LHC: a problem?
The Grid: a possible solution!
University of Babylon
Mt. Blanc
(4.8 Km)
Replication Overview
University of Babylon
Data problem solved by, Storing data in different
hosts.
SVA Compressed Data
University of Babylon
Replication Overview
• Single server will be flooded
when the requests increase.
• Latency increases with long
queue of waited jobs.
Resource Broker and Resource Optimizer
University of Babylon
Replication
Strategies!!
Replication
Strategies!!
…Which file I
should be used
when it is
needed?!!
Optimizatio
n Strategies!!
Resource
Optimizer
University of Babylon
The replica optimiser provides:
A run time selection service to get the best file replica.
User Interface
Job1: f1,f2,...,fz
Job2: f7,f8,...,fz
.
Resource Broker
Job1: f7,f8,...,fz
Job exe. site
Replica Manager
Replica
Optimizer
Jobt: f11,f12,...,fz
Job2: f1,f2,...,fz
Job exe. site
Replica Manager
Replica
Optimizer Get best
sites having:
f1,f2,...,fz
Computing
Element
Storage
Element
Computing
Element
University of Babylon
Storage
Element
Association rules technique
University of Babylon
Association rules: Pincer-Search algorithm
This algorithm is used to combine different
associated sites, having uncongested links at the
time of the file(s) transfer.
Preliminary Concepts
 Association rules: association rules are mostly
used in mining transaction data. Crucial terms in
association rules terminology are:
 Item (in DM terminology corresponds to
attribute-value pair).
 Transaction (a set of items; corresponds to
example).
 A Set (data set) of transactions containing more
different items.
University of Babylon
Efficient Set algorithm
University of Babylon
Get the first Job from job queue
Get all related LFN from job description file
Get the physical adds of replicas’ sites from RLS
Using Ipref, Probing info. are stored in NHD
Calculate the score and update the TABLE
Apply EST to get Ln which is list of associated sites
Use (GridFTP) transfer files from efficient set of sites.
Get the next Job from job queue
Yes
Do we have history
for current job in
NHD?
Figure 4. Execution flows using ESM
No
Experiments
University of Babylon
Study case:
Let us assume
there are eight sites
having a requested
file.
 Ipref, network
service is used
to get RTT report
 RLS, to collect
physical names of
replicas
PRAGMA Grid, 35 institutions in 25 regions
University of Babylon
University of Babylon
1- Network History File NHF/Using Ipref
RTT/STT
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
S1
60
65
303
305
300
313
310
307
310
310
310
307
310
50
316
74
S2
176
211
175
213
210
176
175
216
217
211
175
176
211
175
214
209
S3
202
209
298
203
223
207
298
260
260
224
204
205
227
202
260
206
S4
336
343
338
273
334
335
273
271
342
339
272
271
344
270
336
341
S5
78.7
76.5
64.8
92.6
95
66.7
94.2
94.4
95.2
66.3
92.8
69
92.5
66.3
63.4
94.3
University of Babylon
S6
256
256
258
255
292
298
255
256
289
257
262
256
299
299
296
287
S7
76.4
86.8
57.2
85.1
55
55.3
85.8
60
90.1
90.3
91.4
88.7
64.3
90.5
57.8
56.4
S8
213
202
205
202
212
212
202
212
212
212
202
210
212
216
224
222
1. RTT:
Round
Trip
Time
2. STT:
Single
Trip
Time
3. Unit:
ms
2- Logical History File (LHF)
University of Babylon
Affected Factors
 Factors affect directly on data transfer time.
 Bandwidth
 Latency of links
Factors affect the performance data transfer.
 CPU
 I/O slightly
University of Babylon
Cont…
According to the first three factors, in our
algorithm, a score function is defined as the
following:
Scorei = piBW× wiBW+piCPU× wiCPU + piI/O × wiI/O
Sum = ∑ Scorej , 1 < J < M
AVG = sum / M , M= Number of sites
If scorei ≥ AVG then Scorei=1 otherwise
Scorei= 0
University of Babylon
Cont…
Where,
- Scorei. the replica selection cost model for server i.
- piBW. percentage of Bandwidth available from server i to
the computing site.
- wiBW. network and Bandwidth ratio of replica server i
defined by the administrator of the Data Grid organization
piCPU. percentage CPU idle states of replica server i
- wiCPU. CPU load ratio of server i defined by the
administrator.
- piI/O. percentage of memory free space of server i
- wiI/O. memory free space ratio defined by the administrator.
University of Babylon
Rules
• "The rule":
Rule #1: if Site(s) S4, S7 are selected then this implies that site(s)
S3 can also be selected at the same time. This rule has 100%
confidence.
I(Rule1)= S4, S7  S3
Support(S4,S7,S3)/Support(S4,S7)*Support(S3)= 1.109
The confidence of this rule is 100%. The requested files will be
sent by S3 , S4, S7
To compute the correlation of this rule and see how far it is
better than choosing the site randomly, we use an
Improvement equation.
University of Babylon
EST vs Traditional Model
University of Babylon
EST vs Traditional Model
Computing Site (CS) is (S0)
RS={S14 ,S1, S3 } . Red stars referring to congested routers. Traditional
selection S14 less number of Hops
EST, (S3 ) uncongested links.
Data Grid and theirUniversity
associated
of Babylonnetwork geometry
Comparison with Traditional selection strategy
Traditional selection strategy and EST
University of Babylon
Conclusion
University of Babylon
Conclusion
 Efficient Selection Technique (EST) is a
dynamic replica selection strategy.
 Aims to adapt at run-time its criteria to
flexible QoS binding contracts specified by
the service provider and/or the client.
 The concept of association rules of data
mining approach is used.
 To reduce the searching space, response time
and network resources consumed.
University of Babylon
Related work
 During 2001-2003, Sudharshan et al. In their work they used the history of previous
file transfer information to predict the best site holding a copy of the requested file.
 R. Kavitha, and I. Foster, in 2001 they used Network Bandwidth and link’s latency.
Corina and M. Mesaac in 2003 and Ceryen and M. Kevin 2005 used different
algorithms such as greedy, random, partitioned and weight algorithms in the selection
engine. Their selection methods were with respect to some static metric such as the
geographical distance in miles and the topological distance in number of hops.
 Rashedur et al. in 2005 exploited a replica selection technique with the K-Nearest
Neighbor (KNN) rule used to select the best replica from the information gathered
locally.
 Rashedur et al., In 2008 proposed a Neural Network predictive technique (NN based)
to estimate the transfer time between sites.
 A. Jaradat et al., In 2009, proposed a new approach that utilizes availability, security
and time as selection criteria between different replicas, by adopting k-means clustering
algorithm concepts to create a balanced (best) solution.
 Omer Aljadaan in 2010, used a Genetic algorithm for clustering replicas and get best
replica matching QoS of user’s requirements.
Most of these methods have a limitations and some of them have a
drawback and others work under a specific conditions.
University of Babylon
References
•
•
•
•
•
•
•
•
•
•
•
•
•
M. Rashedur Rahman, Ken Barker, Reda Alhajj, "Replica selection in grid environment:a data-mining
approach", Distributed systems and grid computing (DSGC),pp: 695 – 700 , 2005
J. Gwertzman and M. Seltzer, "The case for geographical push cashing", In Proceeding of the 5th
Workshop on Hot ZTopic in Operating Systems, 1995.
R. Kavitha, I. Foster, "Design and evaluation of replication strategies for a high performance data grid",
in, Proceedings of Computing and High Energy and, Nuclear Physics, 2001.
S. Vazhkudai, J. Schopf, I. Foster, "Predicting the performance of wide-area data transfers", in: 16th
International PDPS, 2002.
S. Vazhkudai, J. Schopf, "Using regression techniques to predict large data transfers", in: Computing:
Infrastructure and Applications, The International Journal of High Performance Computing Applications,
IJHPCA , August, 2003.
A. Abbas, Grid Computing:" A Practical Guide to Technology and APPLICATIONS", 2006.
http://goc.pragma-grid.net/wiki/index.php/UoHyd.
S. Vazhkudai, S Tuecke, I. Foster, "Replica selection in the globus data grid", in: First IEEE/ACM
International Conference on Cluster Computing and the Grid, CCGrid 2001.
J. Guyton and M. Schwartz."Locating nearby copies of replicated internet servers", In Proceeding of ACM
SIGCOMM’ 95, 1995.
A. Tirumala, J. Ferguson, Iperf 1.2 - The TCP/UDP Bandwidth Measurement Tool, 2002.
R. Wolski, Dynamically forecasting network performance using the Network Weather Service, Cluster
Computing (1998).
Yunhong Gu, Robert L. Grossman, "UDT: UDP-based data transfer for high-speed wide area networks",
Computer Networks, Volume 51, Issue 7, 16 May 2007, Pages 1777 1799. Elsevier.
R.M. Rahman, K. Barker, R. Alhajj, "Predicting the performance of GridFTP transfers", in: Proceedings of
IEEE Symposium of Parallel and Distributed Systems, 2004, New Mexico, USA, p. 238a.
University of Babylon
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
J. F. Kurose, K.W. Ross, "Computer Networking A Top-Down Approach Featuring the Internet", 3rd edition.
S. Venugopal, . R. Buyya,"The Gridbus Toolkit for Service Oriented Grid and Utility Computing: An Overview and
Status Report"2004.
R. Agrawal, T. Imielinski, A.Swami, "Mining association rules between sets of items in large databases". In: Proc.
ACM SIGMOD Intl. Conf. Management Data, 1993
R.M Rahman, K Barker and R Alhajj, "Replica selection strategies in data grid", Journal of Parallel and Distributed
Computing, Volume 68, Issue 12, Pages 1561-1574, December 2008.
A. Jaradat, R. Salleh and A. Abid, "Imitating K-Means to Enhance Data Selection", Journal of Applied Sciences 9
(19): 3569-3574, 2009, ISSN 1812-5654, Asian Network for Scientific Information-2009.
S. Venugopal, . R. Buyya, K. Ramamohanarao, "A taxonomy of Data Grids for distributed data sharing,
management, and processing". ACM Comput. Surv. 38, 1 (Jun. 2006), ACM New York, NY, USA
http://www.resample.com/xlminer/help/Index.htm
A. K Pujari, "Data mining techniques", Hyderabad : Universities Press, 2002.
G. Williams, M. Hegland and S. Roberts, "A Data Mining Tutorial", IASTED International Conference on Parallel
and Distributed Computing and Networks (PDCN’98) 14 December 1998.
T. Ceryen, and M. Kevin, 2005. "Performance characterization of decentralized algorithms for replica selection in
dstributed object systems". Proceedngs of 5th International Workshop on Software and Performance, July 11 -14,
Palma, de Mallorca, Spain, pp: 257-262.
F. Corina, and M. Mesaac, 2003, "A scalable replica selection strategy based on flexible contracts". Proceedings
of the 3rd IEEE Workshop on Internet Applications, June 23-24, IEEE Computer Society Washington, DC, USA,
pp: 95-99.
R. M. almuttari, R. Wankar, A. Negi, C.R. Rao. "Intelligent Replica Selection Strategy for Data Grid", In proceeding
of the 10th International conference on Parallel and Distributed Proceeding Techniques and Applications, IEEE
Computer Society Washington, DC, WorldComp2010, GCA2010, LasVegas, USA, Volume3, pp: 95-100, July 1215-2010.
Cisco Distributed Director, http://www.cisco.com/warp/public/cc/pd/cxsr/dd/index.shtml
M. Sayal, Y. Breitbart, P. Scheuermann, R. Vingralek, "Selection algorithms for replicated web servers". In
Proceeding of the Workshop on Internet Server Performance,1998.
E. Zegura, M. Ammar, Z. Fei, and S. Bhattacharjee, "Application-layer anycasting: a server selection architecture
and use in a replicated web service", IEEE/ACM Transactions on Networking, vol. 8, no. 4, pp. 455–466, Aug.
2000.
http://www-wanmon.slac.stanford.edu/cgi-wrap/pingtable.pl?from=DZ.ARN.N3&to=WORLD
University of Babylon
?
????
[email protected]
[email protected]