Transcript Data Set - Washington University in St. Louis

SHARING DATA USING THE
STORAGE RESOURCE BROKER (SRB)
Ken Wong
The Applied Research Laboratory
(ARL)
and The Department of Computer
Science
Washington University in St. Louis
[email protected],
http://www.arl.wustl.edu/~kenw
Wong
4/10/2016
1
OUTLINE OF TALK





Wong
SRB and HPSS Overview
SRB Concepts and Examples
Alternatives to SRB
Other SRB Projects
Our Experience
4/10/2016
2
WU DATA CACHE AND THE SRB
vBNS
45 Mbps
ATM
155 Mbps
622 Mbps
ghidora
(MCAT)
Wong
hpss
brainmap
v1 petsun-23, 24
(Archives) (1.36 TB) (SUMS) (scanners)
4/10/2016
3
WU DATA CACHE

1.4 TB DEC Storage Works RAID (Level 5)
– 2-processor Sun Enterprise 450, 1 GB main memory
– 622 Mbps ATM interface, 10/100 Mbps Ethernet
interface
– 1.7 TB (raw) = 48 x 9 + 24 x 18 + 24 x 36 GB

Backups
– Incremental: Tue, Wed, Thu
– Full: Mon, Fri, Sat

Wong
Data Volume
– Used: 560 GB
– Burn Rate: 7.0 GB/week (This Year); 5.5 GB/week
(Lifetime)
4/10/2016
4
INSTALLATION HISTORY

Jun/Jul 98: Sun host and then 432 GB RAID
– 3 year extended warranty and 3 year maintenance on
controllers


Sep 98: SRB
Aug 99: 24 x18.2 GB disks
– 3 year maintenance upgrade on controllers

Wong
Dec 99: 24 x 36.4 GB disks
4/10/2016
5
Gigabytes
BRAINMAP DATA GROWTH
1400
1200
1000
800
600
400
200
0
Capacity
Usage
0 10 20 30 40 50 60 70 80 90
Week
Wong
4/10/2016
6
Gigabytes
BRAINMAP DISK USAGE
40
35
30
25
20
15
10
5
0
File System
Wong
4/10/2016
7
STORAGE RESOURCE BROKER (SRB)
Application
(SRB Client)
MCAT
SRB Server
DB2, Oracle, Illustra, ObjectStore
HPSS, UniTree
Unix, ftp
Distributed Storage Resources
Wong
4/10/2016
8
HIGH-PERFORMANCE STORAGE SYSTEM
Network
IBM SP Front End:
8 Servers (disk and tape movers)
1 Server (HiPPi mover)
OC12-ATM, HiPPi, SP Switch
3 Silos
0.5 Terabyte Disk Subsystem
Wong
8 KB Chunk
330 Terabyte Tape Library
4/10/2016
9
HIGH-PERFORMANCE STORAGE SYSTEM

Current Usage
– 150 TB (terabytes; trillion)
– 15 million files


Wong
Current Capacity: 500 TBs of data (assuming a
compression ratio of 1.5)
Projected Capacity: 1 PB (10^15) within a year
4/10/2016
10
SRB CONCEPTS


SRB Server: Responds to SRB requests from
clients
MCAT (Metadata Catalogue)
– Information about data sets and collections (Oracle
DB)


SRB Client
SRB Resource: A logical storage resource
– Example: HPSS storage and container cache


Wong

Data Set: A file registered with the SRB
Collection: Group of registered data
sets/collections
Container: Data sets 4/10/2016
stored as one physical unit
11
SRB SYSTEM CAPABILITIES





Collection-based management of data sets
Persistent identifiers for data sets
Management of data sets (copies or replicas)
Containers for aggregating data sets before
archiving
Support for grid security infrastructure
authentication
– Uses public key certificates

Wong
Support for integrating data set collections
across file systems, archives, and databases
4/10/2016
12
SRB INTERFACES

Scommands (Unix commands)
– Sinit/Sexit, Sput/Sget, Smkdir/Srmdir, Sls/Srm
– Smkcont/Ssyncont, Slscont/Srmcont
– SgetR/SgetU/SgetD


Wong
C-Programming API
Browser
4/10/2016
13
PUBLISHING A DATA SET

Define the SRB environment (.srb/.MdasEnv
file)
mdasCollectionHome ‘/home/kenw.neurodb’
mdasDomainHome ‘neurodb’
srbUser ‘kenw’
srbHost ‘ghidorah.sdsc.edu’
defaultResource ‘cont-sdsc’

Interact with SRB server
%Sinit
# Connect to SRB server
%sls
# See what is in my collection
%Sput ./mydata brain043 # Copy file to SRB space
%Schmod r public npaci brain043
# Give read access
%SgetD -a brain043
# Check access permissions
%Sexit
# disconnect from SRB server
Wong
4/10/2016
14
GETTING A DATA SET
(SCOMMANDS)
%
%
%
%
Sinit
Scd /home/colin.neurodb # go to Colin's collection
Sls -l
# see what is there
Sget colin_avg20_1.0mm_at0.5mm.mnc .
# copy to this directory
% Sexit
Wong
4/10/2016
15
JINGHUA ZHOU'S WORK

Experiments
– Test SRB functionality
– Measures performance of basic SRB functions

Archiving (Perl Scripts)
– Archive an arbitrary Unix directory to HPSS
– Verify files were archived
– Recover files from archival storage
Wong
4/10/2016
16
RETRIEVAL EXPERIMENTS





Wong
Load 100 MB container with 1 MB files
Measure time required to retrieve N files
Divide time by N to get average time for each
file
Repeat after container has been moved to tape
Repeat above steps for 10 MB container (instead
of 100 MB)
4/10/2016
17
Seconds
AVERAGE RETRIEVAL TIME (OLD FILES)
180
160
140
120
100
80
60
40
20
0
10MB Container
100MB Container
0
Wong
5
10 15 20 25
Number of 1 MB Files
4/10/2016
30
18
Seconds
AVERAGE RETRIEVAL TIME (FRESH FILES)
70
60
50
40
30
20
10
0
10MB Container
100MB Container
0
Wong
5
10 15 20 25 30 35 40
Number of 1 MB Files
4/10/2016
19
COMMENTS

SRB Overhead Per Object (File)
– 5-7 seconds (Early Measurements)
– 2-4 seconds (Recent Measurements)


Tape Overhead Per Object (File): 100 seconds
TCP Connection Needs Tuning
–
–
–
–
–
Wong
Assymetric routing, bottleneck, ...
snoop and tcptrace analysis
Max Sget effective bandwidth is 8 Mbps
Max Sput effective bandwidth is 4 Mbps
Goal is 32 Mbps
4/10/2016
20
ARCHIVING





Wong
Reflect Unix directory structure in SRB
collection structure
archiver NPACI/Unix account
Look for inactive files within a directory
Multiple versions handled by appending
modification date to file name
Log all archival requests
4/10/2016
21
CURRENT WORK


TCP Tuning and SRB 1.1.7 Performance
Enhance Archival Scripts
– Improve usability
– Resilience to HPSS Blackouts
– Parallel Archiving
Wong
4/10/2016
22
RECENT SRB DEVELOPMENTS


Data Cutter
GSI authentication
– UsesX.509 certificates

Container redesign
– To handle multiple archival and cache resources


Wong
Remote proxy (Spcommand)
Textual annotation stored in MCAT
4/10/2016
23
ALTERNATIVES TO SRB

Distributed Database
– Do not deal with file data => Requires other means of accessing
files
– A heavyweight solution; i.e., expense (money, expertise)
– Need instances running wherever you want to have storage
– If it is only meta-data, then a case can be made but ...
• Tied to a particular vendor at all sites
• Have to cross link all the databases

AFS (Andrew File System)
– Doesn't have concept of application metadata
• SRB has some metadata facilities now and more to come
• Comments, annotations, user-controlled metadata
– SRB provides a uniform authentication and authorization
Wong
system
4/10/2016
24
TOP SRB PROJECTS (SUMMARY)

2-Micron All Sky Survey
– 10 TB of data from Caltech
– 5 million images sorted into 130,000 containers

Digital Embryo Project (NLM funded)
– Digitizing existing slides for storage in HPSS

Particle Physics Data Grid (DOE funded)
– Data mining


Information Power Grid (NASA funded)
Data Visualization Corridor (DOE funded)
– Handles terabyte sized data sets for interactive
viewing

Wong
Neuroscience Data Set
Federation
4/10/2016
25
TOP SRB PROJECTS

2-Micron All Sky Survey (2MASS)
– 10 TB of data from Caltech (3 TB done)
– 5 million images sorted into 130,000 containers
– SRB container technology used to manage the aggregation
process on a disk cache
– Replicate Caltech data

Digital Embryo Project (NLM funded)
– Digitizing existing slides for storage in HPSS
– SRB used to manage data movement, aggregation into
containers, and metadata catalog
– Queries against the collection

Particle Physics Data Grid (DOE funded)
– Replicate data sets that are pulled into local disk caches
Wong
4/10/2016
26
TOP SRB PROJECTS

Information Power Grid (NASA funded)
– SRB used to support data mining against a distributed data set
collection
– Data transmission rate: 58 Mbps from SDSC to NASA Ames
– Put collection management in front of storage archives through
use of the MCAT

Data Visualization Corridor (DOE funded)
– SRB has been integrated with the Data Cutter system
• For remote manipulation of data sets
– Handles terabyte sized data sets for interactive viewing

Wong
Neuroscience Data Set Federation
4/10/2016
27
CONCLUDING REMARKS

Documentation
– http://www.sdsc.edu/DICE/SRB/index.html
– http://www.arl.wustl.edu/kenw/npaci/index.html

Software
– Follow SRB link
– Get PGP key from SDSC
– Can install subset (e.g., client only)

Wong
Applications?
4/10/2016
28
Wong
4/10/2016
29
WU DATA CACHE
vBNS
45 Mbps
sdsc.edu
ghidorah
(MCAT)
hpss
ATM
wustl.edu
622 Mbps
155 Mbps
brainmap stp, v1 petsun-23
(1.3 TB) (SUMS) (Scanners)
(12 Major Users)
UCSD, UCLA, John Hopkins, U. Montana, Caltech
Wong
4/10/2016
30