Transcript DDMInstallation_v3 - Indico

OSG DDM Installation
Patrick McGuigan,Horst Severini
mcguigan@uta,[email protected]
USATLAS DDM Ops & Prod.
Workshop
1
Outline
Background
DQ2 + dependencies
LRC + dependencies
Using pacman
Issues
Discussion
USATLAS DDM Ops & Prod.
Workshop
2
Background
Use Pacman to manage installation

http://physics.bu.edu/pacman/
Developed by Horst for DQ2 v0.1.4
Additional work done by Patrick to install DQ2
v0.2.11+
Package is named DQ2-PandaSite

Documented at:
https://twiki.cern.ch/twiki/bin/view/Atlas/PandaDataService

Later slides give installation details
Installs DQ2 Site services + MySQL based LRC
USATLAS DDM Ops & Prod.
Workshop
3
DQ2 Site Services
Provides the infrastructure to
identify/schedule/monitor dataset
movement among ATLAS sites
Does not provide mapping service
Do you know the difference?
We currently support 0.2.11+, but may be
supporting 0.2.12 (accidentally)
USATLAS DDM Ops & Prod.
Workshop
4
DQ2 Site Services Dependencies
Direct Dependiencies
Indirect Dependiencies
Managed by “borrowed” scripts
Managed by pacman scripts
MySQL 4.0.24
Python 2.3.4,
Curl,
Python modules:
pycurl
Mysql
FormEncode
SQLObject
setup-tools (not managed)
USATLAS DDM Ops & Prod.
Workshop
FTS-Client
LRC
edg-gridftp-client (US requirement)
Globus
Myproxy client
5
DQ2 Site Service Issues
The default mechanism for new releases
may overwrite local customizations
Autoupdate of TOACache.py (tends to be
realistically disabled in US installations)
USATLAS DDM Ops & Prod.
Workshop
6
Local Replica Catalog
Service provides mapping from LFN/GUID
to PFN
Not part of DQ2, per se.
Used by DQ2 to determine if files in a
dataset already exist
Used by production system (pilots) to get a
physical location for a given LFN
Used by production system to generate a
new entry when storing output file
USATLAS DDM Ops & Prod.
Workshop
7
LRC Dependencies
MySQL 4.0.24
Python 2.3.4
Apache 2.0.54
apache modules:


GridSite (not used) 1.0.2
mod_py v3.0.4
POOL v2_1_2
HTTP Scripts (not versioned)
USATLAS DDM Ops & Prod.
Workshop
8
LRC Issues
Security is an issue in the current install

Will be addressed in packaging.
Desire to move to new version of MySQL


Longer PFN’s
GSI Enabled Access
Automating backups
USATLAS DDM Ops & Prod.
Workshop
9
Installation
Pre-install checklist





Identify transfer host (gridftp/srm server)
Ask for entry in TOA
Ask for BNL FTS Channel
Suggested: create a local account for service
Identify User Certificate to be used (CERN VOMS)
Get Pacman:

wget http://physics.bu.edu/pacman/sample_cache/tarballs/pacman-latest.tar.gz
Setup Pacman
Create directory
Get Package

pacman -trust-all-caches -v shell-out -get OU:DQ2-PandaSite
Wait for ~ 1hour
USATLAS DDM Ops & Prod.
Workshop
10
Details of Installation Layout
$DQ2_LOCATION
post_install/
(BASE DIR)
agents.cron
(crontab file)
(setup most packages)
(setup for http server)
(local customizations)
(agents log file)
(FTS log)
/setup.sh
/setup_httpd.sh
config/SITE
config/SITE/subscriptions.log
config/SITE/progressFTS.log
/httpd/
mods/
(gridSite only)
htdocs/lrc/???
(HTTP interface to LRC)
/externals/
(MySQL/PYTHON from CERN)
Python/2.3.4/slc3_ia32_gcc323/lib/python2.3/site-packages
mysql/4.0.24/slc3_ia32_gcc323/
mysql.sh
(init.d script)
data/queued_transfers_UTA_SWT2
data/localreplicas
/fts-cli
(FTS CLIENT)
/fts-cli.sh
(needed for glite-*)
/pool
(POOL)
VDT-Directories
11
Firewall/Security
TCP Port 8000 (HTTP port for LRC scripts)



open to your worker nodes
open to Panda server(s)
open to Assitants (BNL, others?)
TCP Port 3306 (MySQL port)

open to everyone
MySQL accounts



root (should be changed from no password in post_isntall
instructions)
dq2 (created during DQ2 site services installation has
equivalent to root privileges)
Need to update installation for one reader for both DB's and a
writer for each.
USATLAS DDM Ops & Prod.
Workshop
12
V&V Strategy
1. Create several well known, small,
permanent test datasets
2. Automate subscription/monitoring for
results?
3. Delete entries (files,LRC entries)
before/after testing
4. Additional testing required for
unrestrained delivery?
USATLAS DDM Ops & Prod.
Workshop
13
Issue with Pacman installation
Size of the install (1+GB) pool and VDT are both large packages
Supporting more platforms (SLC4 X86_64)
Age of components
Removing unneeded components
Software resides in many locations
Unkown error lately with mysql installations (need to replicate)
Need better testing/release strategy
Automate mysql user accounts
Automate mysql backup strategy
Adding new components for unrestrained transfers (Dan’s Proposal)
Add DQ2 client commands to installations
USATLAS DDM Ops & Prod.
Workshop
14
Discussion
USATLAS DDM Ops & Prod.
Workshop
15