Transcript CM11 - Friedrich Lehn | Consultant, Coach & Trainer

Release Management
for the UBS Data Warehouse Project
Friedrich Lehn
UBS AG, Switzerland
[email protected]
Session CM11
©1998, 1999, 2000, 2001 Rational Software - All rights reserved
Agenda
 Project Overview
 Project Infrastructure
 Change Management
 Release Process
 Summary
Project Overview
UBS
 Global, integrated investment services firm
and leading bank in Switzerland
 World’s largest private bank
 Total client assets over US$ 1.5 trillion
 Acquired
in 2000
Project Overview
Data Warehouse Program (DWP)
 Establish common infrastructure for analytical data
processing
 Provide a business oriented set of data warehouse
and data mart services
 Standardized business data model
 Align the bank’s data mart portfolio
 Improve flexibility, time-to-market and data quality
Project Overview
Data Flow
System of Records
extract, condition & load
Common Data / Business Warehouse
data mart sourcing
Data Marts
visualization
business user
Project Overview
Delivery Streams
 Main organizational element:
team working on a subject data area / data mart
 Three letter acronym as base for naming standards
 Standardized infrastructure:
UNIX directories, access group, meta data area, ...
 Each delivery stream has business responsible, data
modeler, database administrator, delivery stream manager
 Team size typically between 1 and 5
 Delivery streams release independently from each other
Project Overview
Application Structure
System of Records
RCL
DSF
DSF
MDR
<SDA>
RCL
MDR
DSF
FDS
<SDA>
<MAR>
release control tools
meta data repository
DWP sourcing framework
feed configuration files
subject data area
data mart
...
FDS
<SDA>
Sourcing Framework
<MAR>
...
<MAR>
Project Infrastructure
System Environments
Development
•
•
•
•
•
•
IBM AIX SP2 cluster
DB2 UDB EEE
PowerCenter V1.7
DWP Sourcing Framework
Cognos / Business Objects
ClearCase V3.2
Test
Production
Project Infrastructure
Logical Environments and Release Structure
Development
Test
Production
E
emergency releases
D
A
X
F
T
P
delivery stream development
V
framework test / delivery
stream migration to new
framework releases
framework development
mandatory
optional
Project Infrastructure
Release Cycles
 delivery stream development: D  A  P
 framework development:
after sign-off:
TXFV
XDAP
 migration to new framework
releases:
[DAP]  X  F
 emergency releases:
[DAP]  E  P
Project Infrastructure
Directory Structure
 Two areas:
/dwp_root
/dwp_data
release area, version controlled
dynamic data, archival on demand
 Additional directory level in order to support more than one
logical environment on one system
 /dwp_root is organized by delivery streams, e. g.:
/dwp_root/d/streams/rcl/bin
 ~<user>/dwp_root
user specific development area
Project Infrastructure
Directory Structure (continued)
 /dwp_data is organized by logical processing steps, e. g.:
/dwp_data/p/data/landing
(landing area)
/dwp_data/p/data/tgtfiles
(target files)
/dwp_data/p/logs/system
(framework log area)
 Tool support for generation of directories in source
environments (delta processing)
 Automatic creation of missing directories in target
environments by release procedures
Change Management
Design Principles
 Support different, clearly separated environments with
different responsibilities
 All environments have identical structure (products,
databases, server configurations)
 All program changes are done on the development system
 All changes on test and production systems go through the
release process and are clearly tracked
Change Management
ClearCase Set-up
 One single VOB /vobs/dwp
 Fully automated access layer (freeze and deliver routines)
 Same directory structure as below /dwp_root, directories are
automatically created
 In general: only linear version trees (important:
synchronization with database change management)
 Branch support planned for emergency releases and
migration path only
Change Management
Release Naming
 <delivery stream>_<major>.<minor>.<patch>
e. g.:
RCL_1.1.0
 <major>
major release number
(high level “wave” planning)
 <minor>
minor release number
(delivery stream development plan)
 <patch>
patch level (bug fixes)
 Emergency releases: RCL_1.1.0_sos_<#>
Change Management
Versioning Example
common.pm
/main/1
RCL_1.0.0_sos_1
/main/sos/1
RCL_1.0.0
/main/2
RCL_1.0.1
/main/3
RCL_1.0.2
RLS_P
/main/4
RLS_A
RLS_P
current version in
user acceptance test
current version in production
RCL_2.0.0
/main/5
RLS_A
Release Process
Change Control Board
 Responsible for high level planning and impact analysis
 Defines release scope and release numbers on base
of delivery streams
 Assigns responsibilities (delivery stream manager, data
modeler, database administrator, business responsible)
 Result is documented in “wave plan” document
Release Process
Roles & Responsibilities
Role
Responsibility
developer
development, unit and integration testing
delivery stream manager manager, release planning
database administrator
database change control
release manager
deployment, tracking, configuration control,
administration
Release Process
Process Overview
Development
Test
Freeze
Production
Receive
ClearCase
Deliver
Deployment Package
Release Process
Release Objects
 PowerCenter mappings
 scripts / SQLs
 Job dependency data
 Uniserv jobs (address conversions)
 Database objects (see below)
not included: documentation (intranet database)
Release Process
Database Objects
 tables (regular, summary)
 views (business, security)
 keys (unique, primary, foreign)
 indexes
 triggers
 aliases
 check constraints
not included: table spaces (system dependent)
Release Process
Release Procedure
Responsible
1. Submit database change request *)
delivery stream manager
2. Implement database changes *)
database administrator
3. Prepare release area
(UNIX, PowerCenter, job dependencies, Uniserv)
delivery stream manager
4. Submit release request
delivery stream manager
5. Prepare release area (DDLs) *)
database administrator
6. Create new release (Freeze)
release manager
7. Create deployment package (Deliver)
release manager
8. Apply database changes to target system *)
database administrator
9. Install release in daily deployment window
(Receive)
release manager
(IT integration / IT operation)
*) in case of database changes only
Release Process
Freeze Process
freeze <delivery stream> [ -patch | -minor | -major | -sos <release> ]
1. Retrieve previous release
2. Compare with release area (check for new, changed, deleted files)
3. Display results and ask for confirmation
4. Apply changes in ClearCase
5. Create and attach release label
Release Process
Deliver Process
deliver <delivery stream> -t <target env.> [ -r <release> ] [ -a ]
1. Retrieve specified / latest release in ClearCase
2. Retrieve target environment file versions and create delta
3. Use -a(ll) for initialization / synchronization
4. Create deployment package (tar file + control file)
5. Update target labels
6. Lock release label
Release Process
Receive Process
receive
1. Check hand-over area for pending releases
2. Remote copy deployment package to target system
3. Install it
4. Standardized post-installation steps: e. g. access permissions
5. Delivery stream defined post-installation steps (PostInstall.ksh file):
e. g. for non-standard path names, generators, setuid bits
Release Process
Release
Request (1)
Release Process
Release
Request (2)
Release Process
Release
Request (3)
Release Process
Database Change Management
 PATROL DB-Change Manager by bmcsoftware
 Scope filter: assign database objects to delivery stream
( view in ClearCase, name equal to delivery stream)
 Apply database changes to development database
 Create release baseline:
freeze all database object versions for a delivery stream
( label in ClearCase, name equal to release label)
 Export DDL to release area
(for documentation & change detection)
Release Process
Database Change Management (target system)
 Target baseline: target database version
(delivery stream plus timestamp as name)
 create delta DDL depending on release baseline
and latest target baseline (PATROL)
 apply delta DDL
 create new target baseline
Release Process
Database Retrofitting Process
 Rationale: certain database changes have to be applied
and tested directly on the Production system (load
performance optimization: indexes, summary tables, ...)
 In order to include target system changes into next regular
release, changes are promoted back to Development
using the Retrofitting Process
 In principle: new, database administrator driven release in
ClearCase and PATROL that is not delivered
Release Process
Releasing Meta Data
 PowerCenter and JDM data are stored in the database
 Uniserv jobs are stored in vendor specific directories
 Unload utilities to export the corresponding data
and store it in the delivery stream´s release area (tar file)
 Meta data is frozen and delivered together with all other
file system objects
 Load utility for loading objects in target environment
 e. g.: meta_jdm <delivery stream> [ unload | load ]
Release Process
Release
Database
Release Process
Release Database
(continued)
Summary
Experiences
 Over 1500 releases since May 2000
 Effort for creation and installation of new release:
5 - 60 minutes depending on amount of meta data
(without database changes)
 Main effort necessary for handling of meta data
 No ClearCase problem encountered so far
Summary
Wish List
 Transaction concept for set of cleartool commands:
What happens if the 199th check-in of 200 fails? Roll back?
 Signal handling option for cleartool:
Today it is not possible to ignore SIGINT in cleartool
( manual clean up)
Questions?
Friedrich H. Lehn
[email protected]
www.fhlConsult.com
Thank You!
This presentation will be posted by tomorrow at:
http://www.rational.com/ruc