Transcript CVSreorg

CVS Reorganization,
Installation Reorganization &
A Simple Build System
Steve Fischer
October 24, 2002
Goals

Make GUS portable



Self-explanatory CVS structure



Projects wait for other projects only when necessary
Easily maintain released versions while development continues
Support development on a unified file system (new file server)


Improve collaborators’ ability to understand our code structure
And ours (particularly new folks)
Improved ability to release projects on a schedule



Easily build and configure runnable instances of our projects
Avoid conflicting with the pre-existing structure of external sites
Don’t rely on local file systems having different installed versions
Encourage a move away from individual ownership of projects
CVS Reorganization
CVS Structure

At the root there are projects, which have components, which
have parts:
$CVSROOT/
Project1/
Component1/
bin/
lib/
Component2/
Project2/
Projectn/
parts
Projects

Units of software development having their own release
schedule


May depend on other projects



Have their own release number, eg, GUS 3.1.2, Annotator 1.0.0
Either on a specific release or ‘latest’
No circular dependencies
Dependency example:

DoTS->GUS->CSP
Proposed Projects (phase 1)










AllGenes
Annotator (new)
CSP (CBIL Style Police: web form assistance)
DJob (distributed job controller)
DoTS (build, including DoTS related plugins)
GLE (grammars for transcription)
GUS
ParaDBs
PlasmoDB
RAD (plugins & website) (or should these be separate?)
Projects Contain Components

For example:
GUS/
Common/
DBAdmin
Model/


A component is usually formed from one or a couple of
package directories (eg, java packages).
In addition to code, components have:





Documentation
TODO list
Configuration
Required data
A component may depend on others within its project
Proposed GUS/ Components













SeqMatch/
Common/
DBAdmin/
GenBank/
GOPredictor/
Model/
ObjRelJ/
ObjRelP/
Pipeline/
PluginMgr/
SchemaBrowser/
Util/
WebDevKit/
Proposed GUS Components in Detail - 1

SeqMatch/


Common/


Data model for genbank parser
GOPredictor/


Database admin scripts
GenBank/


Commonly used schema dependent stuff, such as common plugins
DBAdmin/


Blast, BLAST2, and BLAT object models
Predict GO function
Model/


Schema definition files
Hand edited objects (eg, GUS::Model::DoTS::Assembly.pm.man)
Proposed GUS Components in Detail - 2

ObjRelJ/


ObjRelP/


Web based schema browser
Util/


GA and friends
SchemaBrowser/


Pipeline management API
PluginMgr/


Perl object layer: superclasses and code generator
Pipeline/


Java object layer: superclasses
Commonly used schema-independent utilities
WebDevKit/

Servlet and/or JSP based
Components Contain Standard Parts

As needed:
Component1/
bin/
cgi-bin/
config/
data/
doc/
htdocs/
lib/
src/
test/
Component Parts in Detail - 1

Bin


Cgi-bin/


Cgi executables
Config/



executables
Properties files for the component
Files typically named component.prop
Doc/



Specifications
User guides
TODO
Component Parts in Detail - 2

Data/


Htdocs/


Extra data needed by component, eg, matrices,
Static web pages
Test/

Test data
Component Parts in Detail - 3

Lib/


Contains linkable object files
Perl includes full package path (Project::component::module)
lib/
perl/
Project/
component/
*.pm
xml/
*.xml
Component Parts in Detail - 4

Src/

Contains source that needs to be compiled or otherwise
transformed before it can be installed
src/
c/
java/
edu/
cbil/
project/
component/
org/
gusdb/
component/
Process to Migrate CVS Structure
copy
origCVS/
GUS/
perl/
…
www/
perl/
lib/
Fasta.pm
CSP/
orig
fromCVS/
GUS/
perl/
…
www/
perl/
lib/
Transform
script
Fasta.pm
CSP/
pruned
newCVS/
DoTS/
Common/
lib/
perl/
Common/
Fasta.pm
CSP/
GUS/
new
Process to Migrate CVS Structure

Do transform



Check out old to origCVS/
Copy to fromCVS/
Run script to transform from fromCVS/ to newCVS/


moves files (thus pruning fromCVS/)
Validate transform by:


Examing transform script (see where YOUR stuff is going to be)
Examine fromCVS/ to see what I left behind

Create new CVSROOT with newCVS/ (losing history)

How can I get everybody’s OK on the transform??
Releases




A project is the unit of release
When it is released, it is tagged in cvs with a release number,
eg 2.1.1
At that time, other projects can declare a dependency on that
particular release (discussed in detail later)
Bug fixes are applied to the tagged branch
Installation Reorganization
Objectives

Install into a single relocatable location: $GUS_HOME




Be able to find (almost) all GUS installed resources there:








Executables
Libraries
Documentation
Configuration
Some data
Third party resources
Support multiple running instances on a machine, eg dev, beta
Make explicit the versions of each included resource


Don’t conflict with the site’s pre-existing structure
Easy to install and uninstall
Avoid path and classpath conflicts
In a file called $GUS_HOME/versions
Also have installation targets for websites and appl. servers
Installing Project1 to $GUS_HOME
P1/
C1
C2
$GUS_HOME/
Bin/
Lib/
Doc/
P2/
C1
C2
P3/
C1
C2
= dependencies
Understanding $GUS_HOME






The home for a single GUS related installation
Contains one or more projects, and the projects they depend
on
Including third party resources (such as BioJava or BioPerl)
May contain projects that are running separately, as long as
they don’t have conflicting dependencies
(May even contain the entire set of GUS projects)
It contains a version file, eg:
DoTS 1.2.1
GUS 2.0.0
CSP 3.0.1
BioJava 1.1.1
Sample $GUS_HOME
$GUS_HOME/
bin/
dotsbuild
extractSeqs
config/
dotsbuild.prop
csp.prop
gus.prop
data/
matrices/
doc/
DoTS/Dotsbuild/TODO
GUS/Model/uml/*.uml
CSP/UserGuide.html
(cont’d)…
Sample $GUS_HOME (cont’d)
$GUS_HOME/
…
lib/
java/
gusmodel.jar
guswdk.jar
perl/
DoTS/DotsBuild/*.pm
GUS/Model/*.pm
GUS/ObjRelP/*.pm
GUS/WebDevKit/*.pm
CSP/*.pm
xml/
*.xml
Installation Targets - Website

Contains cgi-bin and htdocs
$/world/www.allgenes/
cgi-bin/
*.pl
htdocs/
*.html
Installation Targets – Application Server


EG, Tomcat
Will copy .jar files and other stuff as needed
Configuration






Code never refers to absolute file paths
Many resources are conveniently located relative to
$GUS_HOME, so don’t need configuration
Otherwise, code relies on property files for configuration
We provide a perl and java PropertySet object to make this
easy
Property files are found in… $GUS_HOME/config
Named after the component that is using them:


gusmodel.prop
Consider using properties instead of macro substitutions when
possible
A Simple Ant-based Build System
Objectives








Install projects to installation targets
Serves two kinds of installs:

From $PROJECT_HOME to installation targets (developers)

From .tar file to installation targets (external users)
Support inter-project dependencies
Support dependencies on third party resources
Handle object layer code generation
Be easy to use
Be relatively easy to maintain
Be generic across projects and components, but also flexible
$PROJECT_HOME -> $GUS_HOME (developers)

Install the build system




Check out the project of interest




Create $PROJECT_HOME
Check out the install/ project
Copy build to a local bin/
build DoTS install $GUS_HOME –co
Now $PROJECT_HOME is an image of CVS containing your
project and all projects it depends on.
Edit away
Install for testing

build DoTS install $GUS_HOME
.tar -> $GUS_HOME (external users)




Untar the download file
cd install/
setenv $GUS_HOME /usr/local/somewhere
./build DoTS install $GUS_HOME

Also works for installing to a website

./build GeneDB installweb /world/www.genedb
The Mechanics

Uses Ant (http://jakarta.apache.org/ant/)




Each project contains its own build.xml file
The install/ directory contains the main build.xml file



Uses xml to declaratively specify the build
Used extensively in the java community
Calls the appropriate project’s build.xml
Houses “subroutines”
The project build.xml file



Specifies the project’s dependencies on other projects
Specifies the project’s component’s dependencies on each other
Calls default project and component build routines, unless custom
ones supplied
The Mechanics (cont’d)

Java compilation




Copying from $PROJECT_HOME to target (eg, $GUS_HOME)




The build compiles java into the classes/ directory of a component
It places .jar files into lib/java/ directory of a component
Does compilation in dependency order
Copies and merges all dirs, such as bin/, lib/ doc/
Can perform macro substitution on the way
Creates version file
Because Ant is kind of slow, build can do just a component
instead of a whole project
Special case: object layer code generation

Generates into:

Perl:




Java:




GUS/Model/DoTS/Assembly.pm.man
Only generates if files are older than “schema definition” file:


$PROJECT_HOME/GUS/Model/src/java/org/gusdb/model/dots
$PROJECT_HOME/GUS/Model/src/java/org/gusdb/model/core
Etc
Hand created files have .man suffix, eg:


$PROJECT_HOME/GUS/Model/lib/perl/GUS/Model/DoTS
$PROJECT_HOME/GUS/Model/lib/perl/GUS/Model/Core
Etc
$PROJECT_HOME/GUS/Model/schema/definition.sql
This file is generated by schema modification process (and
lives in CVS)
Making it All Happen
Goals


Getting it done quickly
With a minimum of disruption
Overarching strategy


Work on projects and components one at a time
Start with ones that are most central and not in development











GUS/ObjRelP
GUS/Model/lib/perl/Model/DoTS
GUS/PluginMgr
GUS/Common
DoTS
DJob
Get that core working with new build system and $GUS_HOME
In parallel, convert to GUS 3.0
In parallel, bring in GUS/ObjRelJ and Annotator (Dave)
Upgrade WebDevKit install apparatus
Bring in Web sites one at a time
Inconveniences

During the time that a project is being upgraded:


Everybody will need to commit their work to cvs
Will need to maintain 2 copies in cvs:




Old
New
Will need to merge changes to both
Hopefully this will be short for most projects