Transcript Persistency Components in Gaudi - Seal

Status of SEAL Project
SC2 Meeting
9 April 2003
P. Mato / CERN
Contents
Overview
 Organization
 Resources
 Work Packages
 Current Problems
 Summary

9 April 2003
SEAL Project Status
P. Mato/CERN
2
SEAL Overview

SEAL aims to
–
–
–

Provide the software infrastructure, basic frameworks, libraries
and tools that are common among the LHC experiments
Select, integrate, develop and support foundation and utility class
libraries
Develop a coherent set of basic framework services to facilitate
the integration of LCG and non - LCG software
Scope
–
–
Foundation Class Libraries
» Basic types (STL, Boost, CLHEP, …), utility libraries, system
isolation libraries, domain specific foundation libraries
Basic Framework Services
» Component model, reflection, plugin management, incident
(event) management, distributed computing, grid, scripting
9 April 2003
SEAL Project Status
P. Mato/CERN
3
Project Organization
Foundation and Utility Libraries
Math Libraries
Component Model and Plug-in
Manager
LCG Object Dictionary
New work package added to the
project by incorporating the
MathLib project (F. James et al.)
This work package will probably be
split between “Foundation and
Utility Libraries” and “Basic
Framework services” later
Basic Framework Services
Scripting Services
Grid Services
Education and Documentation
9 April 2003
Not yet started working in this
area
SEAL Project Status
P. Mato/CERN
4
Resources
Work Package
Names
FTE
Foundation and Utility Libraries
Lassi Tuura (25), Lorenzo Moneta(50), Massimo
Marino (25)
1.0
Math Libraries
Fred James(50), Matthias Winkler (100) + CAT
1.5
Component Model and Plug-in
Manager
Lassi Tuura (25), Massimo Marino (25), Radovan
Chytracek (30)
0.8
LCG Object Dictionary
Stefan Roiser(50), Christian Arnault(20); RD
Schaffer(10); Zhen Xie(10), Pere Mato(20)
1.1
Basic Framework Services
Alain Bazan(50), Thierry Bouedo(50)
1.0
Scripting Services
Jacek Generowicz(60), Pere Mato(20), Wim
Lavrijsen (20)
1.0
Jacek Generowicz(20)
0.2
Grid Services
Education and Documentation
6.6
9 April 2003
SEAL Project Status
P. Mato/CERN
5
Project Overview: Planned Schedule
Initial work plan being presented to SC2 on
January 10th including detail contents of version
v1 alpha
 March 2003 - V1 alpha

– Essential elements with sufficient functionality for the
other existing LCG projects (POOL, …)
– Frequent internal releases (monthly?)

June 2003 - V1 beta
– Complete list of the currently proposed elements with
sufficient functionality to be adopted by experiments
9 April 2003
SEAL Project Status
P. Mato/CERN
6
SEAL Versions Road Map
Release
Date
Status
Description (goals)
V 0.1.0
14/02/03
internal
Establish
V 0.2.0
31/03/03
public
Essential
V 0.3.0
16/05/03
internal
Improve
V 1.0.0
30/06/03
public
Essential
9 April 2003
dependency between POOL and SEAL
Dictionary generation from header files
functionality sufficient for the other existing
LCG projects (POOL)
Foundation library, system abstraction, etc.
Plugin management
functionality required by POOL
Basic framework base classes
functionality sufficient to be adopted by
experiments
Collection of basic framework services
Scripting support
SEAL Project Status
P. Mato/CERN
7
Work Packages
1.
2.
3.
4.
5.
6.
7.
8.
Foundation and Utility libraries
Math Libraries
Component Model and Plug-in Manager
LCG Object Dictionary
Basic Framework Services
Scripting Services
Grid Services
Education and Documentation
9 April 2003
SEAL Project Status
P. Mato/CERN
8
1. Foundation and Utility libraries

Inventory of existing utility classes
– Ongoing activity. Results in the SEAL Web.

Support for Boost library
– Installation, …

Participation to CLHEP project
– CLHEP workshop Jan 27-31 at CERN
– Proposal from LCG/SEAL+SPI to help in the maintenance and
infrastructure
– LCG participation (A. Pfeiffer, E. Tcherniaev)
» Savannah project portal (operational)
» CVS repository (ready to import FNAL work items)
» Working on template version of the Geometry classes and new
version of Evaluator package
9 April 2003
SEAL Project Status
P. Mato/CERN
9
1. Foundation and Utility libraries (2)

Imported classlib into Imports/classlib
– A set of foundation and utility classes mainly used in Iguana (CMS)
– Not yet fully incorporated as a proper SEAL library. Work ongoing.

Development of SealUtil and SealKernel packages
– The idea is to develop SEAL utility and system library
complementary to Boost and STL from existing code in classLib,
Gaudi, HepUtilities, etc.
– Prioritized by needs of POOL and other SEAL services
– Classes already available: Exception, Status, Service, Timer,
SimpleTime, FileName, SharedLibrary, Callback, …

Guidelines for selecting external libraries
– Not yet a proposal. Working on it.
9 April 2003
SEAL Project Status
P. Mato/CERN
10
2. Math Libraries

GSL evaluation by CAT, Indore
– Project is now very advanced
– Studied needed functionality by LHC experiments by looking what
NAG routines are being used today
– CAT team has made some implementations of functionality that
could be found already in GSL
– Next steps:
» Contact GSL team to incorporate new developed functionality to
GSL and find out their current and future plans
» Automatic testing of GSL (with cppUnit) as part of every SEAL
release

Provide to experiments with math and statistics libraries
to be used in analysis, reconstruction, simulation.
– GSL support, …
9 April 2003
SEAL Project Status
P. Mato/CERN
11
2. Math Libraries (2)

Minuit
– Prototype available in current SEAL release (0.2.0)
– Migrad and Minos available (MathLib/Minuit)
» The numerical results of the two prototypes compared to the
Fortran version. Compatible within the errors.
» No systematic testing done, neither for numerical accuracy nor
performance.
» Aim of these prototypes is to get an overall view of the
required functionality, as well as feedback on the C++ API.

Other libraries
– MathLibs/GSLAlgebra. Prototype wrapper around GSL. Not
complete but restricted to Minuit needs.
9 April 2003
SEAL Project Status
P. Mato/CERN
12
3. Component Model and Plug-in Manager

Plugin Management
– Service in charge of managing, querying, [un]loading plug-ins

Implementation in two levels
– Lower level (no framework constraints)
» Ability to create “module” libraries that contain “plugins”
» Ability to instantiate concrete implementations knowing the
plugin category and the concrete type
» Caching information about what modules contain what plugins
and their categories
– Plugin instances management (assuming a given framework model)
» Locate other plugins and manage their lifetime
» Framework base classes to obey plugin/framework model
9 April 2003
SEAL Project Status
P. Mato/CERN
13
3. Component Model and Plug-in Manager (2)

Available in current release (0.2.0)
– Low level plugin management (Foundation/PluginManager)
– Originated from CMS/Iguana
PluginManager
PluginCache
*
PluginFactoryBase
PluginFactory
XXXFactory
ModuleBase
T
*
PluginInfo
*
XXXInfo
Module
ModuleDef
MyModuleDef
Module Library
Client
9 April 2003
SEAL Project Status
P. Mato/CERN
14
4. Object Dictionary

Reflection packages
– The Dictionary/Reflection package provides the capability to
introspect and interact with any C++ object at run-time
– The Dictionary/ReflectionBuilder package is the “write” interface
of the reflection information. Typically generated code uses this
interface to built the reflection information at run-time

Dictionary generation
– Main goal: Full support of C++ without any class instrumentation
– DictionaryGenerator package provides the command to do it

Experiment examples
– CMSExamples to test the CMS event model (PSimHit)
– ATLAS is testing their event model (GuineaEvent)
– No problem found so far
9 April 2003
SEAL Project Status
P. Mato/CERN
15
4. Object Dictionary (2)

Dictionary generation from header files
– Uses gccxml (0.4.1) to parse header-files (extension to gcccompiler), generates intermediate XML file with dictionary
information
– XML file parsed by a python script which generates dictionary
building C++ code (for “selected classes”)
– Compiled to shared library and loaded at run-time to create
dictionary in memory
.xml selection file
.xml
.h
gccxml
.h
.h
.h
(python script)
par
ser
filter gendict
+
exten
_dict.cpp
make
.so
lcgdict
#include files
9 April 2003
SEAL Project Status
P. Mato/CERN
16
LCG Object Dictionary: Usage
.adl
.xml
.h
Population
ROOTCINT
GCC-XML
CINT generated
code
Dict generating
code
ROOT I/O
in
(1)
CINT
Dict
LCG to CINT
Dict gateway
Conversion
Streamer
ADL/GOD
LCG
Dictionary
(2)
out
Other
Clients:
(python,
GUI, etc.)
Reflection
9 April 2003
SEAL Project Status
P. Mato/CERN
17
4. Object Dictionary (3)

Object Dictionary being used by POOL
– RootStorageSvc from POOL uses the LCG Object Dictionary to
populate ROOT/CINT dictionary
– Examples to write/read complex C++ object models involving many
levels of STL containers exist using the DictionaryGenerator from
header files

Common dictionaries
– Started to produce the dictionary for CLHEP
» Useful to share between experiments events models
» Not yet released.
9 April 2003
SEAL Project Status
P. Mato/CERN
18
5. Basic Framework Services

Developing the basic framework
– Component model (identification, lifetime strategy, interfaces,
etc.)
– Started to review existing designs

Developing basic services
– Started design discussions for message reporting service
– Other services like component configuration, “event” management,
object white board, etc. will continue soon after

Nothing released yet
– Expected rapid progress
– Basic framework and few basic services by release 0.3.0 (mid-May)
9 April 2003
SEAL Project Status
P. Mato/CERN
19
6. Scripting Services

Define guidelines for developing Python bindings
– Evaluate existing options: SWIG, Boost.Python, SIP, and raw
Python C-API
– Study how each technology handles a number of predefined cases
» Examples: Method overloading, inheritance across boundaries,
templated classes and methods, natural mapping of Python
constructs, etc.
– Study interoperability between binding technologies
– Produce an evaluation report. Expected by mid-May

Failed to develop GSL bindings using raw Python C-API
– Selected as an example for automating the generation of Python
bindings
– To achieve a “natural” Python binding requires manual intervention
9 April 2003
SEAL Project Status
P. Mato/CERN
20
6. Scripting Services (2)

PyROOT: Python bindings for ROOT (former RootPython)
– PyROOT is a Python extension module that allows the user to
interact with any ROOT class
» Done generically using the ROOT dictionary. No need to
generate any Python wrapper code to include new ROOT classes
– Upgrading to version 2 of Boost.Python required a complete rewrite (~ 6 classes)
– Many added features with respect previous version
– Released in SEAL 0.2.0

Started to work on the LCG Dictionary bindings
– Basically repeating the work of PyROOT using the LCG dictionary
instead
9 April 2003
SEAL Project Status
P. Mato/CERN
21
8. Education/Documentation

Main activities and tasks
– Produce documentation
– Produce training material (tutorials)
– Help incorporating SEAL components into LCG projects and
experiment frameworks

Existing documentation
– Very limited for the time being
– Produced a number of HowTo pages for the released elements
» Plugin Manager, Dictionary generation, PyROOT, etc.

Helping POOL to incorporate released components
– We have a team member working for both POOL and SEAL

Will be preparing tutorials for June release
– Time of possible adoption by experiments
9 April 2003
SEAL Project Status
P. Mato/CERN
22
Software Process

SPI provided infrastructure

Efforts to “standardize” tool usage and conventions

Release procedures

Platforms
– Savannah Web portal, CVS repository, development tools (SCRAM,
Doxygen, CppUnit, etc.) and services (external tool repository, etc.)
– Slowly getting the good dynamics and practices
– Avoiding divergences between LCG projects
– Adaptation of tools to “LCG proper” working models
– Developing SEAL proper procedures
– Tagging, building, testing, releasing, documentation generation,
announcing, etc.
– Automation of the process is essential (rotating release manager
role)
– Single release platform for the moment: Linux RedHat 7.3/gcc-3.2
9 April 2003
SEAL Project Status
P. Mato/CERN
23
Current Problems


I personally was expecting faster progress
What has slow down us?
– Team dynamics. It takes time to develop trust relationships.
Members not working 100%. Absences of key team members.
– Not obvious how to combine various existing designs (lengthy
discussions)
– Development of proper working models (mixture of the current
practices in various experiments)
– Inflexibility of the build system (SCRAM)
– …

Applied some level of de-scoping
– Some shortcuts applied: classlib in Imports/classlib without the
agreed refurbishing
– Not provided all the necessary test programs, documentation, etc.
9 April 2003
SEAL Project Status
P. Mato/CERN
24
What is not a Problem

No technical problems found
– Very pleased that gccxml works well for the generation of Object
Dictionary. Good response from development team.
– The quality of external packages such as Boost are pretty good
– New version of Boost.Pyhton is a major step forward

Resources available
– Sufficient resources available. The problem, if at all, is to keep
everybody busy
– Adequate staff skills

Relations between LCG projects
– Good working relations
– Adequate level of flexibility
9 April 2003
SEAL Project Status
P. Mato/CERN
25
Main Milestones




2002/10/30Establish core libraries and services (SEAL) project
2002/11/30 Define the V1 SEAL software
2002/12/1 Prototype object dictionary service
2003/1/15 Establish external software decision process
– Establish the process and policies by which decisions are made on what
external software is to be used by the LCG applications area.

2003/1/31 Complete the initial SEAL work plan
– Complete the initial SEAL work plan for submission to the SC2. Should
cover (at least) the content and implementation plan for SEAL V1.

2003/3/31 SEAL V1 essentials in alpha
– The most essential elements of the V1 SEAL suite (as requested by
projects needing to use them) are available in alpha.

2003/5/31 Grid enabled services defined
– The SEAL services which must be grid-enabled are defined and their
implementation prioritized.
9 April 2003
SEAL Project Status
P. Mato/CERN
26
Main Milestones Proposed Changes

Proposed changes
– Establish external software decision process
» Establish the process and policies by which decisions are made on what
external software is to be used by the LCG applications area.
 I would keep it but re-schedule it to 2003/05/15
– Grid enabled services defined
» The SEAL services which must be grid-enabled are defined and their
implementation prioritized
 I would just wait to schedule it after the proposed PI Grid RTAG

New Milestone
– 2003/06/30 SEAL version 1.0.0
» Essential elements of the SEAL suite usable not only by others LCG
projects but also directly by experiments frameworks with
documentation
9 April 2003
SEAL Project Status
P. Mato/CERN
27
Summary

SEAL has had three releases: 0.1.0, 0.1.1, 0.2.0 since the beginning of
the year at the scheduled times
– The main emphasis has been to support POOL (Dictionary, Plugin
Management, Foundation classes, etc.)

The functionality available is not enormous but its development has
help us
– To get experience on new tools and procedures
– Build the development team



No technical problem encountered
Sufficient level of resources
Next scheduled releases
– 0.3.0 by mid-May, 1.0.0 by end-June
9 April 2003
SEAL Project Status
P. Mato/CERN
28