Transcript SEAL: Core Libraries and Services Project

SEAL: Core Libraries and Services
Project
CERN/IT After-C5 Meeting
6 June 2003
P. Mato / CERN
Contents
History: The blueprint RTAG
 Project Aims
 Current Work Areas:

–
–
–
–
–
Foundation Libraries
Math Libraries
Plugin Management
LCG Dictionary
Scripting services
Software Process
 Project Status and Summary

6 June 2003
SEAL - Core Libraries and Services Project
P. Mato/CERN
2
The Blueprint RTAG

The mandate of the RTAG was to define the architectural ‘blueprint’
for LCG applications:
– Define the main architectural domains (‘collaborating frameworks’) of LHC
experiments and identify their principal components.
– Define the architectural relationships between these ‘frameworks’ and
components, including Grid aspects, identify the main requirements for
their inter-communication, and suggest possible first implementations.
– Identify the high-level milestones for each domain and provide a first
estimate of the effort needed.
– Derive a set of requirements for the LCG



Any piece of software developed by any LCG common project must
conform to a coherent overall architectural vision
The end goal is the integration of LCG and non-LCG software to build
coherent applications
The blueprint will be established in terms of a set of requirements,
suggested approaches and guidelines, and recommendations
6 June 2003
SEAL - Core Libraries and Services Project
P. Mato/CERN
3
Architecture requirements














Long lifetime: technology evolution
Languages: C++ today; allow multi-language and evolution
Distributed applications
TGV and airplane work
Modularity of components
Component communication via public interfaces
Interchangeability of implementations
Integration into coherent framework
Design for end-user’s convenience more than the developer’s
Re-use existing implementations
Software quality at least as good as any LHC experiment
Meet performance, quality requirements of trigger/DAQ software
Platforms
Trigger/DAQ environment
6 June 2003
SEAL - Core Libraries and Services Project
P. Mato/CERN
4
Architectural Elements
Interface model
– Abstract interfaces, versioning,
guidelines,…
Component model
Design guidelines
– Dependencies
– Exception handling
– Interface to external components
6 June 2003
...
Other
Frameworks

Applications
Visualization
Framework
– Communication via public interfaces (no
hidden channels)
– Plug-ins (run-time loading)
– Life-time management (reference
counting)
– Configuration
Simulation
Framework

Software Structure
Reconstruction
Framework

Basic Framework
Foundation
Libraries
SEAL - Core Libraries and Services Project
Optional Libraries
P. Mato/CERN
5
Architectural Elements (2)

Object Dictionary
– The ability to query a class about its internal structure
(Introspection)
– Essential for data browsing, rapid prototyping, persistency, etc.

Object Whiteboard
– Uniform access to
application-defined
objects (equivalent to
the Gaudi transient
stores)

EDG API
PVSS
Database
Python
JPE
PyROOT
PyGaudi
Java
Classes
Root
Classes
Gaudi
Frame
work
Component Bus
– Integration of components
providing a wide variety
of functionality
6 June 2003
XML
SEAL - Core Libraries and Services Project
math
math
P. Mato/CERN
GUI
GUI
shell
6
The Blueprint RTAG Recommendations

RTAG establishes a user/provider relationship between
LCG software an ROOT
– LGC software will not be based on ROOT, it will use ROOT where
adequate




Start common project on core tools and services
 SEAL Project
Start common project on physics interfaces
Start RTAG on analysis, including distributed aspects
Tool/technology recommendations
– CLHEP, CINT, Python, Qt, AIDA, …

Develop a clear process for adopting third party software
6 June 2003
SEAL - Core Libraries and Services Project
P. Mato/CERN
7
SEAL Overview

SEAL aims to
–
–
–

Provide the software infrastructure, basic frameworks, libraries
and tools that are common among the LHC experiments
Select, integrate, develop and support foundation and utility class
libraries
Develop a coherent set of basic framework services to facilitate
the integration of LCG and non - LCG software
Scope
–
–
Foundation Class Libraries
» Basic types (STL, Boost, CLHEP, …), utility libraries, system
isolation libraries, domain specific foundation libraries
Basic Framework Services
» Component model, reflection, plugin management, incident
(event) management, distributed computing, grid, scripting
6 June 2003
SEAL - Core Libraries and Services Project
P. Mato/CERN
8
Domain Coverage
EvtGen
Algorithms
Engine
Event
Generation
Fitter
Detector
Simulation
Scripting
NTuple
Reconstruction
GUI
Analysis
Interactive
Services
Modeler
Geometry
Calibration
Event Model
FileCatalog
StoreMgr
Dictionary
Whiteboard
Persistency
Scheduler
PluginMgr
Core Services
Monitor
Grid
Services
Foundation and Utility Libraries
ROOT
6 June 2003
GEANT4
FLUKA
MySQL
DataGrid
SEAL - Core Libraries and Services Project
Python
Qt
P. Mato/CERN
...
9
Assumptions, constraints, risks

Do not re-invent the wheel
– Most of the core software to be delivered by SEAL exists - more
or less - in experiments’ core software
» We will re-use as much as possible existing software
» Most of the work will be in re- packaging existing pieces of
software

If wheel squeaks…
– Develop / adapt / generalize in order to achieve the necessary
level of coherency and conformance to the architectural vision
already established

Adopt a Seal
– In order to use SEAL, projects will need to replace their own
software elements with SEAL functionally equivalent ones. This will
certainly imply some period of instability for the experiment
applications
6 June 2003
SEAL - Core Libraries and Services Project
P. Mato/CERN
10
Project Work Packages
Foundation and Utility Libraries
Math Libraries
Component Model and Plug-in
Manager
LCG Object Dictionary
New work package added to the
project by incorporating the
MathLib project (F. James et al.)
This work package will probably be
split between “Foundation and
Utility Libraries” and “Basic
Framework services” later
Basic Framework Services
Scripting Services
Grid Services
Education and Documentation
6 June 2003
Not yet started working in this
area
SEAL - Core Libraries and Services Project
P. Mato/CERN
11
SEAL Versions Road Map
Release
Date
Status
Description (goals)
V 0.1.0
14/02/03
internal
Establish
V 0.2.0
31/03/03
public
Essential
V 0.3.0
16/05/03
internal
Improve
V 1.0.0
30/06/03
public
Essential
6 June 2003
dependency between POOL and SEAL
Dictionary generation from header files
functionality sufficient for the other existing
LCG projects (POOL)
Foundation library, system abstraction, etc.
Plugin management
functionality required by POOL
Basic framework base classes
functionality sufficient to be adopted by
experiments
Collection of basic framework services
Scripting support
SEAL - Core Libraries and Services Project
P. Mato/CERN
12
1. Foundation and Utility Libraries


Inventory of existing utility classes
Support for Boost library
– Open source utility library (parts of which are destined to be
included in the following C++ standard)


Participation to CLHEP project
Adopted CMS classlib
– A set of foundation and utility classes mainly used in Iguana (CMS)

Development of SealUtil and SealKernel packages
– The idea is to develop SEAL utility and system isolation library
complementary to Boost, CLHEP and STL

Guidelines for selecting external libraries
– Set of criteria to define
6 June 2003
SEAL - Core Libraries and Services Project
P. Mato/CERN
13
2. Math Libraries

GSL (Gnu Scientific Library) evaluation

Re-implementation of MINUIT in C++

Other libraries

Provide to experiments with math and statistics libraries
to be used in analysis, reconstruction, simulation.
– How it compares with NagC
– Automatic testing of GSL (with cppUnit) as part of every SEAL
release
– Prototype available in current SEAL release
– Migrad and Minos available
– The numerical results of the two prototypes compared to the
Fortran version. Compatible within the errors.
– MathLibs/GSLAlgebra. Prototype wrapper around GSL. Not
complete but restricted to Minuit needs.
– GSL support, …
6 June 2003
SEAL - Core Libraries and Services Project
P. Mato/CERN
14
3. Component Model and Plug-in Manager

Plug-in Management
– Service in charge of managing, querying, [un]loading plug-ins

Implementation in two levels
– Lower level (no framework constraints)
» Ability to create “module” libraries that contain “plug-ins”
» Ability to instantiate concrete implementations knowing the
plug-in category and the concrete type
» Caching information about what modules contain what plugins
and their categories
– Plugin instances management (assuming a given framework model)
» Locate other plug-ins and manage their lifetime
» Framework base classes to obey plug-in/framework model
6 June 2003
SEAL - Core Libraries and Services Project
P. Mato/CERN
15
3. Component Model and Plug-in Manager(2)
Plug-in
Database

Plug-In
Plug-In
Plug-In
Cache
Cache
Cache
Module
Module
Module
Module
Module
Howto’s available
Object
Object
Factory
Factory
Object
Factory
Attached
Unattached
– Host modules in your project
– Create a new plugin factory
– Create a new plugin module library
// Module Definition
DEFINE_SEAL_MODULE();
DEFINE_SEAL_PLUGIN(SomeFactory, MyObject);
// Usage by a client
BaseObject* p;
p = SomeFactory::get()->create(“MyObject”);
6 June 2003
SEAL - Core Libraries and Services Project
P. Mato/CERN
16
4. Object Dictionary


Adding “introspection/reflection” capabilities to C++
Reflection packages
– The Dictionary/Reflection package provides the capability to
introspect and interact with any C++ object at run-time
– The Dictionary/ReflectionBuilder package is the “write” interface
of the reflection information. Typically generated code uses this
interface to built the reflection information at run-time

Dictionary generation
– Main goal: Full support of C++ without any class instrumentation
– DictionaryGenerator package provides the command to do it

Usages so far
– Object persistency (POOL)
– Interactivity and scripting (Python)
6 June 2003
SEAL - Core Libraries and Services Project
P. Mato/CERN
17
4. Object Dictionary (2)

Dictionary generation from header files
– Uses gccxml (0.4.0) to parse header-files (extension to gcccompiler), generates intermediate XML file with dictionary
information
– XML file parsed by a python script which generates dictionary
building C++ code (for “selected classes”)
– Compiled to shared library and loaded at run-time to create
dictionary in memory
.xml selection file
.xml
.h
gccxml
.h
.h
.h
(python script)
par
ser
filter gendict
+
exten
_dict.cpp
make
.so
lcgdict
#include files
6 June 2003
SEAL - Core Libraries and Services Project
P. Mato/CERN
18
LCG Object Dictionary: Usage
.adl
.xml
.h
Population
ROOTCINT
GCC-XML
CINT generated
code
Dict generating
code
ROOT I/O
in
(1)
CINT
Dict
LCG to CINT
Dict gateway
Conversion
Streamer
ADL/GOD
LCG
Dictionary
(2)
out
Other
Clients:
(python,
GUI, etc.)
Reflection
6 June 2003
SEAL - Core Libraries and Services Project
P. Mato/CERN
19
5. Basic Framework Services

Developing the basic framework
– Component model definition
» component identification, configuration, lifetime strategy,
interface model, etc.
– Started to review existing designs (Gaudi, Cobra, Iguana,…)

Basic Component Model
– Set of “base classes” and interfaces that developers will use to
build their components

Developing Basic Common Services
– Started design and implementation for the message reporting
service
– Other services like component configuration, “event” management,
object white board, etc. will continue soon after
6 June 2003
SEAL - Core Libraries and Services Project
P. Mato/CERN
20
5. Basic Framework Services (2)

The current ideas
– Give to the “users” (framework
developers) a number pre-defined
service base classes (from simple to
more complex)
– Developers should “model” their services
using one of the proposed base classes
– Standard functionality (instantiation,
configuration, lifetime management,
etc.) should be provided by these bases
classes
6 June 2003
SEAL - Core Libraries and Services Project
Client
NameSvc
Context
Service
PropMgr
MyService
AServLet
BServLet
P. Mato/CERN
21
6. Scripting Services


Scripting needed for Interactivity, Configuration, Rapid
prototyping, Integration, etc.
GOAL: Bring Python and C++ (CINT) at the same level of
functionality
– Interoperability between them. The end-user choice.

Define guidelines for developing Python bindings
– Evaluate existing options: SWIG, Boost.Python, SIP, and raw
Python C-API
– Study how each technology handles a number of predefined cases
» Examples: Method overloading, inheritance across boundaries,
templated classes and methods, natural mapping of Python
constructs, etc.
– Study interoperability between binding technologies
6 June 2003
SEAL - Core Libraries and Services Project
P. Mato/CERN
22
6. Scripting Services (2)

PyROOT: Python bindings for ROOT (former
RootPython)
– PyROOT is a Python extension module that allows the
user to interact with any ROOT class
» Done generically using the ROOT dictionary. No need
to generate any Python wrapper code to include new
ROOT classes

PyLCGDict: Python extension module
– Started to work on the LCG Dictionary bindings
– Basically repeating the work of PyROOT using the LCG
dictionary instead
6 June 2003
SEAL - Core Libraries and Services Project
P. Mato/CERN
23
8. Education/Documentation

Main activities and tasks
– Produce documentation
– Produce training material (tutorials)
– Help incorporating SEAL components into LCG projects and
experiment frameworks

Existing documentation
– Very limited for the time being
– Produced a number of HowTo pages for the released elements
» Plugin Manager, Dictionary generation, PyROOT, etc.

Helping POOL to incorporate released components
– We have a team member working for both POOL and SEAL

Will be preparing tutorials for June release
– Time of possible adoption by experiments
6 June 2003
SEAL - Core Libraries and Services Project
P. Mato/CERN
24
Software Process

SPI provided infrastructure

Efforts to “standardize” tool usage and conventions

Release procedures

Platforms
– Savannah Web portal, CVS repository, development tools (SCRAM,
Doxygen, CppUnit, etc.) and services (external tool repository, etc.)
– Slowly getting the good dynamics and practices
– Avoiding divergences between LCG projects
– Adaptation of tools to “LCG proper” working models
– Developing SEAL proper procedures
– Tagging, building, testing, releasing, documentation generation,
announcing, etc.
– Automation of the process is essential (rotating release manager
role)
– Single release platform for the moment: Linux RedHat 7.3/gcc-3.2
6 June 2003
SEAL - Core Libraries and Services Project
P. Mato/CERN
25
Status and Summary


The SEAL project started last November as a result of
the “blueprint” RTAG recommendations
SEAL has had three main releases: 0.1.0, 0.2.0, 0.3.0 since
the beginning of the year at the scheduled times
– The main emphasis has been to support POOL (Dictionary, Plugin
Management, Foundation classes, etc.)

The functionality available is not yet enormous but its
development has help us
– To get experience on new tools and procedures
– Build up the development team


No technical problem encountered so far
Next scheduled release
– Basic functionality completed and ready to be adopted by
experiments at end-June
6 June 2003
SEAL - Core Libraries and Services Project
P. Mato/CERN
26
http://seal.cern.ch/
6 June 2003
SEAL - Core Libraries and Services Project
P. Mato/CERN
27