Transcript SEAL Project Overview

SEAL Project Overview
ACAT03
1-5 December, 2003
KEK
IX International Workshop on Advanced Computing
and Analysis Techniques in Physics Research
Lorenzo Moneta/ CERN-EP
on behalf of the SEAL team
Outline

Introduction
– Scope and motivation
SEAL Work Packages
– Description
– Current status
 Future plans and summary

ACAT03, 1-5 Dec. 2003
SEAL Project Overview
L. Moneta / CERN
2
Project motivation

What is SEAL ?

SEAL aims to
–
–
–
–
–
–

project in LCG Application Area
started last year after BluePrint RTAG recommendations
common project on core tools and services
» SEAL: Shared Environment for Applications at LHC
Provide the software infrastructure, basic frameworks, libraries
and tools that are common among the LHC experiments
Select, integrate, develop and support foundation and utility class
libraries
Develop a coherent set of basic framework services to facilitate
the integration of LCG and non - LCG software
Do not re-invent the wheel
–
–
Try to use as much as possible of existing software
Develop/adapt/generalize to achieve necessary level of coherency
ACAT03, 1-5 Dec. 2003
SEAL Project Overview
L. Moneta / CERN
3
Project scope
Software Structure

– Basic types (STL, Boost,…), utility
libraries, system isolation libraries,
domain specific foundation libraries

Other
Frameworks
Visualization
Framework
Reconstruction
Framework
Simulation
Framework
Applications
...
Basic Framework
Foundation
Libraries
Optional Libraries
ACAT03, 1-5 Dec. 2003
Foundation Class Libraries
Mathematical Library
– Basic mathematical functions, linear
algebra, fitting and minimization
libraries

Basic Framework Services
– Component model, reflection, plug-in
management, incident (event)
management, distributed computing,
grid services, scripting
SEAL Project Overview
L. Moneta / CERN
4
Project Work Packages
Foundation
Foundation and Utility Libraries and Plug-in Manager
MathLibs
Math Libraries Support and Coordination
Dictionary
LCG Object Dictionary
Framework
Component Model and Basic Framework services
Scripting
Scripting Services
Grid
Grid Services
Documentation
Education and Documentation
ACAT03, 1-5 Dec. 2003
Not yet active
SEAL Project Overview
L. Moneta / CERN
5
Foundation and Utility Libraries

Inventory of existing C++ utilities
(http://seal.cern.ch/components.html)
– Recommended classes useful for LCG projects and experiments
– Large number from Boost (open source project)
» Support for Boost is provided (in contact with developers)


Develop in SEAL what is missing in STL and Boost
Have a set of low-level, fairly independent class libraries
–
–
–
–

200 classes originated mainly from classlib (CMS)
Basic abstraction and foundation utilities classes
Timing utilities
Stream-oriented I/O, compression and decompression
High quality, efficient, well tested and documented library
– Recommend best practice to LHC developers
ACAT03, 1-5 Dec. 2003
SEAL Project Overview
L. Moneta / CERN
6
Plug-in Manager

Plug-in Management
– Service in charge of managing, querying, [un]loading plug-ins

Low level plug-in management (no framework constraints)
– Ability to create “module” libraries containing “plug-ins”
– Ability to instantiate concrete implementations knowing the plug-in
category and the concrete type
– Caching information for fast loading
» what modules contain
» what kind of plug-ins exist and which categories

Imported from CMS IGuana
– Basic concept: advanced object factory
» Interfaces for object instantiation and plug-in provider
– Dynamic loading completely orthogonal — and optional!
ACAT03, 1-5 Dec. 2003
SEAL Project Overview
L. Moneta / CERN
7
Mathematical Libraries

GSL (Gnu Scientific Library) evaluation

Re-implementation of MINUIT in C++

Linear Algebra


– Extensive comparison with NagC
– Validation and test suite
– Prototype available with Migrad and Minos
– Undergoing numerical and performance tests
» Comparison tests with Fortran version (compatibility within errors)
– Prototype wrapper around GSL matrices.
– Comparison of various linear algebra packages (CLHEP, GSL, [u]CBLAS)
Participation in CLHEP project
Provide to experiments math and statistics libraries to be
used in analysis, reconstruction and simulation.
– support, consultancy, etc..
ACAT03, 1-5 Dec. 2003
SEAL Project Overview
L. Moneta / CERN
8
Object Dictionary



Adding “introspection/reflection” capabilities to C++
Use for persistency and interactivity (e.g. Python)
Modern languages provide reflection inherently
– In C++ reflection is very limited (RTTI)

The basic C++ concepts shall be supported
– e.g. inheritance, methods, data members, accessibility, templates

Seal provides :
– Reflection Builder and Reflection (API interface) packages
– Dictionary generator to produce dict. source from C++ files
» Uses gcc-xml + Python script
– Dictionary for some standard packages (STL, CLHEP)

Used by POOL (persistency)
ACAT03, 1-5 Dec. 2003
SEAL Project Overview
L. Moneta / CERN
9
Dictionary Generator



gcc-xml to parse header-files (extension to gcc )
XML file generated with dictionary information
Python script to generate dictionary C++ code
– use an additional XML file to customize class selection

User
.h
.h
.h
.h
Compiled and loaded at run-time to create dictionary
– Provide meta information about the object to the user
.xml
gccxml
.xml
(Python script)
par
ser
selection file
filter gendict
+
exten
lcgdict
#include files
ACAT03, 1-5 Dec. 2003
SEAL Project Overview
_dict.cpp
make
..Dict.so
Reflection
Libraries
Clients
L. Moneta / CERN
10
Framework Services

Motivation:
– Allow LCG developers and users to write portable and modular code
– Provide abstractions for modelling software systems

Component model definition
– Basic Components
» Set of “base classes” and interfaces needed to build more
complex components
» Provide standard functionality
 instantiation, configuration, lifetime management, etc


Users (framework developers) should model their services
using these proposed base classes
SEAL provide in addition a set of basic services
implemented following the component model
ACAT03, 1-5 Dec. 2003
SEAL Project Overview
L. Moneta / CERN
11
SEAL Component Model
Context
Application
C1
C2
C1
C5
C1
Svc1
C3
C1
Svc2
C4
C1
C6
C1
Svc1
Component
C7
C1

C7
C1
Service
Hierarchy of bases classes
– Components living in a Context
– Services are Components providing their local Context

User classes inherit from Component or Services
– Plug-in functionality for free
ACAT03, 1-5 Dec. 2003
SEAL Project Overview
L. Moneta / CERN
12
Basic Common Framework Services

Application Service
– Defines the top level Context
– Defines initial set of Components to be loaded in the application

Message Service
– Message composition, filtering and reporting

Configuration Service
– Management of Component properties and loading configurations
– Multiple backends foreseen:
» Gaudi style options, .INI style, CMS style, XML, ...

Other services will come later:
– Object Whiteboard (Transient Object repository)
– Dictionary Service (Loading Dictionaries in the application)
ACAT03, 1-5 Dec. 2003
SEAL Project Overview
L. Moneta / CERN
13
Scripting Services



Scripting needed for Interactivity, Configuration, Rapid
prototyping, Integration (Component bus), etc.
Main scripting language :
– Python
XML
Database
EDG API
PVSS
Define guidelines for
developing Python
bindings :
– Various tools available
(Boost.Python, SWIG)
– Recommend solutions:
Python
JPE
PyROOT
PyGaudi
Java
Classes
Root
Classes
Gaudi
Frame
work
math
math
GUI
GUI
shell
» Boost.Python
» LCG Dictionary
ACAT03, 1-5 Dec. 2003
SEAL Project Overview
L. Moneta / CERN
14
Scripting Packages

PyLCGDict
– Python extension module to the LCG dictionary
– Provide access to C++ libraries from Python, “for free”
– Automatically generates Python proxies for C++ objects
using dictionary information
» Everyone can create binding effortlessly

PyROOT
– Python bindings for ROOT classes
» access to ROOT functionality from Python
– Binding using the ROOT dictionary (CINT)
– Missing gateway ROOT -> Python
ACAT03, 1-5 Dec. 2003
SEAL Project Overview
L. Moneta / CERN
15
Education/Documentation

Produce documentation
– Reference documentation (generated with doxygen)
– HowTo pages for the released elements
» Plugin Manager, Dictionary generation, PyROOT, PyLCGDict, etc.
– Release notes

Training material (tutorials)
– Help developers incorporating SEAL into the LCG projects and the
experiment frameworks

Python courses
– Hands-on introduction course to Python programming
– Very positive feedback received
ACAT03, 1-5 Dec. 2003
SEAL Project Overview
L. Moneta / CERN
16
Software Process and QA

Design
– Team design sessions
– Prototypes using Pythons

Code Review
– No formal code reviews
– Coding done very often in pairs (XP style)
– More than one developer knowledgeable for each package

Testing
– More than 200 SEAL unit tests based on CppUnit
– All are driven and run automatically by QmTest

Bug reporting and tracking
– Savannah Portal
ACAT03, 1-5 Dec. 2003
(http://savannah.cern.ch/projects/seal)
SEAL Project Overview
L. Moneta / CERN
17
Future plans (1)

Foundation libraries (Plug-in Manager)
– Follow Boost developments
– Little development planned in SEAL (hash-maps)

MathLibs:
–
–
–
–

Produce inventory of mathematical functions
Evolve Minuit C++ into a final product
Linear Algebra studies
Generate dictionary library for Python and CINT
Dictionary:
– Implementation of new reflection model
» Overcome some existing limitations (typedef,…)
– Converge to a common dictionary with CINT ?
ACAT03, 1-5 Dec. 2003
SEAL Project Overview
L. Moneta / CERN
18
Future plans (2)

Framework:

Scripting :

Grid :

Documentation:
– Develop Whiteboard (object repository) and Dictionary service
(loading of dictionary libraries on-demand)
– PYLCGDict:
» Migrate much of functional core from C++ to Python
» Support more natural Python features (eg iterator protocol)
– PyROOT:
» Unify core with PyLCGDict
» Develop gateway ROOT -> Python
– New LCG project (ARDA) is starting
– Expect need to interface to some of the ARDA services
– Produce tutorials for foundation classes, Boost, etc..
ACAT03, 1-5 Dec. 2003
SEAL Project Overview
L. Moneta / CERN
19
Summary

SEAL has delivered basic foundation, utility libraries and
object dictionary
– The main “client” have been POOL (persistency project) and PI
» Plug-in manager is also used in CMS
– Very positive feedback received for plug-in manager
– Currently being integrated into experiments’ frameworks of LHCB
(Gaudi) and ATLAS (ATHENA)

The first version of the Component Model and Framework
services is available
– Start integrating in POOL and the LHC experiments
– Wait for feedback before developing more services

Scripting based on Python
– Boost.Python and PyLCGDict recommended for Python bindings
– PyLCGDict is used to provide bindings for classes of Physics
Interfaces (PI) project
ACAT03, 1-5 Dec. 2003
SEAL Project Overview
L. Moneta / CERN
20
Conclusions
http://seal.cern.ch/
Acknowledgments:
Seal team members:
R. Chytracek, J. Generowicz, W.
Lavrijsen, M. Marino, P. Mato,
S. Roiser, L. Tuura, M. Winkler
ACAT03, 1-5 Dec. 2003
SEAL Project Overview
L. Moneta / CERN
21
Additional Slides
ACAT03, 1-5 Dec. 2003
SEAL Project Overview
L. Moneta / CERN
22
LCG Application Domain Coverage
EvtGen
Algorithms
Engine
Event
Generation
Fitter
Detector
Simulation
Scripting
NTuple
GUI
Analysis
Reconstruction
Interactive
Services
Modeler
Calibration
Event Model
Geometry
FileCatalog
StoreMgr
Dictionary
Whiteboard
Persistency
Scheduler
PluginMgr
Core Services
Monitor
Grid
Services
Foundation and Utility Libraries
ROOT
GEANT4
ACAT03, 1-5 Dec. 2003
FLUKA
MySQL
DataGrid
SEAL Project Overview
Python
Qt
L. Moneta / CERN
...
23
Plug-in Manager Design

Originated from CMS Iguana
PluginManager
PluginCache
*
PluginFactoryBase
ModuleBase
*
T
PluginFactory
XXXFactory
PluginInfo
*
Module
XXXInfo
ModuleDef
MyModuleDef
Module Library
Client
ACAT03, 1-5 Dec. 2003
SEAL Project Overview
L. Moneta / CERN
24
Reflection System


Reflection Builder: the loading interface
Reflection : the user API
NamespaceBuilder
ItemBuilder
ClassBuilder
FieldBuilder
MethodBuilder
ReflectionBuilder
Item
Namespace
Class
PropertyList
Field
Method
Reflection
ACAT03, 1-5 Dec. 2003
SEAL Project Overview
L. Moneta / CERN
25
LCG Object Dictionary: Usage
.adl
.xml
.h
Population
ROOTCINT
GCC-XML
CINT generated
code
Dict generating
code
ROOT I/O
in
(1)
CINT
Dict
LCG to CINT
Dict gateway
Conversion
Streamer
ADL/GOD
LCG
Dictionary
(2)
out
Other
Clients:
(python,
GUI, etc.)
Reflection
ACAT03, 1-5 Dec. 2003
SEAL Project Overview
L. Moneta / CERN
26
Seal Component Model





Hierarchy of bases classes
Each Component lives in a Context
Contexts form their own hierarchy
Support for component look-up
A Service provides its own local
Context
– Components of a Service live in the
scope defined by its own Context

User classes inherit from
Component or Service
– Plug-in functionality
for free
ACAT03, 1-5 Dec. 2003
Component
Factory
Component
Component
Loader
*
1
Service
1
Context
*
MyComponent
SEAL Project Overview
MyService
L. Moneta / CERN
27