Transcript Summary HENP software architecture and software

Category 7 Summary
Organizers: Torre Wenaus and NK
Nobu Katayama
KEK
HENP software architecture and
software engineering
• In our category, we have discussed mainly about
tools and/or mechanisms to deal with
– Large number of detector components
– Large number of people
– Multiple choices
– Integration
at active development stages
• There are two different scales
– One person trying to help 10 software developers
– 10 people trying to organize development work by
100 people
Goals
• For users, tools should just work; they should be
–
–
–
–
–
Simple to use
Easy to understand
Error prone
Stable and reliable
Supported well
• Toward generic, non-experiment specific tools?
– Standardization
• Common “data” format
– Coordination among experiments/laboratories
– Good tools become being used (slowly) by more and
more people
• Goodness itself doesn’t always make it happen, though
Talk/poster statistics
• LHC
–
–
–
–
ATLAS(6)
CMS(3)
LHCb(2)
ALICE(2)
• Other exp.
– CDF(2)
– Icecube, GLAST(2), CLEO-III(2), LC
• Accelerator/control
– DESY(2), BaBar/PEP-II
• Non-exp. specific (Root, freeHEP)
• LCG projects (SEAL, SCRAM)
Subcategories
• Event model, data definition(4)
• Package/project building, project
management and release tool (7)
• Geometry/detector definition(3)
• Framework and library (8)
• Others (6)
Event model/data definition
• Storegate: a Data Model for the Atlas Software
Architecture (ATLAS)
• Event Data definition in LHCb
• Athena Data Dictionary and Description
Language (ATLAS)
(as I understand it), we are waiting for LCG
version of it
• Event as an Object-Relational Database;
Avoiding the Dependency Nightmare (CLEO-III)
– Dependencies kept in one “library”
Object Relational Approach
Hit:1
1:data:1
Hit:2
2:data:1
Hit:3
Track:1
EM Shower:1
1:data:1,2
4:data:1
EM Shower:2
7:data:1
2:data:2
Hit:4
Hit:5
5:data:2
Hit:6
Hit:7
Hit:8
EM Shower:3
3:data:2
Track:2
6:data:2
8:data:2
Describe N-M relations
Package/project building, project
management, release tool
• OVAL: CMS testing tool
• User-friendly tools for managing projects (VCMT,
GLAST)
• Athena Startup Kit (ASK, ATLAS)
– User need to know only few commands when jumping
into analysis
• GLAST Nightly Package Builds and Code
Release System Tests
• BOA: framework for automated builds (CMS)
• NICOS system of nightly build for distributed
development (ATLAS, LCG)
• SCRAM (poster, LCG)
General Description
• Original goal : help to detect automatically the
changes in the physics performances of the CMS
software used for data production.
• Current features of the tool : compile the
provided test programs, run them in specified
runtime conditions (shell variables + auxiliary
files), compare the output with references.
• User interface is CVS-like, configuration files are
XML-like. It is written in PERL and can be
configured for any build system (scram, make).
From OVAL
Configuration : OvalFile
<diffline expr="^OVAL:">
<diffnumber expr="^OVAL NUM: .* is (.*)$" tolerance="5%">
<file name=".orcarc">
MaxEvents = 500
Random:Seeds = 0 3
</file>
<var name="FEDERATION" value="cmsuf01…">
<environment name="pt15">
<var name="DATASET" value="eg_ele_pt15">
<program name="Clusters.cpp">
<program name="Electrons.cpp" args="-geo detailed">
</environment>
<environment name="flow">
<var name="DATASET" value="jm_minbias">
<program name="EnergyFlow.cpp">
</environment>
From OVAL
GLAST
CHEP 03 March 24-28 2003
VCMT for Windows
T.Burnett
11/12
GLAST
CHEP 03 March 24-28 2003
Architecture of System Tests
Executive Script
(Perl)
Oracle DB
Web Interface
(Carrot, JAS,…,
Scripts)
Suite of Simulations
Run on
Supported
Platforms
Root Files,
Stdio Dumps
K.Young
Root Scripts
Root
Histogram
Files
12/22
Detector/geometry definition
• Detector Description Framework in LHCb
• GraXML – Modular Geometric Modeler
(ALICE)
Waiting for LCG decision
• Use of a generic identification scheme
connecting events and detector
description in the ATLAS experiment
– IDs for 100 million channels in the ATLAS
detector!
Data Diagram
Points to
Resolved on demand
Inherits from
Lvolume
Geometry
Info
Calibration
Alignment
Condition
Readout
Condition
Condition
MuonStation
EcalCluster
Structure
Sébastien Ponce
Material
*
Pvolume
DetElem
Solid
Box
Isotope
Sphere
Geometry
Mixture
Element
Material
EcalClusterCondition
MuonStationAlignment
Condition
VeloReadout
Conditions
CHEP 2003 - 3/21/03
14/18
Transient Store
Ask for
Object
Algorithm
Retrieve
pointer
●
●
●
●
Detector
Data Service
Check
presence
Persistency
Service
Ask creation
Lvolume
Lvolume
Lvolume
Lvolume
Geometry
Load
Cnv
Geometry
Db
Tree-like structure
Items identified by a logical name
Load/update on demand
Automatic update when new event
Sébastien Ponce
CHEP 2003 - 3/21/03
15/18
CondDB Implementation
●
Additional service of the framework
●
Independent of the type of data contained (XML)
●
Based on an abstract interface with several
possible backends (Objectivity, ORACLE, MySQL)
Ask for
Object
Algorithm
Retrieve
pointer
Sébastien Ponce
Detector
Data Service
Check
presence
Persistency
Service
Ask creation
Lvolume
Lvolume
Lvolume
Lvolume
Cnv
Condition
Service
Ask for data
Retrieve them
from Database
Abstract interface
Condition
Db
Geometry
CHEP 2003 - 3/21/03
16/18
Framework/library
• Root: Status and Future (See slides by yourself)
• The FreeHEP Java Library
• SEAL: Common core libraries and services for LHC
applications
• The AliRoot framework (ALICE) (See slides by yourself)
• Dynamically configurable system for fast simulation and
reconstruction for CMS (FAMOS)
• Java Physics Generator and Analysis Modules
• The Virtual Monte Carlo (ALICE)
• Reconstruction and Analysis on Demand: A Success
Story (CLEO-III)
FreeHEP
Java Library
HEP Specific Components
Physics package



3-vector, 4-vector’s and utilities
Jet Finding, Event Shape routines
Diagnostic Event Generator
HEP IO

HEP3D

StdHEP
MCFIO
HBook
Root




XML Particle Property Database
Interfaces and JHepRep - Java
reference implementation
HepRepJNI – C++ wrapper around
HepRep
see several other talks and
posters
Future

Mark Dönszelmann, SLAC
Interfaces and JAIDA - Java
reference implementation
AIDAJNI – C++ wrapper around
AIDA
More on this in Max Turri’s talk on
Tuesday
HepRep - HEP Representables for
Event Display
Extensions to Java3D specific for
Geant4
YaPPI – Yet another Particle
Property Interface



Access to a number of common
HEP formats:




AIDA - Abstract Interfaces for
Data Analysis
Plot Widget (JAS)
CHEP, La Jolla, 24-28 March 2003
FreeHEP
Java Library
General Components
Application Framework

Swing Extensions
FreeHEP Tools
General framework for Java
applications using plugin and
service technology.

AID – Abstract Interface
Definition


Compiler to make C++ headers and
Java Interfaces from aid files.
Graphics3D

Non-Java3D implementation of 3D
graphics, for WIRED.
FreeHEP IO

Extensions to the JavaIO system
to handle specific encodings,
ASCII85, …
PSViewer

PostScript Viewer to embed
viewing of EPS in applications, or
for previews.
Mark Dönszelmann, SLAC
JNeeds – java package
dependency
Ant – extensions to the build
system
VectorGraphics


Low level Input and Output of
binary vector graphics formats
(CGM, EMF, PDF, SVG)
Vector output files, extension to
Java2D
XML

XMLIO – classes to store/restore
state of an application
Future

RecordLoop/EventLoop (IceCube)
and more…
CHEP, La Jolla, 24-28 March 2003
Domain Coverage
EvtGen
Fitter
Algorithms
Engine
Event
Generation
SEAL
Detector
Simulation
Scripting
NTuple
Reconstruction
GUI
Analysis
Interactive
Services
Modeler
Geometry
Event Model
Calibration
FileCatalog
StoreMgr
Dictionary
Whiteboard
Persistency
Scheduler
PluginMgr
Core Services
Monitor
Grid
Services
Foundation and Utility Libraries
ROOT
GEANT4
FLUKA
MySQL
DataGrid
Python
Qt
...
Simulation/Reconstruction Chain
Simulation: OSCAR
MC
MC4-vector
4-vector
100-200 sec
SimHit
Reconstruction: ORCA
1-10 sec
RecHit / Digi
FAMOS/ORCA
FAMOS/OSCAR
FAMOS/ORCA
10-100 sec
Analysis
Analysisobject
object
Start with fastest mode
AIM: 1000 times faster
Interoperability
Pythia
Pandora
LCD FastMC
Event
Selection
Whizard
Event Generators
TESLA
SimDet
JLC
QuickSim
Java
Fortran
C, C++
Detector
Simulation
Analysis
Code
VMC Concept
User
Code
VMC
Geant3
Input
Geant4
Geant3 VMC
Particles
Hits
Fluka
Geant4 VMC
Fluka VMC
Output
Other
• Revision control in the Grid Era – the unmet
challenge (CDF)
• Commissioning the CDF Offline
• Concrete uses of XML in software development
and data analysis (IceCube)
• Improving the Security and Performance of the
BaBar Detector Controls System (poster)
• Concepts for Integrating Information systems
(poster, DESY)
• UML and explorative prototyping for developing
interactive applications (DESY)
Ominous clouds in the LHC era
• The detector components, computer
systems, software, and people are
– Complex
– Inter dependent
– Rapidly changing
– Globally distributed
• How can we be dealing with the situation?
• How can we integrate all software to work
together?
Trends
• XML is coming (+ RDMS is back)
– Describe, describe, describe
• Plug-ins
– Handle multiple choices (how lucky!)
• Nightly builds
– Just to see if anything has broken yesterday
• On demand (just in time?) reconstruction
• Ever expanding roots?
– VMC, AliEn, Geometry ,(PROOF, AliEnFS on other
sessions)
KEKB upgrade strategy
larger beam current
smaller by*
long bunch option
crab crossing
Constraint:
8GeV x 3.5GeV
wall plug pwr.<100MW
crossing angle<30mrad
L~1036
ILER=20A
dt =3000fb-1
L=1035
before
ILER
~10PB/year LHC!!
=9.4A
One year shutdown to:
replace vacuum chambers
double RF power
upgrade inj. linac g C-band
Present KEKB
L=1034
ILER=1.5A2.6A
dt =500fb-1~1PB/year
2002
03
04
05
06
07
08
09
10
11
My personal favorites
• For Belle, when I go home, I would investigate
more about
– OVAL, GraXML, FreeHEP Java library
– Root and related works
• For Super KEKB planning and work,
–
–
–
–
Virtual Monte Carlo and other MC frameworks
Definition tools
Nightly build tools
On demand reconstruction, analysis processing
• I am eager to see progresses of the LCG
projects and adopt