Transcript Higgs Production

XML Summary:
status and plans
Robert W. Lambert
R Lambert, Imperial
Core Software, 29th July 2009
1
Example Summary
 The XML summary is a computer and human-readable
shorter version of the stdout, with salient information kept
<?xml version="1.0" encoding="UTF-8"?>
<summary version="1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="XMLSummary.xsd">
<success>true</success>
<step>finalize</step>
<usage>
<stat unit="b" useOf="MemoryMaximum">1024.0</stat>
</usage>
<input>
<file status="full" name="PFN:/path/file.dst" GUID="0000-0000-0000-0000">500</file>
</input>
<output>
<file status="part" name="PFN:/path/file.dst" GUID="">500</file>
</output>
<counters>
<counter name="UserAlg/UserEventCounter">500</counter>
<statEntity name="UserAlg/UserStatEntity“ format="Flag Entries Flag2"
max="1.0" min="1.0">500.0 500 500.0</statEntity>
</counters>
<lumiCounters>
<lumiCounter name=“LumiAlg/EmptyEvents">1000000</lumiCounter>
<lumiStatEntity name="LumiAlg/SomeComplicatedMeasurement"
format="Flag Entries Flag2" max="1000.0" min="0.0">40050.0 500 2500001.0</lumiStatEntity>
</lumiCounters>
</summary>
R Lambert, Imperial
Core Software, 29th July 2009
2
What’s stored in XML?
 We aim to store:
1. Counters defined in tools/algorithms
 Simple counters and StatEntities
 Configurable for each alg, all as simple counters by default
 Lumi counters (always written if declared)
2. List of files, input and output



Name, GUID
Status: “none” “fail” “part” “full” “mult”
Events processed
3. Status of Gaudi


Initialize, execute, finalize
“Application manager has terminated successfully”
4. Usage statistics (Memory usage, not yet implimented)
R Lambert, Imperial
Core Software, 29th July 2009
3
Defining a schema
 At a LHCb joint meeting we decided to use a schema
 A schema defines what appears in the xml, ensuring stability
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<!-- This is the XML schema for the Gaudi XML Summary -->
<!-- The version of the schema appears in the summary, and must be edited in each version -->
<!-- The summary element is the only pre-defined element -->
<!-- This is the root element of the schema -->
<xs:element name="summary" type="SummaryType"/>
<!-- This version attribute holds the version of the schema, which may change -->
<!-- it is written such that new objects cannot be parsed with an older schema -->
<xs:attribute name="version" default="1.0">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:whiteSpace value="collapse"/>
<xs:enumeration value="1.0"/>
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<!-- We start with defining the simple types --> ...
R Lambert, Imperial
Core Software, 29th July 2009
XML Files are Validated
against the schema.
4
What we really want
XML Schema (XSD)
Gaudi
DIRAC
Simple
Parsing
Code
The XML must never deviate
from the schema.
The schema can change:
getting a new version #
Ganga
XML Summary (XML)
R Lambert, Imperial
Core Software, 29th July 2009
5
C++ option
XML Schema (XSD)
C++ schema code
Gaudi
C++ parsing code
DIRAC
Python Wrapper
Using Xerces.
The C++ code must
be maintained to stick
To the schema
Code duplication
within python wrapper.
Wrapper would need to
be maintained, with
parsing changes
Ganga
XML Summary (XML)
R Lambert, Imperial
Core Software, 29th July 2009
6
Python option
Using the existing python
XML libraries, the schema
can dictate it’s own structure.
Schema evolution is transparent.
XML Schema (XSD)
Gaudi
#include
“Python.h”
DIRAC
Helper
methods
Python
schema
(me)
Python
parsing
(xml.etree)
Ganga
XML Summary (XML)
Code does not
depend on the xml
or schema.
Helper methods can
be developed.
Currently depends
on xml.etree
(ElementTree )
(python 2.5)
.. Could use any
similar tree parser
R Lambert, Imperial
Core Software, 29th July 2009
7
The Gaudi side
 The objects to write come from different times and places
Event Selector
(initialize)
XMLSummarySvc
#include
“Python.h”
XSD
GaudiCommon
(finalize)
Python
Generic
IncidentSvc
XML
Event Selector
RANDOM
Output Files, push
Output Stream
RANDOM
R Lambert, Imperial
Core Software, 29th July 2009
Written in
Initialize, Finalize,
and every time an input
file is completed.
8
Existing Code
 My test package Gaudi/XMLSummary
 Requires GaudiKernel, GaudiPython
 schema.py, summary.py, test_parse_schema.py
 Python code to read/write/parse the XML summary
 XMLSummarySvc
 IXMLSummarySvc -> “declare” a Stat or StatEntity
 IIncidentListener -> “handle” incidents
 Interfaces directly with the python code
 XMLTestAlg : GaudiAlgorithm
 “declare” counters in finalize
R Lambert, Imperial
Core Software, 29th July 2009
9
Test and options
 Tested in DaVinci v23r3 (complicated user program)
#Default parameters are given, where appropriate
#what filename to write
XMLSummarySvc().xmlfile="summary.xml“
#an empty string would not write a file
#what schema to use
XMLSummarySvc().xsdfile=“$XMLSUMMARYROOT/xml/XMLSummary.xsd“
XMLSummarySvc().OutputLevel=1
#what counters to fill, regex
XMLSummarySvc().counter_list=[ “.*” ]
XMLSummarySvc().stat_entity_list=[ “.*” ]
#default saves all declared counters
#test GaudiAlgorithm
sumAlg=XMLTestAlg()
sumAlg.OutputLevel=1
#what counters to declare, regex
sumAlg.counter_list=[ “.*” ]
#sumAlg.stat_entity_list=[]
#default declares all as simple counters, no stat entities
R Lambert, Imperial
Core Software, 29th July 2009
10
XMLTestAlg: Finalize
StatusCode XMLTestAlg::finalize()
{
debug() << "finalize" << endmsg;
//example of how to declare a Lumi stat counter.
m_xmlSummarySvc->declare(name(),"lumi",counter("lumi"), true, false);
//add counters, this bit will no doubt be moved to the base class
for(Statistics::const_iterator i=counters().begin();
i!=counters().end();
i++)
{
if(m_xmlSummarySvc->match(i->first,m_counter_list))
m_xmlSummarySvc->declare(name(),i->first,i->second);
if(m_xmlSummarySvc->match(i->first,m_stat_entity_list))
m_xmlSummarySvc->declare(name(),i->first,i->second,false,false);
}
return GaudiAlgorithm::finalize();
}
R Lambert, Imperial
Core Software, 29th July 2009
11
Test OK
 Test version of summary.xml successfully created
<?xml version="1.0" encoding="UTF-8"?>
<summary version="1.0" xsi:noNamespaceSchemaLocation="$XMLSUMMARYROOT/xml/XMLSummary.xsd“
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<success>True</success>
<step>finalize</step>
<usage />
<input>
<file GUID="" name="PFN:/path/filename.dst" status="part">1</file>
</input>
<output />
<counters>
<counter name="XMLTestAlg/executed">1</counter>
<counter name="XMLTestAlg/lumi">0</counter>
<counter name="XMLSummarySvc/handled">1</counter>
<statEntity format="Flag Entries Flag2" max="1.0" min="1.0"
name="XMLSummarySvc/handled">1.0 1.0 1.0</statEntity>
</counters>
<lumiCounters>
<lumiStatEntity format="Flag Entries Flag2" max="0.479425538604" min="0.479425538604"
name="XMLTestAlg/lumi">0.479425538604 1.0 0.229848847066</lumiStatEntity>
</lumiCounters>
</summary>
 Several missing pieces, requiring editing of Gaudi
R Lambert, Imperial
Core Software, 29th July 2009
12
Wish List
1. Migrate package into Gaudi

Add IXMLSummarySvc somewhere, upload XMLSummary
2. Migrate functionality to GaudiCommon


Requires a pointer to XMLSummarySvc
Requires a common name() method
3. Add new incidents, fire in the right place, source=“GUID,file”




“NEW_OUTPUT_FILE”
"WROTE_TO_FILE”
"FAILED_OUTPUT_FILE”
“END_OUTPUT_FILE”
“NEW_INPUT_FILE”
“FAILED_INPUT_FILE”
“END_INPUT_FILE”
4. Add methods to obtain filename<->GUID pairings
R Lambert, Imperial
Core Software, 29th July 2009
13
Discussion
 XML is always valid thanks to using python.
 Schema evolution nearly transparent
 At some point something has to be hard-coded
 E.g. string “finalize” written in finalize()
 We need to decide who turns on/off the summary
 The user? Atlas/LHCb?
 Ganga/DIRAC ?
 Gaudi/Gauss/Boole/Brunel/DaVinci ?
 We need to decide where to put the interface/services
 Gaudi Project? LHCb Project?
R Lambert, Imperial
Core Software, 29th July 2009
14
Conclusions
 The XML summary will be shorthand for stdout
 The format of the XML is mostly decided
 A few implementation options have been discussed
 A python-based test-case has been demonstrated to work
 Perhaps we can agree on this?
 Further work requires committing to Gaudi, and changing
Gaudi base classes/packages
 Thanks to Marco, Marco, Pere and Vanya for their help!
 Probably still going to need a lot of it
R Lambert, Imperial
Core Software, 29th July 2009
15
End
 Backups hereafter
R Lambert, Imperial
Core Software, 29th July 2009
16
The Ganga Side
 GangaLHCb will depend on Gaudi/XMLSummary
 Helper methods will already exist in the “Summary” class
XSD
Ganga
Job
j.application.summary()
(don’t store in job)
Inherited
ganga
(GPI)
object
Inherited
ganga
object
Helper
methods
isFailure()
Merge([s,s’])
Python
schema
(me)
Python
parsing
(xml.etree)
XML
Gaudi/XMLSummary
 Extra functionality can be added through inheritance
R Lambert, Imperial
Core Software, 29th July 2009
17
Python option
R Lambert, Imperial
Core Software, 29th July 2009
18
Python option
...
R Lambert, Imperial
Core Software, 29th July 2009
19
What XML Summary?
 A GaudiJob outputs thousands of lines stdout
 Buried in this is some useful information, hard to get at
R Lambert, Imperial
Core Software, 29th July 2009
20
What XML Summary?
 A GaudiJob outputs thousands of lines stdout
 Buried in this is some useful information, hard to get at
R Lambert, Imperial
Core Software, 29th July 2009
21
What XML Summary?
 A GaudiJob outputs thousands of lines stdout
 Buried in this is some useful information, hard to get at
R Lambert, Imperial
Core Software, 29th July 2009
22