Transcript cowan_atlas_4may10

Statistics jump-start for early physics
ATLAS Statistics Forum
EVO/Phone, 4 May, 2010
Glen Cowan (RHUL)
Eilam Gross (Weizmann Institute)
G. Cowan, RHUL Physics
Statistics for early physics
page 1
Issues
For early physics we need to finalize methodologies for
setting limits (and quoting discovery significance).
For today focus mainly on limits
Need to agree with CMS what procedures we will use
(a) for comparison (also with e.g. Tevatron)
(b) for combination
Review of methods in use
Profile likelihood
CLs
Bayesian
Status of software tools (RooStats, …)
Look-elsewhere-effect (an issue for discovery, not limits).
G. Cowan, RHUL Physics
Statistics for early physics
page 2
Frequentist discovery and limits
For frequentist methods, focus on p-value
or equivalent significance Z = F-1(1 – p)
“Discovery” if p-value of background-only < 2.9 × 10-7 (Z > 5)
Can compute e.g. median discovery significance assuming
different signal models.
(In some cases may not have well-defined signal model.)
Exclusion at CL = 1 – a if p < a (e.g. a = 0.05)
Can compute e.g. median limit assuming background-only.
For exclusion, parameter (model) being tested is the null;
compute power relative to background-only alternative:
Power = P(reject model(parameter) | background only)
G. Cowan, RHUL Physics
Statistics for early physics
page 3
Bayesian discovery
For Bayesian discovery, compute Bayes factor to compare
e.g. model i (Higgs) to j (no Higgs):
Gives posterior odds if prior odds were 50-50.
Avoids dependence on unlikely data outcomes that were
never seen (cf. tail probabilities for p-values.)
Work needed here, mainly on computational issues.
G. Cowan, RHUL Physics
Statistics for early physics
page 4
Bayesian limits
Just integrate the posterior probability obtained from Bayes thm,
Not widely used so far in ATLAS (?) but will need anyway
for comparison with Tevatron.
Despite recent discussion on reference priors (a la Bernardo,
Jeffereys…), not aware of realistic applications in HEP.
Need survey of code and of who is using this in ATLAS.
Await outcome of CMS discussion on priors.
G. Cowan, RHUL Physics
Statistics for early physics
page 5
Systematics
Connect systematic to nuisance parameters n. Then form e.g.
Profile likelihood:
Marginal likelihood:
and use these to construct e.g. likelihood ratios for tests.
Coverage not guaranteed for all values of the nuisance params.
Literature contains some variants that we do not recommend,
e.g., forming a likelihood ratio and integrating it with a prior.
Need to decide what to do when systematic cannot be dealt
with using nuisance parameters (e.g. corrections using PYTHIA
vs. HERWIG).
G. Cowan, RHUL Physics
Statistics for early physics
page 6
ZBi, ZG, ZN, etc.
Several authors have studied various ways to obtain the
significance in the simple counting experiment with systematic
uncertainty on the background, e.g., Cranmer (PHYSTAT 03,05)
and Cousins, Tucker, Linnemann (physics/0702156).
Our usual profile likelihood method with a subsidiary measurement
for the background gives ZBi, also preferred by CMS.
ZN assumes a Gaussian prior for the background (truncated at
zero) and in many cases is not a realistic model. Should only be
used if the analyst actually believes that an average with respect
to the truncated Gaussian prior is the appropriate model (not likely).
CMS has flagged this as an area where we should reach agreement;
do not anticipate trouble here (does ATLAS use ZN?)
G. Cowan, RHUL Physics
Statistics for early physics
page 7
ATLAS practice
We need to complete our survey of methods used in ATLAS.
In StatForum we have made important progress in using
profile likelihood ratio tests.
Can get significance, limits without any toy MC (valid
for large samples, in practice can even be smallish).
For 95% CL limits for very low lumi, can always turn
to toy MC to calibrate method.
G. Cowan, RHUL Physics
Statistics for early physics
page 8
CLs
If we test parameter values to which we have no sensitivity
(e.g. very large Higgs mass), then there is a probability of
1 – CL (e.g. 5%) that we will reject.
In the CLs method the p-value is reduced according to the
recipe
Statistics community does not smile upon ratio of p-values;
would prefer to regard parameter m as excluded if:
(a) p-value of m < 0.05
(b) power of test of m with respect to background-only
> some threshold (0.5?)
Needs study. In any case should produce CLs result for purposes
of comparison with CMS/Tevatron.
G. Cowan, RHUL Physics
Statistics for early physics
page 9
Choice of likelihood ratio statistic
Ongoing discussion as to whether best to use LEP-style
likleihood ratio
or
and in both cases how to deal with the nuisance parameters.
In simple cases one obtains the same test from both statistics.
G. Cowan, RHUL Physics
Statistics for early physics
page 10
Questions to discuss
Need to decide about CLs (CMS have not yet decided).
Decide on methods for incorporating systematics (Zbi, ZN, …?)
Decide on ATLAS recommended methods to be used in
presentation of results
Check the status of Roostats as a tool for hypothesis testing
and for combinations.
G. Cowan, RHUL Physics
Statistics for early physics
page 11