Transcript cowan_atlas_2feb12

News from the Statistics Forum
ATLAS Week Plenary Meeting
CERN, 2 February, 2012
Glen Cowan (RHUL)
Alex Read (Oslo)
G. Cowan
News from the Statistics Forum / 2 February, 2012
1
Overview
Summarize recent progress (meetings 8.11.11, 19.12.11, 9.1.12,
plus very active email/hypernews discussions).
Next Statistics Forum meeting is today (2.2.12), 17:00, Salle Dirac
Main topics for today:
Refinement for frequentist tests (“uncapped” p-values)
Progress on Bayesian reference analyses
ABCD method revisited
G. Cowan
News from the Statistics Forum / 2 February, 2012
2
Aaron Armbruster, Tim Ayde
Refinement for Frequentist Tests
The usual statistic we define to test, e.g., the background-only
hypothesis reflects disagreement with the hypothesis only if
the data fluctuate higher than the expected background:
This means that the p-value of the background-only hypothesis,
under assumption of background only, is 0.5 ~half the time.
Bug or feature? The interesting cases are when the p-value is
small, e.g., p0 < 2.9 × 10-7 means we reject background-only
hypothesis at 5σ level.
But the plot of p0 versus mH comes out “capped” at 50%.
G. Cowan
News from the Statistics Forum / 2 February, 2012
3
Aaron Armbruster, Tim Ayde
Capped p-values
Here data lower than expected background, p0 capped at 0.5
G. Cowan
News from the Statistics Forum / 2 February, 2012
4
Aaron Armbruster, Tim Ayde
Similar capping for CLs
G. Cowan
News from the Statistics Forum / 2 February, 2012
5
Aaron Armbruster, Tim Ayde
Redefinition of test statistic
Replace q0 by
G. Cowan
ˆˆ
L(0,q )
t0 = -2ln
L(m̂ ,qˆ )
News from the Statistics Forum / 2 February, 2012
6
Aaron Armbruster, Tim Ayde
Uncapped p-values
G. Cowan
News from the Statistics Forum / 2 February, 2012
7
Aaron Armbruster, Tim Ayde
Summary on (un)capped p-values
By modifying the test statistic as proposed, there is no change
in cases where the p-value is less then 0.5, so no important
conclusions change.
But if the data fluctuate away from discovery or exclusion,
the new statistic makes this easily visible.
Some technical issues concerning asymptotic properties of q0′
still under study, but providing these are resolved we will
propose to ATLAS (and CMS) that the “uncapped” method be
adopted as part of the standard frequentist procedure.
G. Cowan
News from the Statistics Forum / 2 February, 2012
8
Progress on Bayesian methods
Although main emphasis has been on frequentist
methods, also important progress on Bayesian techniques:
E.g., many contributions to Statistics Forum on Bayesian
Analysis Toolkit (BAT) and Bayesian tools within RooStats.
G. Cowan
News from the Statistics Forum / 2 February, 2012
9
Bayesian priors
For various reasons (historical,simplicity, conservatism,...) the prior
for the mean number of signal events, s, is usually taken as constant:
This has many well-known features: very conservative
for limits, not invariant under reparametrization,...
In the statistics community there is wide use of so-called “reference”
priors. Recently Diego has developed a method to construct the
reference prior for the signal parameter s in counting experiments:
D. Casadei, Reference analysis of the signal + background model
in counting experiments, JINST 7 (2012) P01012; arXiv:1108.4270
G. Cowan
News from the Statistics Forum / 2 February, 2012
10
Reference analysis à la Diego
Find marginal model for the number of events, k, given a mean
signal s and background b, by assuming b has a gamma prior:
Use this model to compute the Fisher information:
And then use this to find the reference prior for s: (same as the
Jeffreys prior for single parameter):
For details see Diego’s paper!
G. Cowan
News from the Statistics Forum / 2 February, 2012
11
Example of reference priors π(s) for
different
gamma priors used for the background
D. Casadei, JINST 7 (2012)
P01012; arXiv:1108.4270
G. Cowan
News from the Statistics Forum / 2 February, 2012
12
Status of Bayesian Recommendation
The current ATLAS recommendation on limits is to use
(frequentist) CLs, except for analyses with a history of using
Bayesian limits (and if CMS also follows this), in which case a
constant prior is used for the signal rate.
The reference prior has a number of important advantages:
fast convergence of limit’s coverage probability to
the (Bayesian) probability content;
invariance under reparametrization,...
We would like to encourage those people doing Bayesian
analyses to try out the reference prior so that we can gain
experience with it.
G. Cowan
News from the Statistics Forum / 2 February, 2012
13
Alex, Eilam, Ohad,...
Discussions on ABCD Method
Basic idea:
Measure x and y
(~uncorrelated for bkg)
Signal only in A;
estimate background
in A by:
nB
nbkg,A = nC
nD
Refinements for correlations, spillover of signal into B, C, D,...
Difficulty arises when using error propagation for error of
background estimate if some of the boxes have few events.
G. Cowan
News from the Statistics Forum / 2 February, 2012
14
“Write down the likelihood”
Alex, Eilam, Ohad,...
Treat the numbers of events found in A, B, C, D as independent
and Poisson distributed; the likelihood is:
For details see note by Alex on twiki (Statistics Tools in ATLAS):
https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/StatisticsTools
Use the likelihood to test different values of the signal rate
(e.g., profile likelihood ratio); or use in Bayesian analysis.
Straightforward generalization to include further refinements
(correlations, crossover, systematics, etc.)
G. Cowan
News from the Statistics Forum / 2 February, 2012
15
Some other recent and ongoing discussions
Unfolding
Main effort in SM, exotics, top – B. Malaescu, F. Spano
https://indico.cern.ch/conferenceDisplay.py?confId=157136
https://indico.cern.ch/conferenceDisplay.py?confId=156510
Plotting differences between data and expectation
D. Casadei and G. Choudalakis, arXiv:1111.2062
Issues related to unexpectedly tight constraints on nuisance
parameters from profile-likelihood method.
Revisiting the “tevatron” likelihood ratio q = - 2 ln (Ls+b/Lb) (GC).
Multiple regression (Samir Ferag), SusyFitter (Dan Short)
Asymptotic combination of uncorrelated channels (Ohad Silbert)
Your input is welcome!
G. Cowan
News from the Statistics Forum / 2 February, 2012
16
Extra slides
G. Cowan
News from the Statistics Forum / 2 February, 2012
17
Unfolding twiki page (SM & Top Groups)
https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/StandardM
odelUnfolding
G. Cowan
News from the Statistics Forum / 2 February, 2012
18
Workshop on how to package our results
G. Cowan
News from the Statistics Forum / 2 February, 2012
19
Reminder of Editorial Board Guidelines
https://twiki.cern.ch/twiki/bin/viewauth/
AtlasProtected/EditorialBoardGuidelines
Ensure the analysis follows the recommendations of
the ATLAS Statistics Forum with regard to statistical
procedures (e.g. limit setting, unfolding, combinations
of data). In cases of doubt, or when complex or
unusual procedures are being used, make sure the
authors discuss with the Statistics Forum conveners
before the analysis reaches the approval stage.
G. Cowan
News from the Statistics Forum / 2 February, 2012
20