Transcript cowan_atlas_25may10

Proposal for alternative to CLs:
Power-Constrained Limits
ATLAS Statistics Forum
CERN, 25 May, 2010
Glen Cowan, RHUL
Kyle Cranmer, NYU
Eilam Gross, Ofer Vitells, Weizmann Inst.
G. Cowan, RHUL Physics
Alternative to CLs
page 1
The “CLs” issue
When the cross section for the signal process becomes small
(e.g., large Higgs mass), the distribution of the test variable used
in a search becomes the same under both the b and s+b hypotheses:
f (q| b)
f (q| s+b)
In such a case we will reject the signal hypothesis with a
probability approaching a = 1 – CL (i.e. 5%) assuming no signal.
G. Cowan, RHUL Physics
Alternative to CLs
page 2
The CLs solution
The CLs solution (A. Read et al.) is to base the test not on
the usual p-value (CLs+b), but rather to divide this by CLb
(one minus the background of the b-only hypothesis, i.e.,
Define:
f (q| b)
f (q| s+b)
q
Reject signal
hypothesis if:
G. Cowan, RHUL Physics
Reduces “effective” p-value when the two
distributions become close (prevents
exclusion if sensitivity is low).
Alternative to CLs
page 3
Alternative proposal – basic idea
CLs method reduces the p-value according to:
where m = strength parameter, proportional to cross section.
Statistics community does not smile upon ratio of p-values;
would prefer to regard parameter m as excluded if:
(a) p-value of m < 0.05
(b) power of test of m with respect to background-only
> some threshold
Requiring (a) alone gives the standard frequentist interval,
(CLs+b method) which has the correct coverage.
Requiring ANDed combination of (a) and (b) is more conservative;
end effect is similar to CLs, but makes more explicit the minimum
the role of minimum sensitivity (as quantified by power).
G. Cowan, RHUL Physics
Alternative to CLs
page 4
Similar to….
Feldman and Cousins touched on the same idea in connection
with FC limits:
We propose to make this more explicit using the power of the
test of a given strength parameter m with respect to the
alternative background-only hypothesis.
G. Cowan, RHUL Physics
Alternative to CLs
page 5
Formalizing the problem
In the context of tests based on the likelihood ratio l(m),
the p-value can be written
Estimator for
strength parameter
Standard normal
cumulative dist.
The upper limit is found by setting pm = a and solving for m,
G. Cowan, RHUL Physics
Alternative to CLs
page 6
False exclusion rate for no sensitivity
Excluding m if pm < a gives right coverage, but this means
that probability to exclude m in case of no sensitivity is a.
To see this note, probability to exclude m assuming m = 0 is
“No sensitivity” means m /s « . In this limit, the false exclusion
probability becomes
G. Cowan, RHUL Physics
Alternative to CLs
page 7
Power of test of m relative to m = 0
The power of a test of m relative to the alternative m = 0 is
or equivalently in terms of the distribution of mup,
G. Cowan, RHUL Physics
Alternative to CLs
page 8
Criterion for rejecting m
We formulate the criterion for rejecting a hypothesized m by
Requiring pm < a and also that the power be greater than a
minimum threshold 1 – b ′. (i.e. Type-II error rate < b ′ ). The
power-constrained limit is thus
where mb ′ is the m for which the power is b ′.
The requirement
implies
the minimum power requirement can be expressed
so
or equivalently
G. Cowan, RHUL Physics
Alternative to CLs
page 9
Choice of minimum power
Note that if the minimum power 1 – b ′ = a (typically 0.05),
then mb ′ = 0, and then mpc = mup always.
Normally would choose a < 1 – b ′ ≤ 0.5. Convention must be
discussed (also with CMS).
Coverage of power-constrained interval is well defined:
95% for mpc = mup
100% for mpc < mup
G. Cowan, RHUL Physics
Alternative to CLs
page 10
Solution in terms of median p-value
Because of the monotonic relation between the p-value
and estimator for m, the median of pm assuming m = 0 is:
In addition to the median (50% quantile) we can also find the
quantiles corresponding to the +/- Ns deviations of muHat:
G. Cowan, RHUL Physics
Alternative to CLs
page 11
Extra slides
G. Cowan, RHUL Physics
Alternative to CLs
page 12
A possible experimental outcome
Suppose a given experiment gave the following p-value versus m:
Here data have clearly fluctuation low.
G. Cowan, RHUL Physics
Alternative to CLs
page 13
Choice of likelihood ratio statistic
Ongoing discussion as to whether best to use LEP-style
likleihood ratio
or
and in both cases how to deal with the nuisance parameters.
In simple cases one obtains the same test from both statistics.
G. Cowan, RHUL Physics
Alternative to CLs
page 14