Transcript Everything You Always Wanted To Know About Limits

Everything You Always Wanted
To Know About Limits*
Roger Barlow
Manchester University
YETI06
*But were afraid to ask
Summary
Prediction
confronts data & sees
small/zero signal
Bayesian
Probability
(Health
Warning)
Frequentist
probability and
Confidence
Level language
Zero events
Few events:
Confidence belt
Roger Barlow: YETI06
Likelihood
Gaussian
ln L= -½
Extension to
several
parameters
The horrendous
case of large
backgrounds
Everything you wanted to know
about limits
Slide 2
Model predictions
Input model and
parameters
Low energy
Lagrangian
Cuts designed to
bring out signal
Feynman Rules
for Feynman
diagrams
Cross Sections
and Branching
Ratios
Monte Carlo
programs
Experiment duration,
luminosity,
Efficiency etc
Number
of events
Roger Barlow: YETI06
Everything you wanted to know
about limits
Data
Slide 3
What happens if there’s nothing
there?
Even if your analysis finds no events, this
is still useful information about the way
the universe is built
Want to say more than: “We looked for X,
we didn’t see it.”
Need statistics – which can’t prove
anything.
“We show that X probably has a mass
greater than../a coupling smaller than…”
Roger Barlow: YETI06
Everything you wanted to know
about limits
Slide 4
Probability(1): Frequentist
Define Probability of X as
P(X)=Limit N∞ N(X)/N
Examples: coins, dice, cards
For continuous x extend to Probability Density
P(x to x+dx)=p(x)dx
Examples:
• Measuring continuous quantities (p(x) often
Gaussian)
• Parton momentum fractions (proton pdfs)
Roger Barlow: YETI06
Everything you wanted to know
about limits
Slide 5
Digression: likelihood
Probability distribution of random variable x
often depends on some parameter a.
Joint function p(x,a)
Considered as p(x)|a this is the pdf. Normalised:
∫p(x)dx=1
Considered as p(a)|x this is the Likelihood L(a)
Not ‘likelihood of a’ but ‘likelihood that a would
give x’
Not normalised. Indeed, must never be
integrated.
Roger Barlow: YETI06
Everything you wanted to know
about limits
Slide 6
Limitation of Frequentist
Probability
Can’t say
“It will probably rain tomorrow.”
There is only one tomorrow. P is either 1 or 0
Have to say
“The statement ‘It will rain tomorrow.’ is
probably true.”
Can then even quantify (meteorology).
Roger Barlow: YETI06
Everything you wanted to know
about limits
Slide 7
Interpreting physics results:
Mt =173±2 GeV/c2
Can’t say
‘Mt has a 68% probability of lying between 171 and
175 GeV/c2’
Have to say
‘The statement “Mt lies between 171 and 175
GeV/c2”has a 68% probability of being true’
i.e. if you always say a value lies within its error
bars, you will be right 68% of the time.
Say “Mt lies between 171 and 175 GeV/c2” with 68%
Confidence. Or 169-177 with 95% confidence. Or…
Roger Barlow: YETI06
Everything you wanted to know
about limits
Slide 8
Interpreting null result
Your analysis searches for events. Sees none.
Use Poisson formula: P(n; )=e-n/n!
Small  could well give 0 events
•  =0.5 gives P(0)=61%
•  =1.0 gives P(0)=37%
•  =2.3 gives P(0)=10%
•  =3.0 gives P(0)=5%
If you always say ‘ 3.0’ you will be right (at least) 95%
of the time.  3.0 – with 95% confidence (a.k.a 5%
significance.)
‘If  is actually 3, or more, the probability of a
fluctuation as far as zero is only 5%, or less.’
 given by model parameters. Limit on  translates to
limit on mass, coupling,
,branching ratio or whatever
Roger Barlow: YETI06
Everything you wanted to know
Slide 9
about limits
Probability(2): Bayesian
P(X) expresses by degree of belief in X
Can calibrate against cards, dice, etc.
Extend to probability density p(x) as
before
No restrictions on X or x. Rain, MT, MH,
whatever
Interpret physics results using Bayes’
Theorem:
pposterior(a|data)  p(data|a) x pprior(a)
Roger Barlow: YETI06
Everything you wanted to know
about limits
Slide 10
Bayes at work
Zero events seen
x
=

Posterior P()


P(0 events|)
Prior: uniform
(Likelihood)
3
 P() d= 0.95
0
Roger Barlow: YETI06
Same as Frequentist limit Happy coincidence
Everything you wanted to know
about limits
Slide 11
Bayes at work again
Is that uniform prior really credible?
x
=


P(0 events|)
Posterior P()

Prior: uniform in ln 
Upper limit totally different!
3
 P() d >> 0.95
0
Roger Barlow: YETI06
Everything you wanted to know
about limits
Slide 12
Bayes: the bad news
• The prior affects the posterior. It is your choice. That
makes the measurement subjective. This is BAD. (We’re
physicists, dammit!)
• A Uniform Prior does not get you out of this.
• SUSY ‘parameter space’ is not a ‘phase space’
• Attempts to invent universally-agreed priors (‘Objective’
and/or ‘Reference’ Priors) have not worked
Better news: If there is a lot of data then the prejudicial
effects of the choice of prior can be small.
• This should ALWAYS be checked for (‘robustness under
choice of prior’.)
Roger Barlow: YETI06
Everything you wanted to know
about limits
Slide 13
Frequentist versus Bayesian?
Statisticians do a lot of work with Bayesian
statistics and there are a lot of useful ideas.
But they are careful about checking for
robustness under choice of prior.
Beware snake-oil merchants in the physics
community who will sell you Bayesian statistics
(new – cool – easy – intuitive) and don’t bother
about robustness.
Use Frequentist methods when you can and
Bayesian when you can’t (and check for
robustness.) But ALWAYS be aware which
you are using.
Roger Barlow: YETI06
Everything you wanted to know
about limits
Slide 14
A Gaussian Measurement
No problems
p(x)=exp[-(x-)2/2 2]/√2
x: symmetric
x is within ± of  with 68% probability
 is within ± of x at 68% confidence
x is above -1.645  with 95% probability
 is below x+1.645  at 95% confidence
Choice of confidence level and arrangement
Can read regions off log likelihood plot as
L(a)=exp[-(x-)2/2 2]/√2
Ln L  -(-x)2/2 2
68% region corresponds to fall of ½ from peak
Roger Barlow: YETI06
Everything you wanted to know
about limits
Slide 15
A Poisson measurement
You detect 5 events. Best value
5. But what about the
errors?
1. 5±√5=5±2.24 Assumes
e-n/n! is Gaussian in n. True
only for large  - and 5 is
small
2. Find points where log likelihood falls by ½.
Assumes e-n/n! is Gaussian in .
Gives upper error of 2.58, lower error of 1.92
Roger Barlow: YETI06
Everything you wanted to know
about limits
Slide 16
3: Doing it properly: Confidence
belt (Neyman interval) 
+
Use e-n/n!

68%
16%
For any true  the
16%

probability that (n, ) is
within the belt is 68% (or
more) by construction
For any n,  lies in [-, +]
at 68% confidence
n
Technique works for
Get upper error 3.38,
any CL, and single or
lower error 2.16
double sided
-
Roger Barlow: YETI06
Everything you wanted to know
about limits
Slide 17
Consumer guide
ln L =- ½ is a standard and easy to use. Fine
for everyday use. (Though for a simple
count the Neyman limit is quite easy)
For 90% 1-sided (upper) limit use
ln L =-0.82 (1.28 )
For 95% use ln L =-1.35 (1.645 )
Just plot the likelihood and read off the value.
Then translate back to model parameters
Roger Barlow: YETI06
Everything you wanted to know
about limits
Slide 18
Frequency method: the big
problem
Observe 5 events. Expected background of 0.9 events.
Data = signal + background
Say with 68% confidence: data in range 2.84 to 8.38
So say with 68% confidence: signal in range 1.94 to 7.48
Suppose expected* background 4.9. Or 6.9. Or 10.9 ?
“We say at 68% confidence that the number of signal
events lies between -8.06 and -2.52”
This is technically correct. We are allowed to be wrong
32% of the time. But stupid. We know that the
background happens to have a downward fluctuation
but have no way of incorporating that knowledge
*We assume that the background is calculated correctly
Roger Barlow: YETI06
Everything you wanted to know
about limits
Slide 19
Strategy 1: Bayes
Prior is uniform for positive , zero for
negative . No problem.
Get requirement (for n observed, known
background b, 90% upper limit)
0.1=nexp(-+-b) (++b)r/r!
nexp(-b) br/r!
Known as “the old PDG formula” or
“Helene’s formula” or “that heap of crap”
Roger Barlow: YETI06
Everything you wanted to know
about limits
Slide 20
Strategy 2: Feldman-Cousins
Also called* ‘the Unified Approach’
Real physicists wait to see their result and then
decide whether to quote an upper limit or a range.
This ‘flip-flopping’ invalidates the method.
They provide a procedure that incorporates it
automatically, and always gives non-stupid results.
Critics say (1) can lead to experiments quoting a range
when they’re not claiming a discovery (2) is
computationally intensive and (3) For zero observed
events, the higher the background estimate the
better (i.e. lower) the limit on signal
* By Feldman and Cousins, principally
Roger Barlow: YETI06
Everything you wanted to know
about limits
Slide 21
Strategy 3: CLs
As used by LEP Higgs working group
Generalisation of Helene formula
Some quantity Q. Could be number of
events, or something more clever
CLb=P(Q or less|b) CLs+b=P(Q or less|s+b)
CLs=CLs+b/CLb
Used as confidence level. Optimise
strategy using it and quote results
Roger Barlow: YETI06
Everything you wanted to know
about limits
Slide 22
2(+) parameters
Fix b, find 68% confidenceb
range for a, using ln L=-½
Fix a, find 68% range for b
Combination (square) has
0.682=46%
L(a,b)
a
ln L=-½ circle has 39% Confidence
Define regions through contours of log L – Confidence
content given by 2ln L= 2 for which P(2 ,n)=CL
Caution! Cannot redefine a as b+c+d, claim 3
parameters and cut with P(2,3) instead of P(2 ,1)
Roger Barlow: YETI06
Everything you wanted to know
about limits
Slide 23
Summary
Prediction
confronts data & sees
small/zero signal
Bayesian
Probability
(Health
Warning)
Frequentist
probability and
Confidence
Level language
Zero events
Few events:
Confidence belt
Roger Barlow: YETI06
Likelihood
Gaussian
ln L= -½
Extension to
several
parameters
The horrendous
case of large
backgrounds
Everything you wanted to know
about limits
Slide 24
Remember!
Zero events = 95% CL upper limit of 3 events
If it’s more involved, plot the likelihood function
and use
ln L=-½ for 68% central, etc
Be suspicious of anything you don’t understand
If you’re integrating the likelihood you are a
Bayesian. I hope you know what you’re doing.
Roger Barlow: YETI06
Everything you wanted to know
about limits
Slide 25
Further Reading
• Workshop on Confidence Limits, CERN
yellow report 2000-005
• Proc. Conf. Advanced Statistical
Techniques in Particle Physics, Durham,
IPPP02/39
• Proc. PHYSTAT03 – SLAC-R-703
• Proc PHYSTAT05, Oxford forthcoming
Roger Barlow: YETI06
Everything you wanted to know
about limits
Slide 26