Transcript Slides

Statistical Significance
& Its Systematic Uncertainties
Shan JIN
Institute of High Energy Physics (IHEP)
[email protected]
November 26, 2007
• HEP Experiments have always been at the
frontier of searching for and discovering
new signals – new particles and new
physics phenomena.
• How to quantify the possibility of a
new discovery?
– How can your results be accepted?
International Convention in
HEP Community
• 3 sigma – evidence of a possible signal
• 5 sigma – discovery of a signal
 What is statistical significance?
How to calculate/obtain RELIABLE or
RIGOROUS statistical significance?
We need common “statistical
language” to understand each other!
Outline
 Essence of Statistical Significance
and its expressions
 3 reliable/rigorous methods
calculating statistical significance and
their systematic uncertainties
Essence of Statistical Significance and
Its Expressions
• Essence：The PROBABILITY of a
statistical test on the consistency with
background hypothesis
• Expressions:
–It can be directly expressed as a
Prob., of course.
–It is more often intuitively to “be
translated into n  ” according to a
Gaussian prob. distribution.
Method I:
Frequentist Method – 1- CLb
(Widely used by LEP HiggsWG)
When we can obtain full knowledge
of background from MC simulation
cb  1  CLb  Pr ob  Pr ob( b   0 )
ε
-----Statistical Estimator
Test Statistic
0
Example: Simple Event Counting:
We expext: B=10000 events,
We observe: N0=10500 events
Pr ob( N b  N 0 )  2.8 10
7
This is the probability that we observe Nb large
than N0 in the pure background distribution.
Or we can understand it is as 5  deviation from
no signal.
Systematic Uncertainties for Method I
• In this method, all possible factors causing the
uncertainty of b should be taken into account.
Example: ALEPH’s observation of “3 golden Higgs
candidate events” with ~3.0  significance:

L ( s  b)
L(b)
Likelihood function includes number of events, mass and btagging distributions, etc
ALEPH Collaboration, PLB526 (2002) 191
Method II:
Goodness of fit -- 2 tests
(with known background shape)
p  Pr ob(  2 , d .o. f )
(CERNLIB)
d.o.f = Nbin – Npara
In statistics books, it reads “p-value it is the probability,
under the assumption of a hypothesis H0, of obtaining
data at least as incompatible with H0 as the data actually
observed.”
Sometimes，we can also” translate” this probability as
“n ” deviation from hypothesis H0.
Some examples
Observation of an anomalous enhancement near
the threshold of pp mass spectrum at BES II
BES II J/ygpp
acceptance
weighted BW
M=1859 +3 +5 MeV/c2
10 25
G < 30 MeV/c2 (90% CL)
2/dof=56/56
0
Phys. Rev. Lett. 91, 022001 (2003)
3-body phase space
0.1
0.2
M(pp)-2mp (GeV)
acceptance
0.3
Could it be a tail of a known
resonance?
0 resonances in PDG tables:
h(1760) M=1760 G = 60 MeV
p(1800) M=1801 G = 210 MeV
2/dof=323/58
2/dof=412/58
Pure FSI disfavored
I=0 S-wave FSI CANNOT fit the BES data.
FSI curve from A.Sirbirtsev
et al. ( Phys.Rev.D71:054010,
2005 ) in the fit (I=0)
 2 / d .o. f  192 / 58
FSI * PS * eff + bck
M pp  2m p
Systematic Uncertainties for Method II
• Only those that may change the background
shape need to be taken into account:
Some systematic errors, such as tracking efficiency,
photon efficiency and 4c-fit, which have very small
impact on the background shapes and the shape of
acceptance curve, can be ignored, since they have
littile contribution in the 2 calculation.
Method III:
Likelihood Ratio Tests
• This method can be applied to background
shapes obtained from sideband fit. It is widely
used by many experiments.
• Two fits:
– with signal: fit1  L1
;
–without signal: fit0  L0
Rigorous statistical theorem tell us that
 2  2 ln(
L1
)  2 ln L
L0
follow the 2 distribution with D.O.F  ( N para ) signal
So, we have:
p  PROB ( 2 , D.O.F )
(CERNLIB)
• Using TOY MC experiments, it can also be
easily shown that Δχ 2  2 ln( L1 ) follows the 2
L0
distribution with D.O.F  ( N para ) signal .
• So, when we apply likelihood ratio test
method, the number of d.o.f.
must be taken into account. D.O.F  ( N para ) signal
– For a BW-like new signal, usually we have at least
3 parameters for the signal (mass, width and
amplitude), so using 2 ln L to estimate signal
significance is incorrect and it over estimates the
significance by 0.7  when claiming a 5 
discovery, i.e., the actual significance is only 4.3 .
( The probability is more than 10 times larger
BE CAREFUL! )

Observation of X(1835) in
J y  g p p h 
Statistical Significance 7.7 
J y  g p p h 
L1
  2 ln( )
L0
2
 68.1
D.O.F  3
BESII
The p+p-h mass spectrum for h decaying into
hp+p-h and h g
Systematic Uncertainties for Method III
• The factors cause the change of 2 ln L
should be taken into account into the
systematic uncertainties.
– Since we obtain the background shape from
sideband information, so the systematic
uncertainties are mainly from the uncertainties of
different choice of fitting functions and fitting range.
– Some systematic errors, such as tracking
efficiency, photon efficiency and 4c-fit, which have
very small impact on the background shapes, can
be ignored.
Example on Systematic Uncertainty of
Significance:
Observation of Y(2175) in
J/y  hf0(980) at BESII
Fit with one resonance
• BG shape is fixed to h, f0 sideband BG
5.5 
M =2.186±0.010 0.006 GeV/c2
G=0.065±0.023  0.017 GeV/c2
N events= 5212
M(f0(980)) GeV/c2
B(J/ψ  ηY(2175)B(Y(2175)  φf 0 (980))B(f 0 (980)  π  π  ) 
(3.23  0.75( stat )  0.73( syst )) 104
Fit with one resonance
• BG is represented by a 3-order polynomial
4.9 
M =2.182±0.010 GeV/c2
G=0.073±0.024 GeV/c2
N events= 6114
B(J/ψ  ηY(2175)B(Y(2175)  φf 0 (980))B(f 0 (980)  π  π  ) 
(3.79  0.87( stat )) 104
Fit with two resonances
• BG shape is fixed to h, f0 sideband BG
• the mass and width of the second peak
are fixed to those of from BaBar.
5.8 
2.5 
M =2.186±0.010GeV/c2
G= 0.065±0.022GeV/c2
N1 events= 4714
N2 events= 2211
B(J/ψ  ηY(2175)B(Y(2175)  φf 0 (980))B(f 0 (980)  π  π  ) 
(2.92  0.87( stat )) 104
Summary
• The Essence of the statistical significance is
the PROBABILITY of a statistical test on the
consistency with background hypothesis.
It can be expressed as “n” according to a
Gaussian p.d.f.
• Three RELIABLE/RIGOROUS methods are
recommended and discussed.
The systematic uncertainty considerations on the
significance depend on different information used.
•
or S / S  B is NOT
recommended in the significance
calculation.
S/ B
Comments on statistical significance
（Plenary Talk by S.Jin at ICHEP04)
• Using S / B to estimate statistical significance seems too
optimistic. Even if we have firm knowledge on the
background  CLb as LEP Higgs used is recommended.
• When the background is estimated from the fit of sideband, the
likelihood ratio with D.O.F. taken into consideration is a better
estimator of statistical significance.
– In this case, the uncertainty of all possible
background shapes should be included in the
uncertainty of significance.
• Do not optimize/tune the cuts on the data! Determine the
cuts based on MC optimization before looking at data.
– The sys. uncertainty on significance from “bias” cut
is hard to estimate.
谢 谢！
Thanks！
A peak around 2175 MeV/c2 is observed
in J/y  hf0(980)
phase space
efficiency curve
M(f0(980)) GeV/c2
Backgrounds from
sideband estimation