Transcript Distinguishability of Hypotheses

Institute for High Energy Physics, Protvino,
Russia
Institute for Nuclear Research RAS, Moscow,
Russia
Distinguishability of Hypotheses
S.Bityukov (IHEP,Protvino; INR RAS, Moscow)
N.Krasnikov (INR RAS, Moscow)
December 1, 2003
ACAT’2003
KEK, Japan
S.Bityukov
Concept
Distinguishability of hypotheses: What is it ?
Let us consider
the planned experiment for searching for new phenomenon.
Planned experiment will give information about :
(a) existence of new phenomenon (yes or no),
(b) magnitude and accuracy of measured value.
We consider the case (a) from the frequentist point of view,
namely, we will calculate the total amount of possible cases and,
after that, the amount of cases in favor of one of the statements.
This approach allows to estimate the probability of making a
correct decision and, correspondingly, the quality of planned
experiment by using the hypotheses testing. Also we can
introduce the conception of the distinguishability of
hypotheses.
December 1, 2003
ACAT’2003
KEK, Japan
S.Bityukov
Hypotheses testing
Hypothesis H0 : new physics is present in Nature
Hypothesis H1 : new physics is absent
Many researchers prefer to exchange the places of these hypotheses
a = P(reject H0 | H0 is true) -- Type I error
b = P(accept H0 | H0 is false) -- Type II error
a is a significance of the test
1- b is a power of the test
Note that a and b can be considered as random variable. If H0 is
true then a takes place and b is absent. If H1 is true then a is
absent and b takes place.
December 1, 2003
ACAT’2003
KEK, Japan
S.Bityukov
What to test ?
Suppose the probability of the realization of n events in the
experiment is described by function f(n;  ) with parameter 
Expected number of signal events in experiment is
Expected number of background events
is
s
b
Hypothesis H0 corresponds to  =  s +  b , i.e. f(n;  s +  b)
H1 corresponds to  =  b
, i.e. f(n;  b)
The Type I error a and the Type II error b allows to estimate
the probability of making a correct decision when testing H0
versus H1 with an equal-tailed test and to estimate the
distinguishability of these hypotheses H0 and H1 with an equal
probability test.
December 1, 2003
ACAT’2003
KEK, Japan
S.Bityukov
The probability of making a correct decision in hypotheses testing
Let us consider the random variable k = a + b =
^
where the estimator
^
k=
 =
^
^
Here a , b
k  ,
^
a b
2
^
^
is a constant term and
^
a b
is a stochastic
2
term.
are the estimators of Type I (a) and Type II (b ) errors. In
^
^
the case of applying the equal-tailed test a = b the stochastic term is
equal to 0 independently of whether H0 or H1 is true. Hence the estimator
^
k can be named the probability of making incorrect choice in favor
^
of one of the hypotheses. Correspondingly, 1  k is the probability to
make a correct decision in hypotheses testing.
December 1, 2003
ACAT’2003
KEK, Japan
S.Bityukov
^
k
: advantages and disadvantages
Advantages
+1° This probability is independent of whether H0 or H1 is true
+2° In the case of discrete distributions the error  of this estimator
can be taken into account
+3° This is an estimator of quality of planned experiment
…
However, the probability of making a correct decision has
disadvantages to be the measure of distinguishability of hypotheses.
-1°
-2°
-3°
…
Disadvantages
Non minimal estimation of possible error in hypotheses testing
The region of determination is [0, 0.5] (desirable area [0, 1])
Difficulties in applying of equal-tailed test for complex
distributions
Goal: disadvantages ---> advantages
December 1, 2003
ACAT’2003
KEK, Japan
S.Bityukov
Distinguishability of hypotheses
-1° --> + 4° The applying of the equal probability test gives the
minimal half-sum of estimators of Type I error a and Type II b
by the definition of this test. The critical value n0 is chosen by the
condition f(n0;  b) = f(n0;  s +  b) .
December 1, 2003
ACAT’2003
KEK, Japan
S.Bityukov
The relative number of incorrect decisions under equal probability test
^
~
-2° --> + 5° The transformation
makes a good candidate to be a
measure of distinguishability
k =
^
k
^
1k
=
^
a b
^
^
2  (a  b )
It is a relative number of incorrect decisions.
The region of determination is [0, 1] .
-3° --> + 6° The applicability of equal probability test to the
complex distributions is obviously.
December 1, 2003
ACAT’2003
KEK, Japan
S.Bityukov
The equal probability test : the critical value for Poisson
distribution
The applying of the equal probability test for Poisson distribution
gives the critical value


s
n0 = 
,

 ln(  s   b )  ln  b 
where square brackets mean the the integer part of a number.
Note that the critical value conserves the liner dependence on
the time of measurements


st
n0 t = 
.

 ln(  s   b )  ln  b 
December 1, 2003
ACAT’2003
KEK, Japan
S.Bityukov
The estimation of the hypotheses distinguishability
The relative number of correct decisions under equal probability test
is a measure of distinguishability of hypotheses at given  s and  b.
The magnitude of this value can be found from equations



s
n 0 = 

ln(



)

ln

s
b
b 


n0
^
a =  f (i;  s   b )

i =0

n0
 ^
 b = 1   f (i ;  b )

i =0
^
^

~
a b

^
^
1  k = 1 

2  (a  b )

December 1, 2003
ACAT’2003
KEK, Japan
S.Bityukov
Dependence of relative number correct decisions on the measurement
time
December 1, 2003
ACAT’2003
KEK, Japan
S.Bityukov
Gamma- and Poisson distributions
Let us consider the Gamma-distributions with probability density

x
x a 1 e b
g x ( b , a ) = b a (a )
If to redefine the parameters and variable 1/b, a and x via
a, n+1, , correspondingly, the probability density will be
g
n
( a,  ) =
a
n 1
 e
n
 a
(n  1)
If a = 1 then the probability density looks like Poisson probabilities
g
December 1, 2003
( ; n) =
 ne
ACAT’2003
n!
,   0, n  1.
KEK, Japan
S.Bityukov
Conditional probability density
The probability density g( ;N) of the true value of the parameter of
Poisson distribution to be  in the case of the single observation N
has Gamma distribution , where
 N e
g(; N ) =
N!
It follows from
identity
n 
1 e

n!
n = N 1


1

2
1
N 
n 
2 e
 e
d  
N!
n!
n =0
N

2
=1
We checked the statement about g(;N) by the Monte Carlo and
these calculations give the confirmation of given supposition.
As a result we can take into account the statistical uncertainties by
the simple way (see, NIM A502(2003)795; JHEP09(2002)060).
December 1, 2003
ACAT’2003
KEK, Japan
S.Bityukov
Statistical uncertainties in determination of s
and b
Suppose that the expected number of signal events s and expected
background b are determined with statistical errors.
If we can reconstruct the distribution g(; N ) of the true value of the
parameter  of the function f(n; ) in the case of the observed N
events then we can determine the relative number of correct
decisions: 
0

^
a = 0 g (  ;  b   s )  f ( n;  ) d
n =0

n0
^


1  b = 0 g (  ;  b )  f ( n;  ) d
n =0

^
^

~
a b
1  k = 1 
^
^

2  (a  b )

n
December 1, 2003
ACAT’2003
KEK, Japan
S.Bityukov
Conclusions
The probability language in terms of a and b is more acceptable
for estimation of the quality of planned experiments than the
language of standard deviations.
The proposed approach allows to use the relative number of correct
decisions in the equal probability test as a measure of
distinguishability of hypotheses.
This approach gives the easy way for including of the systematics
and statistical uncertainties into the estimation of distinguishability
of hypotheses.
December 1, 2003
ACAT’2003
KEK, Japan
S.Bityukov