Transcript Slides

Bonferonni correction+
Adapted from presentation of
Рубанович А.В.
Experiments in finding people with paranormal
powers:
Joseph Rhine (1950)
1000 people guessed the sequence of 10 cards: red
or black?
12 persons guessed 9 of 10 cards, two of them all 10 cards
All these “physics” in further experiments did’t confirm their
paranormal abilities
Problems of «multiple comparisons» ?
 Genome-wide association: gene expression studies with
DNA chips – 500 000 SNP.
For the significance
0.01 we can large
expect up
to 5000 falseto
Multiple
testing islevel
dangerous:
probability
associations
find false association!
 Meta-studies: joining and comparison of different
results obtained by different authors
Let us generate How
two identically
distributed samples
it happens?
with
100 personsof
with
20-locus
genotypes
Appearance
false
associations
Gene
111
222
333
444
555
666
777
888
999
10
10
10
11
11
11
12
12
12
13
13
13
14
14
14
15
15
15
16
16
16
17
17
17
18
18
18
19
19
19
20
20
20
Cases
Sample 1
gen case
case
gen
gen
case
6
14
710
12
12
10
12
8
15
17
12
9
8
13
10
9
5
12
11
9
9
713
10
7
12
10
8
10
12
14
8
12
13
14
7
10
914
12
12
17
13
14
998
15
812
10
10
7
815
7
6
11
14
912
9
17
10
10
11
11
11
8
10
13
10
15
16
7
11
11
12
Should be
OR=1
Controls
Sample 2
gen control
control
gen
gen
control
7
897
9
5
210
7
14
513
9
12
16
9
8
11
13
11
10
13
6
410
12
12
9811
9
9
812
14
12
11
9
12
910
9
510
14
7
910
13
9
8910
6
96
12
11
10
716
9
14
7
13
9
7
810
10
10
13
10
6
865
23
41
OR
Odd
OR real
real
OR
OR
real
0.85
2.2
0.87
1.1
1.4
2.6
5.4
1.2
1.2
1.1
3.9
0.91
1.0
0.64
1.1
0.58
1.0
0.61
1.1
0.83
0.80
0.89
1.0
0.68
1.0
1.2
3.3
0.81
0.70
0.81
1.6
1.6
0.88
1.4
1.1
1.9
0.46
0.81
1.3
0.73
1.4
1.0
1.8
1.5
0.78
1.6
1.9
0.61
1.2
2.3
0.88
0.74
1.0
0.76
1.8
1.0
1.2
1.0
1.2
1.2
1.1
0.52
0.89
2.7
1.1
0.68
1.6
0.83
1.2
1.1
1.2
1.3
1.3
1.0
1.2
1.7
1.4
1.9
1.9
1.6
p –w/o association OR=1
Ratio
ppp
0.782
0.127
0.796
0.819
0.513
0.0896
0.0209
0.670
0.796
0.853
0.0105
0.841
1.00
0.371
0.841
0.239
1.00
0.405
0.835
0.683
0.655
0.819
1.00
0.467
1.00
0.782
0.0455
0.670
0.491
0.670
0.371
0.297
0.808
0.513
0.841
0.201
0.127
0.670
0.549
0.513
0.513
1.00
0.178
0.394
0.637
0.297
0.285
0.670
0.0881
0.808
0.532
1.00
0.617
0.221
1.00
0.782
1.00
0.655
0.695
0.835
0.162
0.819
0.0412
0.819
0.414
0.346
0.683
0.655
0.827
0.796
0.637
0.532
1.00
0.705
0.239
0.564
0.225
0.225
0.371
Significant!
All 3 loci are
Associated with
a disease!
How to avoid false associations?
Applying m independent statistical tests with significance level a, a
probability of at least one false association should be
1-(1-a)m < 0.05
Carlo Bonferroni (1935):
When applying m independent statistical test, only significant results
are results with
Bonferroni correction kills the significance of certain
results:
Two mutations associated with the disease
Control
(100)
Cases
(100)
OR
p
Mutation 1
1
8
8,61
0,044
Mutation 2
5
15
3,35
0,024
But adjusted by Bonferroni it should be:
p < 0,05/2=0,025
1 against 8 with equal size samples :
Example to
compute OR
case_mut1=matrix(1,8,1)
case_non_mut1=matrix(0,92,1)
control_mut1=matrix(1,1,1)
control_non_mut1=matrix(0,99,1)
data=rbind(case_mut1,case_non_mut1,control_mut1,control_non_mut1)
res=rbind(matrix(1,100,1),matrix(0,100,1))
mylogit<- glm(as.formula(res~data), family=binomial(link="logit"),
na.action=na.pass)
exp(mylogit$coefficients[2])
summary(mylogit)[["coefficients"]][,"Pr(>|z|)"]
case_mut1=matrix(1,15,1)
case_non_mut1=matrix(0,85,1)
control_mut1=matrix(1,5,1)
control_non_mut1=matrix(0,95,1)
data=rbind(case_mut1,case_non_mut1,control_mut1,control_non_mut1)
res=rbind(matrix(1,100,1),matrix(0,100,1))
mylogit<- glm(as.formula(res~data), family=binomial(link="logit"),
na.action=na.pass)
exp(mylogit$coefficients[2])
summary(mylogit)[["coefficients"]][,"Pr(>|z|)"]
Assessment of individual sensitivity to ionizing radiation and
DNA repair efficiency in a healthy population
F. Marcona, C. Andreoli, et al. Mut. Res., 541 (2003)
Not significant! According to
Bonferroni shoud be:
Genotypes
High-Throughput Detection of GST Polymorphic Alleles in a Pediatric Cancer Population
P. Barnette, R. Scholl, et al. Cancer Epidemiology, Biomarkers & PreventionVol. 13, 304–313, 2004
Control
13 genotypes
OR=6,4
P=0,007
8 diseases
Homozygocity in GST prevents cancer!
OR=2,3
P=0,018
Not significant!
Bonferroni correction requests:
Bonferroni method creates more problems than it solves
(Thomas Perneger, 1998):
 Bonferroni correction leads to very high probability to miss
proper association!
“Bonferroni adjustments are, at best, unnecessary and,
at worst, deleterious to sound statistical inference…”
Errors by statistical testing
Null hypothesis – usually about absence of differences in two
… and is not taking care about
samples
the possibility to miss discovery (Type II Error)
Type I Error
a biologist is trying to
Probability to Traditionally
reject null hypothesis=probability
to avoid
find
Typethere
I error,
differences where
are i.e.
anyto= guarantee avoidance of
False discoveries
Probability of false discovery
Type II Error
Probability to accept wrong null hypothesis
= Probability not to find existing differences = Probability
to miss proper discovery
Test power = 1- Type II error =
Probability to reject correctly null hypothesis = Probability to
make a discovery
Dependence of Type II error on number of tests using
the Bonferroni correction
Probability to miss gene with OR=2.7
with sample sizes 100 (case) and 100 (control)
error
Type II II
Ошибка
рода
With 1000,8
comparisons to guarantee avoidance of 1 false
discovery, we miss 88% proper discoveries!
0,6
For m=100 the probability of error
is 0.88
In single
test a5probability
to we miss 50%
With
comparisons
of isdiscoveries
miss the discovery
0.2
0,4
0,2
0
0 1
5
10
15
Number of
tests
Число
тестов
20
New algorithm to test statistical hypothesis:
FDR-control
False Discovery Rate control: Benjamini, Hochberg (1995))
Probability of false discovery < Significance level
Type I Error < 0.05
Traditional principle
is replaced by
>105 papers in
Average fraction of false discoveries < Significance level chosen
Algorithm of FDR control
(Benjamini, Hochberg, 1995)
 Order tests according to p-value :
p1 < p2 < … < pm.
Order
number oflevel
Significance
 For FDR
control
on
α
level
(e.g.
0.05),
gene
P-value for j-th test
required
we find (gene)
j 

j*  max  j : p j   
m 


Total number of tests
significant(genes)
Differences are assumed to be
for j = 1, …, j*.
 For j > j* differences are assumed not to be
significant
Example: multiple comparisons on 10 tests
Test
pi
1
0,001
2
0,0055
3
4
5
6
7
8
9
10
Order tests FDR
in ascending
Bonferroni
of p-value
Significant
Correction ordercorrection
corrections
0,005
0,005
after FDR control
0,005
0,010
0,005
0,015
0,01
In first cell
Bonferonni correction
Bonferroni
p-value
0,005
0,020
0,015
In second
leaves only first value
0,005
0,025
0,02
two times larger
Three times larger
0,005
0,030
0,04
and so on …. Significant p-values
0,005
0,035
0,3
without correction
That’s0,040
it!!!
0,5 For 6th0,005
test
0,005
0,045
0,6 is larger
p-value
than FDR
0,8
0,005
0,050
Example: expression of 3051 genes in leykomia
Golub T.R. Molecular classification of cancer: class discovery and class
prediction by gene expression monitoring. // Science. 2001, v.286.
Number of genes
with this level of t-statistics
t-statistics for the comparison
of gene expression in healthy
and ill patients



t-test: 1045 genes, for which p<0.05
Bonferroni correction: 98 genes with p’<0.000016
FDR: 681 genes, for which FDR< 0.05