Transcript Significance and Meaningfulness

1
Significance and
Meaningfulness
Effect Size & Statistical
Power
1. Effect Size
1
How “meaningful” is the
significant difference?
KNR 445
Statistics
Effect sizes
Slide 3
Significance vs. meaningfulness
 As sample size increases, likelihood of significant
difference increases
The fact that this sample
size is buried down here
in the denominator of
the test statistic means
that as n  , p  0. So
if your sample is big
enough, it will generate
significant results
t 
2
X1  X 2
SE X 1  X 2
SE X 1  X 2 
SE X 
1
SE X 1  SE X 2
SD s ample
n
KNR 445
Statistics
Effect sizes
Slide 4
Significance vs. meaningfulness
 As sample size increases, likelihood of significant
difference increases
 So statistical difference does not always mean
important difference
 What to do about this?
 Calculate a measure of the difference that is
1
standardized to be expressed in terms of the
variability in the 2 samples, but independent of
sample size
 = EFFECT SIZE
KNR 445
Statistics
Effect sizes
Slide 5
Effect Size
 EFFECT SIZE - FORMULA
1
d 
X1  X 2
SD pooled
2

X1  X 2
SS 1  SS 2
n1  n 2  2
KNR 445
Statistics
Effect sizes
Slide 6
Effect Size
 EFFECT SIZE – from SPSS
 Using appendix B data set 2, and submitting DV salary
to test of difference across gender, gives the following
output (squashed here to fit):
T-Test
1
Group Statistics
SALAR Y
SEX
male
N
female
Mean
Std. Deviation
Std. Error
Mean
6
36833.33
19913.9817
8129.8490
6
32500.00
14110.2799
5760.4977
Independent Samples Test
Levene's Test for
Equality of Variances
F
SALAR Y
Equal variances
as sumed
Equal variances
not assumed
Sig.
.011
.918
t-tes t for Equality of Means
t
df
Sig. (2-tailed)
95% C onfidence
Interval of the
Difference
Mean
Difference
Std. Error
Difference
Lower
Upper
.435
10
.673
4333.3333
9963.8235
-17867.4
26534.12
.435
9.010
.674
4333.3333
9963.8235
-18202.6
26869.31
KNR 445
Statistics
Effect sizes
Slide 7
Effect Size
1
 EFFECT SIZE – from SPSS
T-Test
Mean
difference
to use
Group Statistics
SALAR Y
SEX
male
N
Mean
female
Std. Deviation
Std. Error
Mean
6
36833.33
19913.9817
8129.8490
6
32500.00
14110.2799
5760.4977
SD’s to
pool
Independent Samples Test
Levene's Test for
Equality of Variances
F
SALAR Y
Equal variances
as sumed
Equal variances
not assumed
Sig.
.011
.918
t-tes t for Equality of Means
t
df
Sig. (2-tailed)
95% C onfidence
Interval of the
Difference
Mean
Difference
Std. Error
Difference
Lower
Upper
.435
10
.673
4333.3333
9963.8235
-17867.4
26534.12
.435
9.010
.674
4333.3333
9963.8235
-18202.6
26869.31
KNR 445
Statistics
Effect sizes
Slide 8
Effect Size
 EFFECT SIZE – from SPSS
1
SD est 
So…
SS sample
n 1
d 
2
so
SS sample  ( n  1) SD sample
2
Mean diff
( n1  1)SD  ( n 2  1)SD
2
1
n1  n 2  2
2
2
KNR 445
Statistics
Effect sizes
Slide 9
Effect Size
 EFFECT SIZE – from SPSS
d 
1
Mean diff
( n1  1)SD 1  ( n 2  1)SD 2
2
2
n1  n 2  2
Substituting…
d 
2
4333 . 33
( 5)19913 . 98  ( 5)14110 . 28
2
10
2
KNR 445
Statistics
Effect sizes
Slide 10
Effect Size
 EFFECT SIZE – from SPSS
d 
4333 . 33
( 5)19913 . 98  ( 5)14110 . 28
2
10
Calculating…
1
d 
4333 . 33
17257 . 85
 0 . 25
2
KNR 445
Statistics
Effect sizes
Slide 11
1
2
Effect Size
 From Cohen, 1988:
 d = .20 is small
 d = .50 is moderate
 d = .80 is large
 So our effect size of .25 is small, and concurs on
this occasion with the insignificant result
 The finding is both insignificant and small
 (a pathetic, measly, piddling little difference of no consequence
whatsoever – trivial and beneath us)
1
2
3
4
2. Statistical Power
Maximizing the likelihood of
significance
KNR 445
Statistics
Effect sizes
Slide 13
Statistical Power
 The likelihood of getting a significant relationship
when you should (i.e. when there is a relationship
in reality)
1
 Recall from truth table, power = 1 - 
Truth Table
Reality (unknown)
Null True
Null False
Accept Null

Type II error
(β)
Reject Null
Type I error
(α)
Power = 1 - β
Decision
(1- type II error)
KNR 445
Statistics
Effect sizes
Slide 14
Factors Affecting Statistical Power
The big ones:
 Effect size (bit obvious)
1
 Select samples such that difference between them is
maximized
 Combines the effects of sample SD (need to decrease)
and mean difference (need to increase)
2
 Sample size
 Most commonly discussed: as n increases, SEM
decreases, and test statistic then increases
KNR 445
Statistics
Effect sizes
Slide 15
1
Factors Affecting Statistical Power
The others:
 Level of significance
 Smaller , less power
 Larger , more power
2
3
 1-tailed vs. 2-tailed tests
 With good a priori info (i.e. research literature),
selecting 1-tailed test increases power
 Dependent samples
 Correlation between samples reduces standard error,
and thus increases test statistic
KNR 445
Statistics
Effect sizes
Slide 16
1
Calculating sample size a priori
1. Specify effect size
2. Set desired level of power
3. Enter values for effect size and power in
appropriate table, and generate desired sample
size:
 Applet for calculating sample size based on above:
http://www.stat.uiowa.edu/~rlenth/Power/
 Applets for seeing power acting (and interacting) with sample
size, effect size, etc…
http://statman.stat.sc.edu/~west/applets/power.html
http://acad.cgu.edu/wise/power/powerapplet1.html
http://www.stat.sc.edu/%7Eogden/javahtml/power/power.html