Transcript Statistics for the Social Sciences - the Department of Psychology at

Statistics for the Social Sciences
Psychology 340
Fall 2006
Effect sizes & Statistical Power
Outline
Statistics for the
Social Sciences
• Error types revisited
• Effect size: Cohen’s d
• Statistical Power Analysis
Performing your statistical test
Statistics for the
Social Sciences
Real world (‘truth’)
There really
isn’t an effect
Experimenter’s
conclusions
H0 is
correct
Reject
H0
Fail to
Reject
H0
H0 is
wrong
There
really is
an effect
Performing your statistical test
Statistics for the
Social Sciences
Real world (‘truth’)
H0 is
correct
H0 is
wrong
Performing your statistical test
Statistics for the
Social Sciences
Real world (‘truth’)
Real world (‘truth’)
H0 is
correct
H0 is
correct
H0 is
wrong
Type I
error

Type
II error
Real world (‘truth’)

H0 is
wrong
Type I
error

Type
II error

H0 is
correct
So there is only one distribution
The original (null)
distribution
H0 is
wrong
So there are two distributions
The new (treatment) The original (null)
distribution
distribution
Performing your statistical test
Real world (‘truth’)
Statistics for the
Social Sciences
H0 is
correct
Real world (‘truth’)
H0 is
wrong
Type I
error

Type
II error

H0 is
correct
So there is only one distribution
The original (null)
distribution
H0 is
wrong
So there are two distributions
The new (treatment) The original (null)
distribution
distribution
Effect Size
Real world (‘truth’)
Statistics for the
Social Sciences
• Hypothesis test tells us
whether the observed
difference is probably
due to chance or not
• It does not tell us how
big the difference is
– Effect size tells us how
much the two
populations don’t
overlap
H0 is
correct
H0 is
wrong
Type I
error

Type
II error

H0 is
wrong
So there are two distributions
The new (treatment) The original (null)
distribution
distribution
Effect Size
Statistics for the
Social Sciences
• Figuring effect size
1   2
But this is tied to the
particular units of
measurement
The new (treatment) The original (null)
distribution
distribution
– Effect size tells us how
much the two
populations don’t
overlap
2
1
Effect Size
Statistics for the
Social Sciences
• Standardized effect size
Cohen’s d
1   2
d

– Puts into neutral units for
comparison (same logic as zscores)
The new (treatment) The original (null)
distribution
distribution
– Effect size tells us how
much the two
populations don’t
overlap
2
1
Effect Size
Statistics for the
Social Sciences
• Effect size conventions
– small
– medium
– large
d = .2
d = .5
d = .8
1   2
d

The new (treatment) The original (null)
distribution
distribution
– Effect size tells us how
much the two
populations don’t
overlap
2
1
Error types
Statistics for the
Social Sciences
There really
isn’t an effect
I conclude that
there is an
effect
Real world (‘truth’)
H0 is
correct
Reject
H0
Experimenter’s
conclusions
Fail to
Reject
H0
I can’t detect
an effect
H0 is
wrong
There
really is
an effect
Error types
Statistics for the
Social Sciences
Type I error (): concluding that
there is a difference between groups
(“an effect”) when there really isn’t.
Reject
H0
Experimenter’s
conclusions
Fail to
Reject
H0
Real world (‘truth’)
H0 is
correct
H0 is
wrong
Type I
error
Type II error (): concluding that
there isn’t an effect, when there really is.

Type II
error

Statistical Power
Statistics for the
Social Sciences
• The probability of making a Type II error is related
to Statistical Power
– Statistical Power: The probability that the study will
produce a statistically significant results if the research
hypothesis is true (there is an effect)
Power  1  
• So how do we compute this?
Statistical Power
Statistics for the
Social Sciences
Real world (‘truth’)
H0 is
correct
H0 is
wrong
Real world (‘truth’) H0: is true (is no treatment effect)
Type I
error

The original (null) distribution
Type
II error

 = 0.05
Reject H0
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Real world (‘truth’)
H0 is
correct
H0 is
wrong
Real world (‘truth’) H0: is false (is a treatment effect)
Type I
error

Type
II error

The new (treatment) distribution
The original (null) distribution
 = 0.05
Reject H0
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Real world (‘truth’)
H0 is
correct
H0 is
wrong
Real world (‘truth’) H0: is false (is a treatment effect)
Type I
error

Type
II error

The new (treatment) distribution
 = 0.05
Reject H0
The original (null) distribution
 = probability
of a Type II
error
Fail to reject H0
Failing to
Reject H0, even
though there is
a treatment effect
Statistical Power
Statistics for the
Social Sciences
Real world (‘truth’)
H0 is
correct
H0 is
wrong
Real world (‘truth’) H0: is false (is a treatment effect)
Type I
error

Type
II error

The new (treatment) distribution
 = 0.05
Power = 1 - 
Probability of
(correctly)
Rejecting H0
Reject H0
The original (null) distribution
 = probability
of a Type II
error
Fail to reject H0
Failing to
Reject H0, even
though there is
a treatment effect
Statistical Power
Statistics for the
Social Sciences
• Steps for figuring power
1) Gather the needed information: mean and standard deviation
of the Null Population and the predicted mean of Treatment
Population
1  60;   2.5
2  55;   2.5
2
1
Statistical Power
Statistics for the
Social Sciences
• Steps for figuring power
2) Figure the raw-score cutoff point on the comparison distribution
to reject the null hypothesis
1  60;   2.5
From the unit normal  = 0.05
table: Z = -1.645
Transform this z-score to a
raw score
1
raw score  1   (Z)  60  (2.5)(1.645)  55.89
Statistical Power
Statistics for the
Social Sciences
• Steps for figuring power
3) Figure the Z score for this same point, but on the distribution
of means for treatment Population
2  55;   2.5
 0.355
Z
X

Remember to use the
properties of the
treatment population!
55.88  55

2.5
Transform this raw score to
a z-score
55.89
Statistical Power
Statistics for the
Social Sciences
• Steps for figuring power
4) Use the normal curve table to figure the probability of getting
a score more extreme than that Z score
 = probability
of a Type II
error
From the unit normal table:
Z(0.355) = 0.3594
 0.355
Power = 1 - 
Power  1  0.3594  0.64
55.89
The probability of detecting this an effect of this size from these
populations is 64%
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power:
 -level
– Sample size
– Population standard deviation 
– Effect size
– 1-tail vs. 2-tailed
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: -level
Change from  = 0.05 to 0.01
 = 0.05

Power = 1 - 
Reject H0
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: -level
Change from  = 0.05 to 0.01
 = 0.01
 = 0.05

Power = 1 - 
Reject H0
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: -level
Change from  = 0.05 to 0.01
 = 0.01
 = 0.05

Power = 1 - 
Reject H0
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: -level
Change from  = 0.05 to 0.01
 = 0.01
 = 0.05

Power = 1 - 
Reject H0
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: -level
Change from  = 0.05 to 0.01
 = 0.01
 = 0.05

Power = 1 - 
Reject H0
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: -level
So as the  level gets
smaller, so does the
Power of the test
Change from  = 0.05 to 0.01
 = 0.01
 = 0.05

Power = 1 - 
Reject H0
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: Sample size
Recall that sample size
is related to the spread
of the distribution
Change from n = 25 to 100
 = 0.05

Power = 1 - 

Reject H0
Fail to reject H0
X 

n
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: Sample size
Change from n = 25 to 100
 = 0.05

Power = 1 - 
Reject H0
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: Sample size
Change from n = 25 to 100
 = 0.05

Power = 1 - 
Reject H0
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: Sample size
Change from n = 25 to 100
 = 0.05

Power = 1 - 
Reject H0
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: Sample size
Change from n = 25 to 100
 = 0.05

Power = 1 - 
Reject H0
Fail to reject H0
As the sample gets
bigger, the standard
error gets smaller and
the Power gets larger
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: Population standard deviation
Change from  = 25 to 20
Recall that standard
error is related to
the spread of the
distribution 
 = 0.05

X 
Power = 1 - 
Reject H0

Fail to reject H0
n
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: Population standard deviation
Change from  = 25 to 20
 = 0.05

Power = 1 - 
Reject H0
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: Population standard deviation
Change from  = 25 to 20
 = 0.05

Power = 1 - 
Reject H0
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: Population standard deviation
Change from  = 25 to 20
 = 0.05

Power = 1 - 
Reject H0
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: Population standard deviation
Change from  = 25 to 20
 = 0.05

Power = 1 - 
Reject H0
Fail to reject H0
As the  gets smaller,
the standard error
gets smaller and the
Power gets larger
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: Effect size
Compare a small effect (difference) to a big effect
 = 0.05

Power = 1 - 
Reject H0
treatment
Fail to reject H0
no treatment
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: Effect size
Compare a small effect (difference) to a big effect
 = 0.05

Power = 1 - 
Reject H0
treatment
Fail to reject H0
no treatment
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: Effect size
Compare a small effect (difference) to a big effect
 = 0.05

Power = 1 - 
Reject H0
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: Effect size
Compare a small effect (difference) to a big effect
 = 0.05

Power = 1 - 
Reject H0
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: Effect size
Compare a small effect (difference) to a big effect
 = 0.05

Power = 1 - 
Reject H0
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: Effect size
Compare a small effect (difference) to a big effect
 = 0.05

Power = 1 - 
Reject H0
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: Effect size
Compare a small effect (difference) to a big effect
 = 0.05

Power = 1 - 
Reject H0
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: Effect size
Compare a small effect (difference) to a big effect
 = 0.05

Power = 1 - 
Reject H0
Fail to reject H0
As the effect gets
bigger, the Power
gets larger
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: 1-tail vs. 2-tailed
Change from  = 0.05 two-tailed to  = 0.05
two-tailed
 = 0.05

Power = 1 - 
Reject H0
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: 1-tail vs. 2-tailed
Change from  = 0.05 two-tailed to  = 0.05
two-tailed
 = 0.05
p = 0.025

Power = 1 - 
Reject H0
p = 0.025
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: 1-tail vs. 2-tailed
Change from  = 0.05 two-tailed to  = 0.05
two-tailed
 = 0.05
p = 0.025

Power = 1 - 
Reject H0
p = 0.025
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: 1-tail vs. 2-tailed
Change from  = 0.05 two-tailed to  = 0.05
two-tailed
 = 0.05
p = 0.025

Power = 1 - 
Reject H0
p = 0.025
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: 1-tail vs. 2-tailed
Change from  = 0.05 two-tailed to  = 0.05
two-tailed
 = 0.05
p = 0.025

Power = 1 - 
Reject H0
p = 0.025
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power: 1-tail vs. 2-tailed
Change from  = 0.05 two-tailed to  = 0.05
two-tailed
 = 0.05
p = 0.025

Power = 1 - 
Reject H0
Two tailed functionally
cuts the -level in half,
which decreases the
power.
p = 0.025
Fail to reject H0
Statistical Power
Statistics for the
Social Sciences
Factors that affect Power:
 -level: So as the  level gets smaller, so does the Power of the test
– Sample size: As the sample gets bigger, the standard error gets
smaller and the Power gets larger
– Population standard deviation: As the population standard
deviation gets smaller, the standard error gets smaller and the
Power gets larger
– Effect size: As the effect gets bigger, the Power gets larger
– 1-tail vs. 2-tailed: Two tailed functionally cuts the -level in half,
which decreases the power
Why care about Power?
Statistics for the
Social Sciences
• Determining your sample size
– Using an estimate of effect size, and population standard
deviation, you can determine how many participants
need to achieve a particular level of power
• When a result if not statistically significant
– Is is because there is no effect, or not enough power
• When a result is significant
– Statistical significance versus practical significance
Ways of Increasing Power
Statistics for the
Social Sciences