Transcript Effect Size and Strength of Association Measures

Empirically Based Characteristics of
Effect Sizes used in ANOVA
J. Jackson Barnette, PhD
Community and Behavioral Health
College of Public Health
University of Iowa
Examine characteristics of four
commonly used effect sizes
• Standardized Effect Size
• Measures of Association:
– Eta-Squared
– Omega-Squared
– Intraclass Correlation Coefficient
Standardized Effect Size
It represents mean differences in units of common
population standard deviation.
Population Form
1 – µ2
=

Statistic Form
X1 – X2
d=
s
In practice, the Std. Deviation is typically replaced
with the Root Mean Square Error
Standardized Effect Size in ANOVA
Mean Range
Std. Effect Size =
MSE
Cohen’s Standards
Cohen needed to base his research on power
on some effect sizes so he pretty much
arbitrarily chose three values that had been
used extensively as standards for effect sizes:
.2 is a “small effect”
.5 is a “moderate effect”
.8 is a “large effect”
Mean Standardized Effect Size by Sample Size for K=2, 4, and 10
1.4
1.2
1
0.8
SES
K=2
K=4
K=10
0.6
0.4
0.2
0
0
25
50
75
100
125
150
175
200
225
250
275
Sample Size
300
325
350
375
400
425
450
475
500
Observed Effect Sizes when K= 2
n= 5, mean= .55, sd= .47, p>.2= .76, p>.5= .44, p>.8= .24
n= 30, mean= .21, sd= .16, p>.2= .44, p>.5= .06, p>.8= .00
n= 60, mean= .15, sd= .11, p>.2= .27, p>.5= .01, p>.8= .00
n=100, mean= .11, sd= .09, p>.2= .16, p>.5= .00, p>.8= .00
Observed Effect Sizes when K= 4
n=
5, mean= .97, sd= .46, p>.2= .99, p>.5= .85, p>.8= .59
n= 30, mean= .38, sd= .16, p>.2= .87, p>.5= .22, p>.8= .01
n= 60, mean= .29, sd= .11, p>.2= .70, p>.5= .03, p>.8= .00
n=100, mean= .21, sd= .09, p>.2= .49, p>.5= .00, p>.8= .00
Observed Effect Sizes when K= 10
n= 5, mean= 1.40, sd= .40, p>.2= 1.00, p>.5= 1.00, p>.8= .96
n= 30, mean= .56, sd= .15, p>.2= 1.00, p>.5= .64, p>.8= .06
n= 60, mean= .40, sd= .10, p>.2= .99, p>.5= .15, p>.8= .00
n=100, mean= .31, sd= .08, p>.2= .92, p>.5= .00, p>.8= .00
Eta-Squared (Pearson and Fisher)
SStreatment
2=
Sstotal
A 2 of .25 would indicate that 25% of the
total variation is accounted for by the
treatment variation.
Eta-Squared
Positives: easy to compute and easy to
interpret.
Negatives: it is more of a descriptive than
inferential statistic, it has a tendency to be
positively biased and chance values are a
function of number and size of samples.
Mean Eta Squared by Sample Size for K= 2, 4, and 10
0.12
0.1
Eta Sq.
0.08
K=2
0.06
K=4
K=10
0.04
0.02
0
0
25
50
75
100
125
150
175
200
225
250
275
Sample Size
300
325
350
375
400
425
450
475
500
The Bias in Eta-Squared
Mean sampled 2
Sample Size
K
5
30
60 100
2 .110 .017 .008 .005
4 .159 .025 .013 .008
6 .173 .028 .014 .008
8 .180 .029 .015 .009
10 .183 .030 .015 .009
Omega-Squared (Hays, 1963)
When a fixed effect model of ANOVA is used, Hays
proposed more of an inferential strength of association
measure, referred to as Omega-Squared (2) to
specifically reduce the recognized bias in 2.
It provides an estimate of the proportion of variance
that may be attributed to the treatment in a fixed
design. 2 = .32 means 32% of variance attributed to
the treatment.
Omega-Squared
2 is computed using terms from the ANOVA
SStreatment – ( K – 1) MSerror
2 =
SStotal – MSerror
Mean Omega Squared by Sample Size for K=2, 4, and 10
0.002
0.0015
Omega Sq.
0.001
K=2
K=4
K=10
0.0005
0
0
50
100
150
200
250
-0.0005
Sample Size
300
350
400
450
500
Omega-Squared
Positives and Negatives (Pun intended) of 2
Positives: it is an inferential statistic that can be used
for predicting population values, easily computed, it
does remove much of the bias found in 2.
Negatives: it can have negative values, not just
rounding error type, but relatively different than 0.
If you get one that is negative, call it zero.
Intraclass Correlation
Omega-squared is used when the independent
variable is fixed. Occasionally, the
independent variable may be “random” in
which case the intraclass correlation is used to
assess strength of association.
Intraclass Correlation
Values to determine the ICC come from the ANOVA.
MStreatment – MSerror
 I=
MStreatment + ( n – 1) Mserror
The ICC is a variance-accounted-for statistic,
interpreted in the same way as is Omega-Squared. It
also has the same strengths and weaknesses.
Mean Intraclass Correlation by Sample Size for K=2, 4, and 10
0.005
0
0
50
100
150
200
250
300
350
400
450
500
-0.005
-0.01
ICC
K=2
-0.015
K=4
K=10
-0.02
-0.025
-0.03
-0.035
Sample Size