Transcript Effect Sizes - faculty.ucmerced.edu

Effect Sizes
Coding
• Coding Effect Sizes
– Usually the first thing I do because you can’t
include a study in a meta-analysis if you can’t
compute an effect size from it.
– However, I will present it second here, after
• Coding Studies
• Separate coding of studies from coding of
effect sizes to reduce biased coding.
Many Kinds of Effect Sizes
• For Continuous Variables
– d (the standardized mean difference statistic)
– r (the correlation coefficient)
• For Dichotomous Variables
–
–
–
–
o (the odds ratio, for dichotomous outcomes)
l (the log odds ratio)
The relative risk
The rate difference etc.
• All have the effect of standardizing results into a
common metric.
Effect Sizes for Dichotomous
Data
The Odds Ratio
• Most widely used effect size measure for
dichotomous outcomes
AD
OR 
BC
• Where A, B, C, and D are cell frequencies
• OR = 1 is no effect.
– Lower Bound 0 (Control better than
Treatment)
– upper bound infinity (Treatment < Control)
The Original Simpson & Pearson’s
Hospital Staff Incidence Data
A Fourfold Table from
Pearson’s Hospital Staff
Incidence Data
Group
Inoculated
Not Inoculated
Total
Immune
A = 265
C = 204
N  469
Condition of Interest
Diseased
All
B = 32
M 297
D = 75
M 279
N  107
T = 576
265 * 75
OR 
 3.04
204 * 32
OR from Proportions
• The odds ratio can also be computed from
proportions
p11 p22 .4600 * .1302
OR 

 3.04
p12 p21 .3541* .0556
• And there are many variations on this
formula depending on which proportions
you do or do not know
Log Odds Ratio
• Easier to work with statistically
LOR  ln (OR )
• Makes interpretation more intuitive, similar
in some respects to d.
– 0 = no effect
– Range is +/-.
• In example, LOR = ln(3.04) = 1.11
Converting LOR to d
• May wish to do this if most of your effect
sizes are in d and just a few in OR, and
you want to pool them all:
• Cox (1970):
dCox  LOR / 1.65
• Sanchez-Meca et al (Psych Methods)
showed this approximation works well
The Relative Risk (Risk Ratio)
• Also used for dichotomous outcomes
RR  ( p11 / p1 ) /( p21 / p2 )
• Where p1+ and p2+ are the marginal
proportions of the first row and the second
row, respectively.
• Commonly converted to the Log Risk
Ratio
LogRiskRatio  ln( RR)
Example
Group
Inoculated
.5156
Not Inoculated
.4843
Total
Condition of Interest
Immune
Diseased
All
p11 = .4600
p12 = .0556
p1+ =
p21 = .3541
p22 = .1302
p+1 = .8141
p+2 = .1858
p2+ =
.4600 / .5156 .8922
RR 

 1.22
.3541 / .4843 .7312
The probability of being immune if inoculated is 1.22 times
higher than the probability if not inoculated.
Converting RR to Odds Ratio
OR  RR1  p21 / p2  /1  p11 / p1 
• Which in our example is
 1  .3541 / .4843 
OR  1.22
  3.04
 1  .4600 / .5156 
Number Needed to Treat
• How many units must be treated to produce a
successful outcome
1
1
NNT 

 9.44
R1  R0 .4600  .3541
• Where R1 is the success rate in the treatment
group and R0 is the success rate in the control
group
• That is, treat 9 units with T to obtain one more
success than would have occurred under C
• A measure of cost-effectiveness of tmt
Difference Between Proportions
(Risk Difference)
• Intuitive:
Di  pi1  pi 2
• For Pearson’s Data:
– 89.2% of those inoculated were immune
– 73.1% of those not inoculated were immute,
so
Di  .892  .731  .161
• The difference in immunity rates for those
inoculated or not is about 16.1%
Analyzing Fourfold Tables by
the Correlation
• Tetrachoric, approximated by
OR
3/ 4

 1 / OR
3/ 4

 1  .39
• Useful if binary measures created by
dichotomizing continuous ones
• Bad to use the phi coefficient (ordinary
Pearson correlation applied to binary data)
– Sensitive to marginal distributions
Preferences?
• Most tend to prefer OR or LOR
– Likelihood ratio test of the null
• But all measures for fourfold tables have
potential interpretation problems
• Reporting success rates, and NNT, advisable.
• Bonnet (2007) American Psychologist, also
Kraemer (2004) Statistics in Medicine are nice
summaries of issues in summarizing fourfold
tables
Effect Sizes for Continuous
Variables
• r (the Pearson correlation)
– Not widely used or reported in treatment
outcome studies
• d (the standardized mean difference
statistic)
– By far the most widely used
d: The Standardized Mean
Difference Statistic
XT  XC
d
,
s
• Where s is an estimate of the standard deviation
of the numerator, and standardizes the
numerator.
– It is often the pooled standard deviation (a weighted
average of the standard deviation of each group)
Estimating d
•
•
•
•
•
d itself
Algebraic equivalents to d
Good approximations to d
Methods that require intraclass correlation
Methods that require ICC and change
scores
• Methods that underestimate effect
Note: Italicized methods will be covered.
Sample Data Set I: Two
Independent Groups
Treatment
Comparison
3
2
4
2
4
4
4
5
5
5
6
6
6
6
6
7
7
8
7
9
Mean
5.2
5.4
Standard Deviation
1.398
2.319
Sample Size
10
10
Correlation between treatment and outcome is r = -.055
Calculating d
t
i
X  X ic
di 
si
Xt  Xc
5.2  5.4
d 

 .104, where
sp
1.915
sp 

( nt  1) st2  ( nc  1) sc2
nt  nc  2
(10  1)1.3982  (10  1)2.319 2
 1.915
10  10  2
Interpreting d
• Cohen suggested:
0 = no effect
.20 = small effect
.50 = medium effect
.80 = large effect
• Lipsey and Wilson found a slightly
narrower range empirically (.3, .5, .67)
• However, remember that what counts as
small, medium or large will vary from topic
to topic (e.g.,SCDs)
More on Interpreting d
• Suppose d = .51 (Shadish et al 1993
MFT)
– Convert to Cohen’s U3 index
– From the Unit Normal Curve
z
Below z Above z Between mean and z Ordinate
0.50 0.6915 0.3085 0.1915
0.3521
0.51 0.6950 0.3050 0.1950
0.3503
0.52 0.6985
0.53 0.7019
0.3015
0.2981
0.1985
0.2019
0.3485
0.3467
– Implies that a therapy client at the mean
was better off than 69.5% of control
clients;
More on Interpreting d
• Use U3 to compute an improvement in
percentile rank:
– Improvement = U3 – 50%
= 69.5% - 50%
= 19.5%
– The average treatment unit improves 19.5%
compared to the average control unit
More on Interpreting d
• Converts to a correlation coefficient of
roughly .25 (Hedges & Olkin, 1985, p.
77)
2
2
d
.51
r

 .2471
2
2
d 4
.51  4
• So that treatment accounts for about r2
= .24712 = 6% of outcome variance
More on Interpreting d
• Rosenthal and Rubin 1982: Translates
into a treatment success rate of about
62% in marital and family therapies
compared to 38% in control groups
.50  r / 2  .50  .2471/ 2  .6236
Hedges’ Correction for Small
Sample Bias
• d overestimates effect size in small
samples (< 10-15 total)
3 

d

1

• Correction is
 4 N  9  g
• I always use this correction as it never
harms estimation. In SPSS
– COMPUTE D = ES*(1-(3/((4*(NT+NC))-9))).