Transcript The Effect Size

The Effect Size
• The effect size (ES) makes meta-analysis
possible
• The ES encodes the selected research
findings on a numeric scale
• There are many different types of ES
measures, each suited to different research
situations
• Each ES type may also have multiple
methods of computation
Practical Meta-Analysis -- D. B. Wilson
Examples of Different Types of
Effect Sizes (ES)
• Standardized mean difference
– Group contrast research
• Treatment groups
• Naturally occurring groups
– Inherently continuous construct
• Odds-ratio (OR)
– Group contrast research
• Treatment groups
• Naturally occurring groups
– Inherently dichotomous construct
• Correlation coefficient
– Association between variables research
Practical Meta-Analysis -- D. B. Wilson
Examples of Different Types of
Effect Sizes
• Risk-ratio or Relative Risk (RR)
– Group differences research
(naturally occurring groups)
– Commonly used by epidemiologist and
medical meta-analyses
– Inherently dichotomous construct
– Easier to interpret than the odds-ratio (OR)
– Does not overstate ES like OR does.
Practical Meta-Analysis -- D. B. Wilson
Examples of Different Types of
Effect Sizes
• Proportion
– Central tendency research
• HIV/AIDS prevalence rates
• Proportion of homeless persons found to be
alcohol abusers
• Standardized gain score
– Gain or change between two measurement
points on the same variable
• Cholesterol level before and after completing a
therapy
• Others?
Practical Meta-Analysis -- D. B. Wilson
What Makes Something an Effect Size
for Meta-analytic Purposes
• The type of ES must be comparable across
the collection of studies of interest
• This is generally accomplished through
standardization
• Must be able to calculate a standard error for
that type of ES
– The standard error is needed to calculate the ES
weights, called inverse variance weights (more on
this later)
– All meta-analytic analyses are weighted
“averages” (simple example on next slide)
Practical Meta-Analysis -- D. B. Wilson
Weighted “Averages”
Example: Calculating GPA’s
Practical Meta-Analysis -- D. B. Wilson
Weighted “Averages”
Example: Calculating GPA’s
Practical Meta-Analysis -- D. B. Wilson
Weighted “Averages”
Example: Calculating GPA’s
Practical Meta-Analysis -- D. B. Wilson
Weighted “Averages”
Example: Calculating GPA’s
Practical Meta-Analysis -- D. B. Wilson
The Standardized Mean Difference
X X
ES 
s
G1
G2
s pooled
s12 n1  1  s22 n2  1

n1  n2  2
pooled
• Represents a standardized group contrast on
an inherently continuous measure
• Uses the pooled standard deviation (some
situations use control group standard
deviation)
• Commonly called “d” or occasionally “g”
• Cohen’s d (see separate short lecture on Cohen’s d)
Practical Meta-Analysis -- D. B. Wilson
The Correlation Coefficient (r)
ES  r
• Represents the strength of linear association
between two inherently continuous measures
• Generally reported directly as “r”
(the Pearson product moment correlation)
Practical Meta-Analysis -- D. B. Wilson
The Odds-Ratio (OR)
• Recall, the odds-ratio is based on a 2 by 2
contingency table, such as the one below
Frequencies
o
o
Success
Failure
Treatment Group
a
b
Control Group
c
d
ad
ES 
bc
The Odds Ratio (OR) is the odds for “success” in the
treatment group relative to the odds for “success” in
the control group.
OR’s can also come from results of logistic regression
analysis, but these would difficult to use in a metaanalysis due to model differences.
Practical Meta-Analysis -- D. B. Wilson
Relative Risk (RR)
• The relative risk (RR) is also based on data
from a 2 by 2 contingency table, and is the
ratio of the probability of success (or failure)
for each group
a / ( a  b)
ES 
c / (c  d )
Practical Meta-Analysis -- D. B. Wilson
Unstandardized Effect Size Metric
• If you are synthesizing a research domain
that using a common measure across
studies, you may wish to use an effect size
that is unstandardized, such as a simple
mean difference.
• Multi-site evaluations or evaluation
contracted by a single granting agency.
Practical Meta-Analysis -- D. B. Wilson
Effect Size Decision Tree for Group
Differences Research (from Wilson & Lipsey)
All dependent variables are
inherently dichotomous
Difference between proportions
as ES [see Note 1]
Group contrast on dependent variable
Odds ratio; Log of the odds ratio
as ES
All dependent variables are
inherently continuous
All dependent variables measured
on a continuous scale
All studies involve same
measure/scale
Unstandardized mean difference
ES
Studies use different
measures/scales
Standardized mean difference
ES
Some dependent variables measured
on a continuous scale, some
artificially dichotomized
Some dependent variables are
inherently dichotomous, some are
inherently continuous
Standardized mean difference ES;
those involving dichotomies
computed using probit [see Note 2]
or arcsine [see Note 3]
Do separate meta-analyses for
dichotomous and continuous
variables
Practical MetaAnalysis Wilson &
Lipsey
Methods of Calculating the
Standardized Mean Difference
• The standardized mean difference
probably has more methods of
calculation than any other effect size
type.
Practical Meta-Analysis -- D. B. Wilson
Poor
Good
Great
Degrees of Approximation to the ES Value
Depending of Method of Computation
–
–
–
–
Direct calculation based on means and standard deviations
Algebraically equivalent formulas (t-test)
Exact probability value for a t-test
Approximations based on continuous data (correlation
coefficient)
– Estimates of the mean difference (adjusted means, regression b
weight, gain score means)
– Estimates of the pooled standard deviation (gain score
standard deviation, one-way ANOVA with 3 or more groups,
ANCOVA)
– Approximations based on dichotomous data
Practical Meta-Analysis -- D. B. Wilson
Methods of Calculating the
Standardized Mean Difference
(Independent Samples)
Direction Calculation Method
ES 
X1  X 2
X1  X 2

s pooled
s12 (n1  1)  s22 (n2  1)
n1  n2  2
Practical Meta-Analysis -- D. B. Wilson
Methods of Calculating the
Standardized Mean Difference
(Independent Samples)
Algebraically Equivalent Formulas:
n1  n2
ES  t
n1n2
ES 
F (n1  n2 )
n1n2
independent samples t-test
two-group one-way ANOVA
Exact p-values from a t-test or F-ratio can be converted
into t-value and the above formula applied.
Practical Meta-Analysis -- D. B. Wilson
Methods of Calculating the
Standardized Mean Difference
A study may report a grouped frequency distribution from
which you can calculate means and standard deviations and
apply to direct calculation method.
𝑋=
𝑘
𝑖=1 𝑓𝑖 𝑋𝑖
𝑘
𝑖=1 𝑓𝑖
𝑠2 =
𝑘
2
𝑓
𝑋
−
𝑋
𝑖=1 𝑖 𝑖
𝑘
𝑖=1 𝑓𝑖 − 1
Practical Meta-Analysis -- D. B. Wilson
Methods of Calculating the
Standardized Mean Difference
(Independent Samples)
Close Approximation Based on Continuous Data:
Point-Biserial Correlation - For example, the correlation
between treatment/no treatment and outcome measured
on a continuous scale.
ES 
2r
1 r
2
Point-Biserial Correlation: Pearson’s Product Moment Correlation (r)
between the response (Y) and group indicator (X) coded as: Group 1 =
0, Group 2 = 1 and treated as a numeric variable.
Practical Meta-Analysis -- D. B. Wilson
Methods of Calculating the
Standardized Mean Difference
(Independent Samples)
Estimates of the Numerator of ES –
The Mean Difference
– difference between covariance adjusted
means
– unstandardized regression coefficient (b) for
group membership
Practical Meta-Analysis -- D. B. Wilson
Methods of Calculating the
Standardized Mean Difference
(Independent Samples, more than two groups)
Estimates of the Denominator of ES Pooled Standard Deviation
s
MS

F
between
pooled
MS between 
X
2
j
nj 
 MS
( X j n j ) 2
k 1
n
j
error
one-way ANOVA
more than 2 groups
𝑴𝑺𝒃𝒆𝒕𝒘𝒆𝒆𝒏 = 𝑴𝑺𝑻𝒓𝒆𝒂𝒕𝒎𝒆𝒏𝒕
should be found in an
ANOVA table in the
paper
Methods of Calculating the
Standardized Mean Difference
Estimates of the Denominator of ES - Standard Deviation (𝑠𝑑 )
of the Paired Differences (Gain Scores)
s  SE n  1
d
SE = standard error of the mean
the paired differences
Paired difference between scores = gain scores
Gain = pre-test – post-test (or vise versa)
Practical Meta-Analysis -- D. B. Wilson
Methods of Calculating the
Standardized Mean Difference
Estimates of the Denominator of ES -Pooled Standard Deviation
s pooled 
s gain
2(1  r )
standard deviation of gain scores,
where r is the correlation between
pretest and posttest scores
Practical Meta-Analysis -- D. B. Wilson
Methods of Calculating the
Standardized Mean Difference
Estimates of the Denominator of ES -Pooled Standard Deviation
s pooled
MS error df error  1


2
1  r df error  2
ANCOVA, where r is the correlation
between the covariate and the
dependent variable.
Practical Meta-Analysis -- D. B. Wilson
Methods of Calculating the
Standardized Mean Difference
Estimates of the Denominator of ES -Pooled Standard Deviation
s pooled
SS B  SS AB  SSW

df B  df AB  dfW
A two-way factorial ANOVA
where B is the irrelevant factor
and AB is the interaction
between the irrelevant factor
and group membership (factor A).
Practical Meta-Analysis -- D. B. Wilson
Methods of Calculating the Standardized
Difference between Two Proportions
Approximations Based on Dichotomous Data
ES  probit ( p group1 )  probit ( p group2 )
- the difference between the probits transformation
of the proportion successful in each group
- converts proportion into a z-value
Practical Meta-Analysis -- D. B. Wilson
Methods of Calculating the
Standardized Mean Difference
Approximations Based on Dichotomous Data
ES 
3

log Odds  ratio 
this represents the rescaling of the logged odds-ratio
(see Sanchez-Meca et al 2004 Psychological Methods article)
Practical Meta-Analysis -- D. B. Wilson
29
Methods of Calculating the
Standardized Mean Difference
Approximations Based on Dichotomous Data
2
ES  2
N 2
2r
ES 
1 r
chi-square must be based on
a 2 by 2 contingency table
(i.e., have only 1 df)
phi coefficient
2
Practical Meta-Analysis -- D. B. Wilson
Practical Meta-Analysis -- D. B. Wilson LINKED TO LECTURE SECTION OF COURSE WEBSITE
Practical Meta-Analysis -- D. B. Wilson
Formulas for the Correlation
Coefficient (r)
• Results typically reported directly as a
correlation.
• Any data for which you can calculate a
standardized mean difference effect size,
you can also calculate a correlation type
effect size.
Practical Meta-Analysis -- D. B. Wilson
Formulas for the Odds Ratio
• Results typically reported in one of three
forms:
– Frequency of successes in each group
– Proportion of successes in each group
– 2 by 2 contingency table
Practical Meta-Analysis -- D. B. Wilson
Data to Code Along With the ES
• The effect size (ES)
–
–
–
–
•
•
•
•
•
•
May want to code the statistics from which the ES is calculated
Confidence in ES calculation
Method of calculation
Any additional data needed for calculation of the inverse
variance weight
Sample size
ES specific attrition
Construct measured
Point in time when variable measured
Reliability of measure
Type of statistical test used
Practical Meta-Analysis -- D. B. Wilson
Issues in Coding Effect Sizes
• Which formula to use when summary
statistics are available for multiple formulas
• Multiple documents/publications reporting
the same data (not always in agreement)
• How much guessing should be allowed
– sample size is important but may not be
presented for both groups
– some numbers matter more than others
Practical Meta-Analysis -- D. B. Wilson