Transcript Estimating Common or Average Effects

Estimating Common or Average Effects
One goal in most meta-analyses is to examine
overall, or typical effects
We might wish to estimate and test the value of
an overall (general) parameter
H0:  = 0
Here the  could represent any of the kinds of
outcomes we listed earlier in class.
1
Estimating Common or Average Effects
Under the random-effects model, we can test
H0: θ . = 0
The average of
the i values
is zero
e.g., H0: ρ. = 0
The average of
the population
correlations is zero
Formulas for estimates of these parameters
will follow.
2
Estimating Common or Average Effects
We saw above that the fixed-effects mean for
the teacher expectancy data was
T .  0.06
Also our SPSS output showed a randomeffects mean for the TE data of
T .  0.11
*
Why are they different?
3
The FE mean is
pulled towards the
large studies with
low effects like 6,
7, and 18.
RE
FE
The RE
mean is just
a bit higher.
4
Estimating Common Effects
For the fixed-effects model, and to get Q, we
compute an effect that we will assume is common
to all studies. Our book calls this M but I prefer to
call it T . .
We use a weighted mean – weighting each data
point by the inverse of its variance (i.e., wi =
1/V(Ti) = 1/Vi ):
wT

T. 
w
i i
i
 [Ti / Vi ]

 1/Vi
5
Estimating Common Effects
Since we need the FE mean for the test of
homogeneity we can get it from SPSS code or
from pull-down analyses or from R code.
However we must decide whether to report it or
not.
6
Estimating Average Effects
For the random-effects model, we compute an
average of the effects that we believe truly vary
across studies. The model for the effect size was
Ti = θi + ei
with variance
V(Ti) = Vi* = s2θ + Vi .
We use an estimate sˆ to get new V* variances
and weights that incorporate both parts of the
variation in the effects Ti.
2
θ
7
These RE
CIs are
made using
the new RE
variances Vi*
RE
FE
The RE CIs are more equal in width and studies have
very similar influence on the mean.
8
Estimating Average Effects
For RE we weight each data point by the new RE
weight -- the inverse of its random-effects
variance:
2
ˆ
w*i = 1/[Vi + s θ ]
Thus to get the RE mean we compute
 w T  Ti /[ Vi  sˆ ]
T. 

w
1/[Vi  sˆ ]
*
*
i i
*
i
2
θ
2
θ
9
The FE variances (V)
range from about .01 to
.14. This is fairly typical.
Large n go with small V.
We added .08 to each V
value to get the RE
variance V*.
10
The ratio of the largest
weight to the smallest
one is 16:1 for the FE
weights (w) but it is only
2.5:1 for the RE weights
(wstar).
Large studies do not
have as much relative
influence under the RE
model.
11
Estimating Common Effects
We also need a variance (or standard error) for the
mean.
To compute the SE for the fixed effects case we use
the inverse variances of the individual effects – or
equivalently the weights. The variance of the fixedeffects mean is
1
1
V(T.) 

 w i 1/Vi
and the SE is the square root of the variance.
12
Estimating Common Effects
The variance (or SE) for the random-effects mean is
very similar to the FE variance
1
1
V(T. ) 

*
2
 w i 1/[Vi  sˆ ]
*
and the SE is the square root of the variance. This
variance will be larger than the FE variance because
of the addition of sˆ θ2 to each Vi
13
Random-effects variances
However we have not seen how to estimate s2
We will consider two method-of-moments
estimators. I call them SVAR and QVAR in our
programs.
SVAR is based on a “typical” sample variance of
the T’s (like you learned in intro stats class) and
QVAR is computed using Q thus it is weighted.
These are not the “best” variance estimates but
are easily obtained using SPSS.
14
Random-effects variances: SVAR
SVAR uses the sample variance of the T’s. In the
random-effects model Ti = i + ei. Therefore
V(Ti) = V(i) + V(ei)
so
V(Ti) - V(ei) = V(i)
We then get the expected values of each part
E[V(Ti)] – E[V(ei)] = E[V(i)] = s2
So our estimator is SVAR = ST2 - V
15
Random-effects variances: QVAR
QVAR is computed using Q and some sums of
the weights wi and squared weights wi2
The amount by which Q exceeds its df (for the
total Q, df=k-1) is the part due to “true”
differences, not sampling error
There is no good non-technical explanation of
how the weights work here, but the formula is
QVAR =
( Q - (k-1) )
wi - (wi2/ wi )
16
The SPSS output also gives the two values of QVAR and
SVAR.
Fixed-effects Homogeneity Test (Q)
35.8254
P-value for Homogeneity test (P)
.0074
Birge's ratio, ratio of Q/(k-1)
1.9903
I-squared, ratio 100 [Q-(k-1)]/Q
.4976
Variance Component based on Homogeneity Test (QVAR)
.0259
Variance Component based on S2 and v-bar (SVAR)
.0804
RE Lower Conf Limit for T_Dot (L_T_DOT)
-.0410
Weighted random-effects average of effect size based on SVAR (T_DOT)
.1143
RE Upper Conf Limit for T_Dot (U_T_DOT)
.2696
17
Finally we return to slides 9 and 13, and add sˆ θ to
the Vi values to get the RE mean and its SE.
2
Model
Fixed
Random
Mean
0.06
0.11
SE of Mean
0.036
0.08
CI width
0.14
0.34
The new RE mean is larger and its SE is larger. The
new SE is over twice the size of the FE standard
error.
Thus the random effects CI is also more than twice
as wide.
18
In this case we decide that the mean effect size is
not different from zero based on the CI. We can
also test H0:  .= 0 using the RE mean and SE:
Z = T.*/SE(T.*) = 0.11/0.079 = 1.329
We compare this sample Z to a critical Z value
such as ZC = +1.96 for a two-tailed test at a = .05.
This Z is not large enough to reject H0. On average
the teacher expectancy effect is about a tenth of a
standard deviation in the sample, but there is
essentially no true difference.
19
However some studies show effects much larger
than 0.11. Now we can separately interpret sˆ θ2 .
Our best estimate of the mean of the population
effects is 0.11 and their estimated variance is 0.08
(SD = .28).
This is NOT the same as the SE of the mean
effect, which was 0.079 on the previous slides.
The values of SVAR and QVAR tell us how
spread out the population of effects seems to be.
20
2
ˆ
We can use the estimate of s θ to draw a picture
of the population of effects. The estimated
variance of the population effects is 0.08 (SD =
.28). We assume a normal shape for the
population and center it on the RE mean.
If the effects were
normally
distributed – the
distribution of
TRUE effects
         

95% of the distributi on
would look like
this.
95% of the is would be between about -0.44 and
0.66.
Probability Density
Normal (Gaussian) Distribution
1.6
1.4
1.2
1
0.8
0.6
0.4
0.2
0
21
The previous slide contains excel code that
allows you to make a graph of your own data. You
need to double click the plot, then you will see a
spreadsheet. Go to the tab labeled normal.
Enter a new mean for the population (where you
see 0.11) and a new SD (replace .283), then go
back to the Chart tab.
Mean, m
Standard Deviation, s
0.11
0.283
Graph Limits
-4
4
f ( x) 
z
1
2s 2
Generate Random Normal
NORMINV(rand(),mean,standard_dev)
-0.11455
xm
s

e
( xm )2
2s 2
Cumulative Probability
xmin
8
xmax
11
Pr(x<xmin)
Pr(x>xmax)
Pr(xmin<x<xmax)
100.00%
0.00%
0.00%
22
Mixed Effects Models
If each population effect differs, and
Ti = i + ei
we may want to model or explain the variation in
the population effects. For example we may have
i = b0 + b1X1i + u’i
Substituting, we have
Ti =
Observed
effect
b0 + b1X1i
+
u’i
Effect predicted + Unexplained between-
from X value
studies variation
+
+
ei
Sampling
error
23