Transcript No Slide Title

Meta-analysis &
psychotherapy outcome research
Meta-analysis
Overview
• What is a meta-analysis?
• How is a meta-analysis conducted?
• Robinson et al, 1990: Is psychotherapy effective as a
treatment for depression?
Meta-analysis
What is a meta-analysis?
• "Meta-analysis refers to the analysis of analyses . . . the
statistical analysis of a large collection of analysis results
from individual studies for the purpose of integrating the
findings. It connotes a rigorous alternative to the casual,
narrative discussions of research studies [= traditional
review papers] which typify our attempts to make sense of
the rapidly expanding research literature." (G. Glass, 1976)
– By comparing results from many different studies, we can look for
general conclusions in domains where conclusions of individual
studies may be uncertain and/or disputed, because they are subject
to many variables, or because the literature is a big mess, offering
selective support for conflicting viewpoints
Meta-analysis
What is meta-analysis?
• Each trial is treated as one estimate of an effect, assumed
to be underlain by some global population value
– This is analogous to individual questions on a
psychometric test, inasmuch as each one provides some
information on the construct to which it relates, but
may be subject to error or contamination by itself
Meta-analysis
Five steps in conducting a meta-analysis
1.) Define a question that you want to answer
2.) Select studies according some specified inclusion criteria
3.) Select your statistical model (fixed effects versus random
effects)
4.) Calculate summary effects
5.) Interpret the results
Meta-analysis
1.) Define a question that you want to answer
• The question may be posed in terms of an independent
variable, or a set of commonly researched variables, or by
causes and consequences of important variables.
– How effective is psychotherapy for depression?
Meta-analysis
2.) Select studies according some specified inclusion criteria
• The purpose is to include only comparable studies of good
quality
• In Robinson et al, 1990:
1.) Studies from 1976-1986
2.) Patients suffering only and explicitly from depression
3.) Outpatients only
4.) Adults only
5.) Included a comparison of treatment versus no
treatment or different types of therapy = no case
histories, no pre/post designs (Why not?)
6.) Verbal psychotherapy only
Meta-analysis
3.) Select your statistical model
• Fixed effects: Assumes that the data are consistent with the
treatment effect being constant (i.e. there is a single fixed
treatment effect = no interaction between study and effect)
• Random effects: Assumes that the studies included in the
meta-analysis are a random sample generalizing to the
domain of all similar studies (under the finding that there is
a study X treatment interaction: i.e. different treatment
effects in different studies)
– We can still generalize, under the assumption that our studies
constitute a random sampling of possible effects, but the
confidence interval will be wider = less certainty in conclusions
Meta-analysis
What is an effect size?
• A standard p-value tells you how certain you can be that two (or more)
groups are really different (how likely it is that any apparent difference
is really due to chance)
– A p-value depends on two things: the size of the effect and the size
of the sample.
– You can get a significant effect either if the effect is very big
(despite a small sample) or if the sample is very big (despite a
small effect size).
– You can't average p-values, because they do not reflect the same
things in different studies
• ‘Effect size’ is a way of quantifying the size of the difference in
standardized terms
– It is the standardized mean difference between two groups
Meta-analysis
How does it work?
• We won't consider the (rather complex) mathematical
details in this class
– Specialized computer programs are available
– The basic idea is to convert values of of significance (i.e. t F, c, or
p values) into some common format: Pearson's r, or Cohen's d (a
measure of effect size = the standardized mean difference between
two groups)
– These common values must be corrected for random error (within
each study) due to sample size, measurement error, and range
restrictions (i.e. selection for studies selecting for extremes in the
possible range)
• It is more difficult to control for (although one can check
for) publication bias (only significant results get published)
and publication quality
Meta-analysis
How does it work?
• When the disparate measures from each study are all
converted to a single measure, they are directly
comparable (assuming they used comparable outcome
measures!)
• The process is analogous to converting disparate measures
(number of hockey goals scored versus number of baskets
achieved) to z-scores to make them directly comparable.
• The effect size measure is standardized and is essentially
equivalent to a z-score
Meta-analysis
The problem of moderator variables
• Moderator variables: Extraneous variables influencing the
results in a particular study
– There are mathematical ways to deal with these
Meta-analysis
4.) Calculate summary effects
• In Robinson et al, the mean effect size of psychotherapy
compared to no treatment (37 studies) was 0.73
• What does this mean?
– An effect size of 0.73 means that patients who received
psychotherapy had outcomes about 3/4 of a standard
deviation better than those who had no treatment
• The mean effect size of psychotherapy compared to
waiting list was 0.84
• The mean effect size of psychotherapy compared to
placebo was 0.28 (p > 0.05)- What does this tell us?
Meta-analysis
4.) Calculate summary effects
• There was no reliable difference between types of therapy,
but in individual planned comparisons cognitive,
behavioral, and cognitive-behavioral were all better than
'general verbal'
• The effect size comparing psychotherapy to (all) drug
therapy was 0.13 (p < 0.05), but there was no difference
between a combination of the two versus psychotherapy
alone (d = 0.01; p > 0.05) or versus drug therapy alone (d =
0.17; p > 0.05)
Meta-analysis
5.) Interpret the results
• The results of this meta-analysis suggest that
psychotherapy does work as a treatment for depression
• BUT it does not work better than placebos
• It works marginally (but significantly) better than drug
therapy, but the two treatments do not have a significantly
additive effect
• Treatment cost- in human and dollar terms- must be
factored into treatment planning
– Some costs may vary between individuals: some hate
drugs and others hate paying more than they need to
Meta-analysis
Some final thoughts on meta-analysis
Items : Testing
::
Studies : Meta-analysis
Testing : Psychological constructs
::
Meta-analysis : Experimental/Clinical Effects
– Just as we can use psychometric testing to quantify the degree to
which a construct matters for any particular purpose, we can use
meta-analysis to quantify the degree to which measured effects
matter for a specified purpose
– Just as constructs exist in a quantifiable haze of certainty, so do
treatments and effects
Meta-analysis
Some final thoughts on meta-analysis
• It is important to distinguish significance testing from
measurement of effect sizes
– When we select from extremes of a normal distribution
(high/low), we can often get highly reliable effects that
are nevertheless of negligible import in explaining the
phenomenon under study
• i.e. Some variables in lexical access have highly
reliable effects on RT when selected from extremes,
but correlate with RT with r < 0.1
• How much of the variance in RT do these highly
reliable effects account for?
Meta-analysis
Some final thoughts on meta-analysis
• More is not always better: Effects that are significant
individually may be accounting for shared variance, and
therefore not sum together
– i.e. drug therapy and psychotherapy are both better than
nothing, but but adding drugs to psychotherapy is not
better than psychotherapy alone
• The question you ask matters: 'Which treatment is better?'
≠ 'Which treatment should I prescribe?'
– It is one thing to show that two treatments differ, but
quite another to make a decision about which one is
best for any particular individual
Meta-analysis