Transcript A Little History - faculty.ucmerced.edu

A Little History
Cotes
• In the early 18th century, the English
mathematician Roger Cotes computed
weighted averages of measurements made by
different astronomers
Pearson (1904)
• The Effects of a Typhoid Vaccine on
– Incidence of Typhoid
– Fatalities from Typhoid
• Pearson averaged the correlations from
six studies, and concluded they were
positive but not high enough to adopt the
vaccine.
R.A. Fisher 1944
“When a number of quite independent tests of
significance have been made, it sometimes
happens that although few or none can be
claimed individually as significant, yet the
aggregate gives an impression that the
probabilities are on the whole lower than would
often have been obtained by chance” (p. 99).
• Source of the idea of cumulating probability
values
•
W.G. Cochran 1953
• Discusses a method of averaging means
across independent studies
• Laid-out much of the statistical foundation
that modern meta-analysis is built upon
(e.g., inverse variance weighting and
homogeneity testing)
Problems with Narrative
Reviews
• Coping with an increasingly large literature
– Memory biases (recall for confirming evidence)
• Vulnerability of narrative summaries to bias
– Though bias can never be eliminated from science
• One person’s bias is another person’s judgment.
• Failure to weight studies by sample size
• Tendency to rely on significance tests rather
than effect size
– “Box score” counts can be misleading
Cooper and Rosenthal (1980)
• Seven studies given to 41 people
– Who were then randomly assigned to a narrative
review or a meta-analysis condition (technically,
combined p-values).
– Studies had ambiguous results—two clearly rejected
the null, the other five reported no effect.
• Meta-analytic group more likely to perceive
support for the hypothesis and a larger effect
based on the combined p-value, e.g.,
– 68% of the meta-analytic group considered rejecting
the null, compared to only 27% of the narrative
reviewers.
Why not the p-value approach?
• There is a long history of methods for combining
probability levels.
– However, this approach does not given a measure of
effect size because of its dependency on sample size.
– And it ignores the direction of the effect.
• Since a form of a combined probability is yielded
by effect-size meta-analysis (the p value
associated with the average effect), the
combined probability method has fallen out of
favor.
Gene Glass (1976): Primary,
Secondary, and Meta-Analysis
• Primary Analysis: The analysis of primary
study data by the person who gathered it.
• Secondary Analysis: The analysis of
primary study data by someone else.
• Meta-Analysis: The analysis of summary
statistics (e.g., means) from primary
studies (typically, no access to individual
data from primary studies).
The Key Problem That Glass
Solved
• How to summarize results from many
studies when each study used different
measures in different metrics (with
different means and standard deviations)
to measure outcome. For depression, for
example:
– Beck Depression Inventory
– MMPI Depression Scale
– Investigator-generated scales
The Solution: Standardized
Effect Sizes
Major Approaches: Social Sciences
• Glass: The pioneer, but not widely used today
(unweighted, failure to aggregate effect sizes
to the study level, use of control group
standard deviation)
• Hunter and Schmidt: Widely used in
industrial-organizational psychology, focus on
correlations (e.g., validity generalization),
uses many adjustments (e.g., for reliability of
measures, for restriction of range), less
formally statistical.
• Hedges and Olkin: Most widely used
approach (the focus of most of this
workshop), most formally statistical, most
general.
Major Approaches: Medicine
• Even though meta-analysis began in education
and psychology, it has spread widely now to many
other fields.
• Particularly well-developed in medicine
– The Cochrane Collaboration
– Evidence-Based Practice
– Lots more money than the social sciences, so able to
fund many more people both to use and to develop
methods for meta-analsyis
• Methods generally follow Hedges and Olkin,
– Often independently developed
– Many major developments beyond Hedges and Olkin,
e.g.,
• Bayesian methods (Spiegelhalter)
• Methods for dichotomous outcome effect sizes
So You Want to Do a MetaAnalysis?
Time and Costs
Meta-analysis uses the same
basic steps as all research
•
•
•
•
Formulating problem (are there enough studies)
Gathering studies (file drawer problem)
Coding studies (creating data)
Computing effect sizes (how to do for nonstandard
data)
• Analyzing data (lots of options)
• Interpreting and presenting results (graphics)
• Some people call this full process “systematic
review” reserving “meta-analysis” for analyzing
data
Costs of Doing Meta-Analysis
• Meta-Analytic work is time-consuming and
needs a substantial budget—though not as high
as many primary studies.
• Steinberg et al. (1997) Am J Epid
– Identified tasks and costs for doing a meta-analysis
on 11 studies of effects of oral contraceptives on
ovarian cancer
– Not fully representative of meta-analysis because
studies drawn from a prior meta-analysis of individual
patient data rather than a literature search.
– Here are the cost and time estimates:
About
1000
hours
of time
over
people
About
$50K
More on Time
• Allen and Olkin (1999) JAMA: “Estimating Time
to Conduct a Meta-Analysis From Number of
Citations Retrieved”.
• Documented time on tasks on 37 meta-analyses
conducted by a private company specializing in
meta-analysis.
– Median time was 1110 hours
– Range 216-2518 hours…
• Then they graphed total number of citations
initially retrieved against total time to do the
meta-analysis:
Total Time  721  0.243x  .0000123x 2
The 721 hours
constant reflects
the fact that there
are some constant
time commitments
required at startup.
Add about 15
minutes for each
citation retrieved,
slightly less as
more and more
citations are found.
More on Time
• Component mean (SD) times were
– Pre-analysis search, retrieval and database
development: 558 (337) hours
• Includes protocol development, searches, library
retrieval, abstract management, study matrix
construction, paper screening and blinding, data
extraction and quality scoring, data entry, and data matrix
construction.
– Statistical analysis: 144 (106) hours
– Report and manuscript writing: 206 (125) hours
– Other (administrative): 201 (193) hours
• Includes proposal development, project-specific
correspondence, project meetings and administration,
project management, and training
Startup Time
• If you have never done a meta-analysis
before, the time allocated should be
increased considerably.
• Particularly underestimated tasks:
– Training and monitoring of coders
– Learning new statistical methods
• Best Advice: Make your first one small.
Computer Programs
• They abound (see Will’s web page at
http://faculty.ucmerced.edu/wshadish/software/othermeta-analysis-software-sites)
• We will focus primarily on R
• But also see
– Lipsey and Wilson’s SPSS macros
• They also have macros for SAS and Stata that work essentially
the same way.
• Go to http://mason.gmu.edu/~dwilsonb/ma.html.
– The Stata metadialog add-on package.
• Instructions for downloading are in your handout package.
• See Sterne, J.A.C. (2009). Meta-Analysis in Stata. College
Station, TX: Stata Press.
– Comprehensive Meta-Analysis
Gathering Studies
The Question
•
The first step is deciding on the question to
be asked:
–
•
E.g., Shadish and Baldwin 2005: What are the
effects of behavioral marital therapy (BMT)
compared to no treatment?
The question initially need not be too
specific—better to be inclusive at first and
then exclude later.
–
–
You may discover operationalizations of
constructs that you did not anticipate
Broader inclusion helps explore generalization
The Literature Search: What?
• Inclusion Exclusion Criteria
– Which treatments, people, outcomes? For
example, in Shadish’s meta-analysis of
BMT:
• Including Cognitive Behavioral?
• Effects on anything? Just on divorce or the
Marital Attitude Survey?
• Excluding studies of “enrichment”?
– Which designs: e.g., RCTs only?
– Language restrictions?
– Limited to certain years?
– Both published and unpublished sources?
The Literature Search: Where?
• The Bibliographies of Past Meta-Analyses and Literature
Reviews.
• The Bibliography of every pertinent study retrieved
• Standard Electronic Databases
– E.g., PubMed, Web of Science, PsycInfo, Dissertation Abstracts
International
– What keywords, how far back?
• Hand Searches of Journals (Recent Years)
• Funding agencies for final reports.
• A serious search for unpublished studies?
– Contacts with Experts
– Registries of Studies Initiated
– Archives of convention programs and papers
• Keep a record of keywords used:
• Keep a flow chart and record of inclusions-exclusions: