Transcript sys_review_meta_analyses

Conducting and Interpreting
Systematic Reviews and MetaAnalyses
July 12, 2007
Initial Notes about Reviews
Reviews are considered the “highest” form of
evidence in the EBM evidence pyramid
 Techniques/issues discussed here are different
from the type of review (e.g., Psychological
Bulletin) you are used to
 Reviews are “research of research” (this is the
derivation of the ‘meta’ component in ‘metaanalysis’)
 As such, reviews must follow, and must make
explicit, their methodology just like primary
studies do

Evidence Pyramid for Treatment Effectiveness Questions
Goal: Use best available evidence
Types of Reviews

Systematic Review:

Meta-analysis:

Integrative Review:

Qualitative Review:
literature review focused on a single
question which tries to identify, appraise, select and synthesize all high
quality research evidence relevant to that question
a statistical technique often used in systematic
reviews that enables the results from a number of studies to be combined
into a common metric to determine the average effect of a given technique.
Comparisons can then be made about the relative effectiveness of various
techniques
literature review that attempts to combine
experimental and nonexperimental studies, or from experimental and
theoretical work. Can be used to review theories, evidence, or to analyze
methodological issues
methods for combining qualitative research
studies. Techniques include meta-synthesis, formal grounded theory, and
meta-ethnography
Review Stages
Problem formulation
 Literature search
 Data evaluation/extraction
 Data analysis
 Presentation

Systematic Reviews



Definition: summary of the medical literature that uses explicit
methods to perform a thorough literature search and critical
appraisal of individual studies and that uses appropriate statistical
techniques to combine these valid studies
Most evaluate the effectiveness of therapy, but systematic reviews
of diagnostic test are also important
Systematic reviews are:
Systematic in identification of literature
 Explicit in statement of objectives, materials, and methods
 Reproducible in its methodology and conclusions


Best individual sources: Cochrane, PubMed Clinical Queries
Systematic Reviews: Search
Strategies

Search Strategies

Electronic Databases
(Medline, EMBASE, Cochrane; Clinical Trials Registry Database)
http://ssrc.tums.ac.ir/SystematicReview/CTRDB.asp
Hand searching for published studies not in electronic databases
 Checking reference lists
 Contacting authors




Balance between comprehensiveness and precision
Document search strategy in final review
Additional resources:
Cochrane handbook:
http://www.cochrane.org/resources/handbook/index.htm
 Searchable database of research papers (dissertations and theses)
http://dissertationsandtheses.com/
 Current Contents Connect http://scientific.thomson.com/products/ccc/

Systematic Reviews: Critical
Appraisal



Critical appraisal = quality assessment
Is the evidence valid and relevant for your clinical question or
situation?
Ways validity assessments are used
As
 As
 As
 As


Dimensions of critical appraisal




a threshold for inclusion
an explanation for differences in results between studies
a sensitivity measure
a means for weighting studies in statistical analyses
Internal validity
Applicability
Precision (size of CI)
Useful resources


http://ssrc.tums.ac.ir/SystematicReview/Appraisal-Tools.asp
http://ssrc.tums.ac.ir/SystematicReview/RCTtools.asp
Systematic Reviews: Conclusions







Methodological Issues
Number of studies and participants
Size of treatment effect
Precision of treatment effect
Consistency of outcomes and results
Apparent dose-response relationships, if any
Generalizability and applicability of findings to
other populations and settings
Meta-Analysis
Meta analysis is a set of statistical techniques for combining information
from different studies to derive an overall estimate of a treatment's effect.
 Potential issues (a “Catch 22”):




Publication bias
Varying quality of studies
Suggestions for high quality meta-analyses:






A formal protocol should be written specifying the exact question under
investigation and describing the studies that will be included in the analysis.
All research, not just published research, should be included. The issue of
“fugitive literature” is important.
Registries should be established so that studies can be tracked from their
inception and not just on publication. This idea has been given a push so that
drug companies would not be able to publish trials showing benefit from their
products while suppressing those that do not.
Measures of study quality should be included
Clear eligibility criteria should be established
Many meta analytic techniques should be used and all results should be
reported. A result would be considered reliable only if all of the techniques give
the same result.
Steps in Statistical Analysis

Select measure of effect (rate difference, odds
ratio, standardized mean differenceetc.)




RCT: rate difference (EER v. CER) or rate ratio
Case Control: odds or rate ratios
Standardized mean difference between E and C
Specify effects model:


Fixed: results generalizable to reviewed studies –
assumes that there is an underlying effect running
through all studies
Random: results generalizable to universe of such
studies (computationally more intense) – assumes
that the true effect estimate varies across studies
Meta-Analysis: Data Analytic
Techniques

Dichotomous outcomes (e.g., cured v. not)
 Mantel-Haenszel:
pooled odds ratio across a
strata of 2x2 tables
 Analysis
example:
http://www.statsdirect.com/help/chi_square_te
sts/mh.htm
Meta-Analysis: Data Analytic
Techniques (cont’d)

Continuous Data: Standardized difference
as an example (e.g., Cohen’s d)
d
= M1-M2/SD
 Pooled SD of experimental and control groups
is usually used; square root of the average of
the two squared SD’s
 Can be computed from t-test (d = 2t/df
 Other measures: Hedges’s g
Therapeutic trials of
streptokinase for MI
Van Emmerik, A.A.P., Kamphuis, J.A., Hulsbosch, A.M., & Emmelkamp,
P.M.G. (2002). Single-session debriefing after psychological trauma: A
meta-analysis. Lancet, 360, 766-771.
Integrative Reviews
Provide review of available literature
according to clear methodological
approach
 Potential sources of bias

 Incomplete
literature search stage
 Errors in data extraction
 Inappropriate data analysis, particularly in
combining experimental and nonexperimental
methods
Integrative Reviews
Defining the review question clearly
 Literature search issues





Obtaining needed literature often challenging
Publication bias
Identify maximum number of primary sources
through multiple strategies; make strategies explicit
(i.e., specify search terms) in review
Data evaluation issues


Explicit assessment of study quality – complex when
included studies have vastly different designs
How to define quality of nonempirical studies?
Integrative Reviews (cont’d)

Data analysis issues
 Developing
a classification system for
managing data from diverse methodologies
 Classifying studies based on some logical,
predetermined system (e.g., type of evidence,
demographic characteristics, predetermined
conceptual classification) and analyzing each
subgroup sequentially
Qualitative Reviews
Qualitative research is typically nonexperimental,
but it may be useful to include in systematic
reviews because it evaluates important aspects
of clinical research, often from the patient
perspective (e.g., “views” research)
 Relies on inductive rather than deductive
reasoning
 Qualitative research involves different measures
of report quality than do RCT’s
 Integrating quantitative and qualitative data
sources is a key challenge to systematic
reviewers

Types of Qualitative Data
http://hsc.uwe.ac.uk/dataanalysis/qualVisDataEx.asp
Major Types of Qualitative Research


Participant Observation: used to assess the impact of
healthcare events from the patient perspective
Interviews: used to assess range of ideas and concepts
among individuals involved in healthcare delivery and
receipt of healthcare services. Oral histories are one
example




Interview example:
http://hsc.uwe.ac.uk/dataanalysis/qualTextDataSelf.asp
Record review: used to assess phenomenology that
leads to health care decisions (e.g., what factors lead to
discharge or prescription of particular medications or
behavioral intervention strategies)
Focus groups
Case studies
Qualitative Research: Analytic Strategy
Examples

Constant Comparison Method: All data
sources are compared to all others to
search for commonalities or differences
 “Grounded
Theory” (Glaser, Strauss) –
systematic method of generating theory from
data

Ethnographic methods: How individuals
or cultural groups conceptualize or enact
health/illness or health-seeking
Integrating Qualitative and Quantitative Date in Systematic Reviews
Thomas, et al., (2004). Integrating qualitative research
with trials in systematic reviews. BMJ, 328, 1010-1012.