Transcript How to Conduct and Interpret Information Skills Training

Systematic reviews and Metaanalyses
Alison Brettle,
Research Fellow (Information)
Salford Centre for Nursing, Midwifery and
Collaborative Research
University of Salford
Aims
To discuss the role and process
of systematic reviews and
meta-analyses
Systematic Review
A review of all
the literature on a
particular topic,
which has been
systematically
identified,
appraised and
summarised
giving a summary
answer.
What is a systematic review?




An overview of primary
research studies conducted
according to explicit and
reproducible methodology
A rigorous method of
summarising research
evidence
Shows what we know and
don’t know about a topic
area
Provides evidence of
effectiveness (or not) by
summarising and appraising
relevant evidence
Systematic reviews aim




To find all relevant
research studies (published
and unpublished)
To assess each study on
basis of defined criteria
Synthesise the findings in
an unbiased way
Present a balanced and
impartial summary of the
findings taking any flaws
into consideration


Advantages
Disadvantages
Advantages of systematic reviews







Summarise evidence, keep people up to date
without reading all published research literature
Allow large amounts of data to be assimilated (eg
by busy clinicians, policy makers etc)
A clearer picture by collating results of research
Break down barriers of getting research into
practice
Reduce bias – removes reviewers personal
opinions, preferences and specialist knowledge
Explicit methods - allow the reader to assess how
review has been compiled
More reliable conclusions because of methods
used
Disadvantages of systematic reviews




Inconclusive conclusions
Applicability to practice?
Time to undertake
Quality?
CRD Guidance


Systematic reviews aim to identify, evaluate and
summarise the findings of all relevant individual
studies, thereby making the available evidence more
accessible to decision-makers. When appropriate,
combining the results of several studies gives a more
reliable and precise estimate of an intervention’s
effectiveness than one study alone.5, 6, 7, 8
Systematic reviews adhere to a strict scientific design
based on explicit, pre-specified and reproducible
methods. Because of this, when carried out well, they
provide reliable estimates about the effects of
interventions so that conclusions are defensible. As
well as setting out what we know about a particular
intervention, systematic reviews can also demonstrate
where knowledge is lacking.4, 9 This can then be used
to guide future research.10
http://www.york.ac.uk/inst/crd/
Systematic review models

Medical/Health care




Cochrane Collaboration, NHS Centre for
Reviews and Dissemination
Usually includes “high quality” research
evidence – RCTs
Often includes meta-analysis (mathematical
synthesis of results of 2+ studies that
addressed same hypothesis in same way)
Social care/Social Sciences



SCIE, EPPI Centre, Campbell Collaboration
Often include wider range of studies including
qualitative
Often narrative synthesis of evidence
Systematic review process







Define/focus the question
Develop a protocol
Search the literature (possibly 2 stages
scoping and actual searches)
Refine the inclusion/exclusion criteria
Assess the studies (data extraction tools,
2 independent reviewers)
Combine the results of the studies to
produce conclusion– can be a qualitative
or quantitative (meta-analysis)
Place findings in context – quality and
heterogeniety of studies, applicability of
findings
What type of study design?
How effective is paracetamol at reducing
pain?
Does smoking increase the risk of oral
cancer?
STRONG
Experimental studies/ clinical trials
Randomised controlled trials
Non-randomised controlled trials
Observational studies
Cohorts
Case-controls
Cross-sectional surveys
Case series
Case reports
WEAK
Expert opinion, consensus
Experimental studies


Randomised
controlled trial
Non-randomised
controlled clinical
trial

Evaluating the
effectiveness of an
intervention
Observational studies


Cohort

Case-control

Cross-sectional
survey


Measuring the incidence of a
disease; looking at the causes
of disease; determining
prognosis
Looking at the causes of
disease; identification of risk
factors; suitable for examining
rare diseases
Measuring the prevalence of a
disease; examining the
association
What is a meta-analysis?
Optional part of a systematic review
Systematic reviews
Meta-analyses
Meta-analysis




The process of using statistical methods to
combine the results of different studies.
The aim is to integrate the findings, pool
the data, and identify the overall trend of
results (Dictionary of Epidemiology, 1995)
Focus is the direction and magnitude of
effects across studies
A method of aggregating research results –
statistical technique for amalgamating,
summarising and reviewing previous
research
What is meta-analysis?



Using meta-analysis, a wide variety of questions
can be investigated, as long as a reasonable body
of primary research studies exist.
Selected parts of the reported results of primary
studies are entered into a database, and this
"meta-data" is "meta-analysed", in similar ways
to working with other data - descriptively and
then inferentially to test certain hypotheses.
Meta analysis can be used as a guide to answer
the question 'does what we are doing make a
difference to X?', even if 'X' has been measured
using different instruments across a range of
different people. Meta-analysis provides a
systematic overview of quantitative research
which has examined a particular question.
When can you do meta-analysis?







Research must be
Empirical rather than theoretical
Have quantitative results
Examine the same constructs and
relationships
Have findings that can be put in a
comparable statistical form (e.g. effects
sizes or odds ratios)
Are comparable given the question in
hand
(adapted from DB Wilson, 1999)
Effect size






The effect size makes meta-analysis possible – it
standardises findings across studies so they can
be directly compared
Any standardised index can be an effect size (for
example, odds ratio, relative risk) as long as it is
comparable across studies, represents magnitude
and direction of relationship of interest, is
independent of sample size
What to include?
Must have clear inclusion and exclusion criteria
Published studies – publication bias?
Important to identify all studies that meet
eligibility criteria
Strengths and weaknesses









Represents findings in a systematic way
Can find relationships across studies
Clarifies interpretation of studies
Can handle large numbers of studies
But
Doesn’t always capture more qualitative
distinctions between studies
Comparability can require judgement
Inclusion of less robust studies
Selection bias (reporting of negative findings)
Meta-analysis
Understanding
the jargon and
the blobs!
Odds Ratio, Relative Risk
Measures of risk
The likelihood of
something happening
V
The likelihood of
something not happening
Relative risk


A ratio of the probability of the event occuring in the
treatment (exposed) group versus a control group (non
exposed)
RR = Probability of event in treatment group




Probability of event in control group
For example, if the probability of developing an infection in
treatment group was 20% and among control 1%, then the
relative risk would be 20 and would favour the control group.
Similarly if the probability of developing an infection in
treatment group was 4% and among control 10%, then the
relative risk would be 0.4 and would favour the treatment
group.
A RR of 1.0 = no difference between groups
Odds ratio



The odds of the event in the
intervention group divided by the
odds of the event in the control
group
OR of 1.0 = No difference between
groups
OR<1.0 means event is less likely in
the intervention group
Odds Ratio Graph (Blobbogram)
LEFT
E
S
S
M
O
RIGHT
E
Line of no significance
0.5
less than 1
1
2
more than 1
Odds Ratio
Best estimate
Confidence Interval
(wobble factor)
0.5
less than 1
1
2
more than 1
Odds Ratio (Blobbogram)
0.5
less than 1
1
2
more than 1
Confidence Interval
Is the range within which the true size
of effect (never exactly known) lies,
with a given degree of assurance (95%
or 99%).
Confidence Intervals
(Wobble factor)
Confidence Interval (CI)
= the wobble factor, how sure are we about
the results?
- the shorter the CI the more certain we are
about the results
Number needed to treat (NNT)
the number of people you
would need to treat with a
specific intervention to see
one additional occurrence
of a specific outcome
The p-value in a nutshell
How often you would see a similar result by chance, when
actually there was no effect by the drug or treatment.
0
Impossible
1
Certain Absolutely
p=0.001
Very unlikely
1 in 1000
p=0.05
Fairly unlikely
1 in 20
p=0.5
Fairly likely
1 in 2
p=0.75
Very likely
3 in 4
Practical Example
Marik PE & Zaloga GP. 2004. Meta-analysis of
parenteral nutrition versus enteral nutrition in
patients with acute pancreatitis. BMJ; 328:1407
Objective To compare the safety and clinical
outcomes of enteral and parenteral nutrition
in patients with acute pancreatitis.
Problem in a ‘nutshell’







Parenteral nutrition: intravenous feeding bypassing ‘eating
and digestion’; typically through an infusion pump;
complications: bacterial infection
Enteral nutrition: feeding through a feeding tube to the gut
Evidence gut is optimal route yet parenteral nutrition remains
widespread
In acute pancreatitis parenteral nutrition standard care but
evidence suggest enteral is feasible
In acute pancreatitis most sever complication is pancreatic
infection with mortality of up to 80%
Studies report parenteral nutrition increases infection rates
in critically ill patients and when compared enteral nutrition is
associated with improved immune function and decreased
infections
Studies under-powered; differences not always statistically
significant; magnitude a treatment effect unknown
From the abstract




Data sources Medline, Embase, Cochrane controlled trials register,
and citation review of relevant primary and review articles.
Study selection Randomised controlled studies that compared
enteral nutrition with parenteral nutrition in patients with acute
pancreatitis. From 117 articles screened, six were identified as
randomised controlled trials and were included for data extraction.
Data extraction Six studies with 263 participants were analysed.
Descriptive and outcome data were extracted. Main outcome
measures were infections, complications other than infections,
operative interventions, length of hospital stay, and mortality. The
meta-analysis was performed with the random effects model*.
*Random effects models are used when observations are not taken
from a simple random sampling to take account of a clustering or
multilevel sampling


Relative risks and continuous data outcomes are
presented with 95% confidence intervals and chi
square tests for heterogeneity
Relative risk is a ratio of the probability of the event
occurring in the exposed group versus a nonexposed group.
RR = Probability of event in enteral group

Probability of event in parenteral group




For example, if the probability of developing an
infection in enteral group was 20% and among
parenteral 1%, then the relative risk would be 20
and would favour the parenteral group.
Similarly if the probability of developing an infection
in enteral group was 4% and among parenteral
10%, then the relative risk would be 0.4 and would
favour the enteral group.
Testing heterogeneity between studies: χ2 test with
p≤ 0.05 indicating significant heterogeneity
Fig 1 Process of study selection of randomised controlled
trials (TPN=total parenteral nutrition; ENT=enteral
nutrition; PN=parenteral nutrition)
Marik, P. E et al. BMJ 2004;328:1407
Copyright ©2004 BMJ Publishing Group Ltd.
Table 1 Demographic data of studies included in metaanalysis. Figures are for enteral nutrition/total parenteral
nutrition, and scores are given as means (SDs)
No of patients
Ranson
criteria
Glasgow
Score
APACHE II
Siting of
nasojejunal
tube
McClave,
1997
16/16
1.3 (0.35) /
1.3 (0.35)
Windsor,
1998
16/18
2/2
8 / 9.5
Fluoroscopic
Kalfarentzos
, 1997
18/20
4.2 (0.9) /
4.6 (1.1)
12.7 (2.6)/
11.8 (1.9)
Fluoroscopic
Abou-Assi,
2002
26/27
Olah, 2002
41/48
Gupta,
2003
8/9
Enoscopic
3.1 (0.5) /
2.5 (0.4)
Fluoroscopic
/
endoscopic
2.6 (1.2) /
2.4 (1.6)
Fluoroscopic
8 / 10
Blind
Table 2 Outcome data of studies included in meta-analysis
(figures are for enteral nutrition/total parental nutrition)
No of
patient
s
Septic
complications
Other
complications
McClave,
1997
16/16
2/2
Windsor,
1998
16/18
0/3
0/5
Kalfarentz
os, 1997
18/20
5/10
AbouAssi,
2002
26/27
Olah,
2002
Gupta,
2003
Surgical
complications
LoS
Mortality
9.7/11.
9
0/0
1/5
12.5/1
5
0/2
3/5
2/4
40/39
1/2
1/9
13/17
1/2
14.2
/18.4
6/8
41/48
5/13
3/4
5/11
16.8
/23.6
2/4
8/9
0/2
0/6
7/10
0/0
Table 3 Jadad quality score of trials included in meta-analysis
Year
Randomisation
method
Blinding
Withdrawals/dr
op outs
accounted for
Jadad
score
McClure,
1997
1997
Not stated
None
Yes
2
Windsor,
1998
1998
Odd/even
hospital
number
None
Yes
1
Kalfarentzos
, 1997
1997
Sealed
number
envelopes
None
Yes
3
Abou-Assi,
2002
2002
Not stated
None
Yes
2
Olah, 2002
2002
Birth date
None
Yes
1
Gupta, 2003
2003
Sealed
number
envelopes
None
Yes
3
Results













Infections:
Relative risk RR = 0.45, (CI 0.26-0.78), p=0.004
Test for heterogeneity between studies p=0.59
Complications:
RR = 0.61 (0.31 – 1.22), p=0.16
Surgical interventions:
RR = 0.48 (0.23 - 0.99), p=0.05
χ2 = 0.62, p=0.89
Length of hospital stay:
Mean reduction 2.9 days (CI 1.6 – 4.3)
χ2 = 16.5, p=0.0056
Mortality
RR = 0.66 (0.32 – 1.37), p=0.3
Risk of infection, complications other than infection,
surgical intervention, and mortality; results from metaanalyses of randomised trials comparing enteral with
parenteral nutrition in pancreatitis
Marik, P. E et al. BMJ 2004;328:1407
Copyright ©2004 BMJ Publishing Group Ltd.
Random effects model of relative risk (95% confidence
interval) of infections associated with enteral feeding
compared with parenteral nutrition
Marik, P. E et al. BMJ 2004;328:1407
Copyright ©2004 BMJ Publishing Group Ltd
.
Limitations








Poor quality studies
No blinding – may overestimate intervention
effect
Different inclusion/exclusion criteria (wide range
of disease severity)
Small sample numbers leading to wide confidence
intervals
Heterogeneity of studies
Possibility of publication bias
Conclusion
Evidence does not support use of parenteral
nutrition in patients with acute pancreatitis
Further reading



Greenhalgh T (1997) How to read a
paper: papers that summarize other
papers (systematic reviews and metaanalyses), BMJ, 315:672-675
Sheldon T (2000) Statistics for evidence
based nursing, Evidence Based Nursing,
3; 4-6
Sheldon T (2000) Estimating treatment
effects: real or the result of chance?
Evidence Based Nursing, 3; 36-39
Further reading

Systematic reviews relevant to your
area of practice



Do they exist?
Are they really applicable?
To what extent to they marry up with
your practice?
Useful resources




Cochrane Collaboration
 http://www.cochrane.org/
 http://www.cochrane.org/docs/irmg.htm
Centre for Reviews and Dissemination
http://www.york.ac.uk/inst/crd/
 Finding studies for systematic reviews
 http://www.york.ac.uk/inst/crd/revs.htm
EPPI-Centre – Stages of a review
 http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=89
SCIE - The conduct of systematic research reviews
for SCIE knowledge reviews
 http://www.scie.org.uk/publications/details.asp?
pubID=111