Transcript Document

How to critically
appraise a study
Nikolaos P. Polyzos M.D. PhD
The material was supported by an educational grant from Ferring
Learning objectives
• At the end of this presentation you should appreciate
that you should not accept the conclusions of any paper
without question.
• This presentation will provide guidance of how you
should question the conclusions of papers.
Essay: Why Most Published Research
Findings Are False - John P.A. Ioannidis
A research finding is less likely to be true when:
• the studies conducted in a field are smaller
• effect sizes are smaller
• there is greater flexibility in designs, definitions, outcomes, and
analytical modes
• there is greater financial and other interest and prejudice
Ioannidis JP, Plos Med 2005
• Why you should question the results in all papers
• How you should question the results
Why you should question the results
in all papers
Top cited articles are not ALWAYS telling
the truth
Top cited articles are not always telling the
truth
• Initial findings…might prove wrong in the future
• Highly cited studies (>1000 citations) with efficacy
claims
– 16% were contradicted by subsequent research
– 16% were found to have initially stronger effects
– 44% were replicated also with a larger sample size in subsequent
research compared with the original highly cited study)
– 24% had remained largely unchallenged
Ioannidis JP, JAMA 2005
Why you should question the results
in all papers
Data fabrication can occur even in highly
esteemed journals
Data fabrication can occur even in
highly estimated journals
Sudbø case (Lancet)
Among 908 patients
included in the study 250
had the same birth date
Why you should question the results
in all papers
Low quality randomized trials may show inflated
treatment effect
Low quality randomized trials may show
spuriously inflated treatment effect
• Cochrane Pregnancy and Childbirth Database
– 250 controlled trials from 33 meta-analyses
• Greater apparent treatment effect in studies of poorer
quality
– Randomisation issues
• Odds Ratios exaggerated up to 41%
– Inadequate blinding
• Odds Ratios exaggerated by 17%
Schulz. JAMA 1995; 273 (5): 408–412
Low quality trials showed treatment effect that
was not confirmed in high quality studies
META-ANALYSIS PLOT FOR PRETERM BIRTH <37 WEEKS OF GESTATION
Study or subgroup
Low quality trails
Subtotal (95% CI)
Treatment
Events/toal
No treatment
Events/total
weight
(%)
Odds ratio
(M-H, fixed)
(95% CI)
114/996
147/725
35.2
0.52 (0.38 to 0.72)
250/2303
219/2290
64.8
1.15 (0.95 to 1.40)
364/3299
366/3015
100
0.93 (0.79 to 1.10)
Odds ratio
(M-H, fixed)
(95% CI)
Test for heterogeneity:
χ2 = 4.95, df = 5, P = 0.42, I2 = 0%
Test for overall effect:
z = 4.01, P<0.001
High quality trials
Subtotal (95% CI)
Test for heterogeneity:
χ2 = 4.02, df = 4, P = 0.40, I2 = 1%
Test for overall effect:
z = 1.45, P = 0.15
Total (95% CI)
Test for heterogeneity:
χ2 = 25.94, df = 10, P = 0.004, I2 = 61%
Test for overall effect:
z = 0.86, P = 0.39
M-H=Mantel-Haenszel fixed effects model
0.1
0.2
0.5
Favours treatment
1
2
5
10
Favours no treatment
Polyzos NP et al., BMJ 2010
Why you should question the results
in all papers
Trials with negative results are not published or
get published later
Trials with negative results are not published
or get published later
Outcome in favour of
the experimental arm
Proportion of abstracts published in full-text
1.0
Not-positive
Positive
Not-positive-censored
Positive-censored
0.8
0.6
0.4
0.2
0.0
0
20
40
60
80
100
Time to publication (months)
Among studies presented in major scientific meetings, trials with positive results are more likely to
get published compared to negative studies
Polyzos NP et al., Hum Reprod 2011
Why you should question the results
in all papers
Meta-analyses often rely on few or inconsistent
evidence
Meta-analyses and systematic reviews may
often rely on few or inconsistent evidence
• 61 systematic reviews published during July 2012 in
Cochrane
– 15% of the reviews included 1or 0 trials
– Half included fewer than 1,000 patient randomized patients
– 31 were updated reviews
– 11 of these 31 updated reviews included the same number of trials
and participants as the previous review they sought to bring up to
date.
Humaidan & Polyzos Nat Med 2012
Why you should question the results
in all papers
Industry supported research may demonstrate
greater treatment effects
Industry supported research may
demonstrate greater treatment effects
Research funded by drug
companies was 4x more
likely to have outcomes
that favour the sponsor's
product than research
funded by other sources
Study (first author)
Odds ratio
Azimi12
Cho14
Clifford15
Davidson16
Dieppe18
Djulbegovic19
Djulbegovic20
Friedberg23*
Friedberg23┼
Kamal-Bahl26╪
Kamal-Bahl26§
Koep30
Mandelkern32
Sacristan36¶
Sacristan36**
Thomas38
Vandenbroucke39
Yaphe41
0.1 0.2 0.5 1 2
OR 4.05; 95% CI 2.98-5.51
5 10
100
1000
Lexchin J et al., BMJ. 2003.
10000
Industry supported research may
demonstrate greater treatment effects
82% of industry sponsored cost-effectiveness analyses
show that drugs are cost effective
Valachis et al., J Clin Oncol 2012
Industry supported research may
demonstrate greater treatment effects
The “Straw-man” comparator
Author affiliation or funding or conflict of interest |with the manufacturer
100
90
80
Sensitivity
70
60
50
40
30
20
10
0
0
10
20
30
40
50
60
70
80
90
100
1 - Specificity
Estimates for CEA’s without authors affiliated, funding or conflict of interest with the manufacturer
Esitmates for CEA’s with at least one author affiliated with the manufacturer or with funding or with conflict of interest with the manufacturer
Cost-effectiveness analyses with industry involvement were associated
with lower baseline assumptions of the sensitivity of the Pap test.
Polyzos NP et al., CMAJ 2011
Why you should question the results
in all papers
Industry sponsored trials and even government
sponsored trials have been published late in
the past
Industry sponsored trials may get published
later than non sponsored trials
• Time to publication
– Industry (24months) vs. non-industry (20 months) after study
completion p<0.001
Ross JS et al., JAMA 2013
• Publication rates within 2 years
– industry (40%) vs. non-government/non-industry (56%)
– RR = 0.73, 95% CI 0.61–0.87; p<0.001
– industry (40%) vs. government trials (47%); p = 0.22
Ross JS et al., Plos Med 2009
Even government sponsored trials fail to get
published soon
• NIH trials NOT PUBLISHED after completion
30 months - half unpublished
51 months - 1/3 unpublished
80
Percentage of studies published
Percentage of studies published
–
–
60
40
20
0
0
20
40
60
100
80
80
Trial completed before 2007
Trial completed in 2007-8
60
40
P<.001
20
0
0
Time from study completion (months)
30
No of unpublished studies
Trial completed before 2007
No at risk
635
5
10
15
20
25
Time from study completion (months)
635
635
635
493
330
220
153
95
54
44
269
264
259
235
221
197
175
324
282
244
215
176
Trial completed in 2007-8
366
356
Ross JS et al., BMJ 2011
• Why you should question the results in all papers
• How you should question the results
How you should question the results
Do not upfront reject a study based on its status
as clinical trial governance and reporting have
improved over time
Do not upfront reject a study
based on its status
Industry connection to a publication reduces its value to physicians
Physician Willingness to Prescribe Drug
1.25
0.68 (95% CI, 0.49-0.94)
P = .02
0.52 (95% CI, 0.37-0.71)
P<.001
Odds Ratio
1.00
0.75
0.50
Industry vs. None
Industry vs. NIH
Funding
Less likely to even want to read the full article !
Industry sponsorship:
•
•
negatively influences physicians' perception of methodological quality
reduces their willingness to believe and act on trial findings, independently of the trial's quality
Kesselheim NEJM 2012
Why you should not upfront reject a study
based on its status now
• The Good Clinical Practice Directive (Directive
2005/28/EC of 8 April 2005 of the European Parliament
and of the Council)
• EudraCT vs. 9 (European trials registry)
• Results must be provided for all interventional clinical
trials which commenced in the European Union from
01 May 2004 onwards
• The reporting and public disclosure includes:
– Trials which have terminated early
– Trials with positive as well as negative trial results
– Trials of products with OR without a marketing authorisation within
the community
https://eudract.ema.europa.eu/index.html
Do not upfront reject a study
based on its status
Has the situation been improved over time?
• For Pharma to introduce a new drug the drug must pass Phase II
studies to establish dosing and safety, followed by Phase III studies to
confirm efficacy and further demonstrate safety which are externally
monitored and must follow high methodological studies
• Phase II success rates for new development projects have fallen from
28% (2006–2007) to 18% (2008–2009)
Arrowsmith. Nature Reviews Drug Discovery 2011
• The combined success rate at Phase III and submission has fallen to
~50% from 2007 to 2010
Arrowsmith. Nature Reviews Drug Discovery 2011
How you should question the results
Use a standard critical appraisal approach when
evaluating a scientific article
Use a standard critical approach when
evaluating a scientific article
Critically evaluate the purpose of the study
• What is the rationale for performing the study
• Is/are the research question(s) clearly defined and if not, should they
be?
Framework for How to Read and Critique a Research Study
American Nurses Association
http://www.nursingworld.org
Use a standard critical approach when
evaluating a scientific article
Critically evaluate the design of the study
•
•
•
•
Is the design appropriate for the study?
Does the sample fit with the research design and is the size sufficient?
How were data collected?
Is the analytical approach consistent with the study questions and
research design?
Framework for How to Read and Critique a Research Study
American Nurses Association
http://www.nursingworld.org
Use a standard critical approach when
evaluating a scientific article
Examine whether the study has been registered in a public
registry
• Has the trial been registered prior starting the trial?
• Are the primary endpoints the same with the registered version of the
trial?
Use a standard critical approach when
evaluating a scientific article
Always review the existing literature
• Is the literature review relevant to the study, comprehensive, and
include recent research?
• Does the literature review support the need for the study?
• Are the findings consistent with existing literature?
Framework for How to Read and Critique a Research Study
American Nurses Association
http://www.nursingworld.org
Use a standard critical approach when
evaluating a scientific article
Critically evaluate the results and conclusions
• Are the results presented clearly in the text, tables and figures?
• Are the statistics clearly explained?
• Are the results explained in relationship to the theoretical framework,
research questions?
• Are the limitations presented and their implications discussed?
• Are there recommendations for clinical practice
Framework for How to Read and Critique a Research Study
American Nurses Association
http://www.nursingworld.org
Conclusions
• No trial is without limitations, results may occur by
chance
• There is even a risk results may be misleading
• The more robust the study the more likely it is to be true
• Always review all the literature and look for consistency
and inconsistency in what is reported
• Do not upfront accept or reject the results of a study just
based on the journal published, research group or
industry involvement
Thank you