Transcript Can unpublished data be of value in systematic

Can unpublished data be of
value in systematic reviews of
adverse effects?
Su Golder, MRC Fellow in Health Services Research
[email protected]
•
•
•
This research is being undertaken as part of an MRC fellowship
The views expressed in this presentation are those of the author
and not necessarily those of the MRC
CRD is part of the National Institute for Health Research (NIHR)
and is a department of the University of York
Introduction
•
Why adverse effects are important?
•
Why include adverse effects in systematic reviews?
•
•
Does unpublished adverse effects data differ from published
adverse effects data?
Should we include unpublished data in systematic reviews of
adverse effects?
Why adverse effects are important
•
•
Definition
• ‘A harmful or undesirable outcome that occurs during or after
the use of a drug or intervention for which there is at least
reasonable possibility of a causal relation’
Chou et al 2010
Why adverse effects are important
• Can be serious – hospitalisation, long-term disability, death
(USA: 4th to 6th leading cause of death)
• Quality of life, Compliance
• Cost (estimates of cost to NHS of £2 billion per year)
Why include harmful effects in
systematic reviews
•
•
Considering only benefits leads to bias
• Need to assess benefit/ harm balance
Detailed evaluation of safety needed when:
• Narrow margin between benefit and harm (aspirin/
CVD)
• A number of equally effective treatments with different
safety profiles
• When adverse effects cause withdrawal from treatment
Methodological Overview of Studies on
Adverse Effects
•
Comparisons of
• Industry funded data versus non-industry funded data
• Different information sources (such as MEDLINE versus
EMBASE)
• Different search strategies (such as adverse effect related
subheadings, textwords and indexing terms)
• Author affiliation (such as clinician or researcher)
• Journal impact factor (high versus low)
• Country and date of study
• Different study designs (such as RCTs versus observational
studies)
• Unpublished versus published data
Introduction: published and unpublished
data
•
•
•
If unpublished data differs systematically from published data,
systematic review limited to published data may be biased
Significant positive outcomes more likely to be published than
nonsignificant outcomes. Hence, systematic reviews based on
published literature may overestimate efficacy of an intervention
Impact of including only published adverse effects not been
clarified
Methods
•
•
•
Search strategy: 10 databases, bibliographies, handsearching key
journals, conference proceedings, websources, contacted experts,
citation searching
Inclusion criteria: compared adverse effects of healthcare
intervention according to publication status
Analysis:
• Descriptive comparison for rates and number of cases
• Ratio of odds ratio where odds ratio or relative risk presented
“pooled risk ratio for the adverse outcome from unpublished
data,’’ divided by ‘‘pooled risk ratio for the adverse outcome
from published studies.’’
Results
Reference
Study Designs
Included
Adverse effects
Intervention
Bennett 2005
Case reports
Serious
16 drugs
Cosmi 2000
Case
reports/series
Thrombotic
thrombocytopenic purpura
Ticlopidine +
aspirin
Hemminki 1980
Trials
Any
Any
Hemminki 2000
Trials
CV and thrombotic events
HRT
Loke 2004
Case reports
Specific
Amiodarone
MacLean 2003
Trials
Dyspepsia
NSAIDs
Ross 1997
Trials
Serious
Isradipine
Tramer 1997
Case reports
Bradycardia
Propofol
Wallace 2006
Trials
Any
SSRIs
Whittington 2004
Trials
Serious
SSRIs
Availability of unpublished data
•
•
Case Reports
• Two out of three studies found more unpublished than
published case reports of adverse effects (Bennett et al 2005,
Tramer et al 1997, Cosmi et al 2000)
Trials
• A higher percentage of unpublished trials report adverse effects
than published trials (Hemminki 1980)
Differences in published versus
unpublished data
•
•
Case reports
• One study of the rank order of adverse effects from published
and unpublished case reports found the most frequent adverse
effects in published cases to be respiratory and nervous
system adverse reactions, and in unpublished cases to be
thyroid and skin disorders (Loke et al 2004)
Trials
• Five studies reported the risk ratio (RR) from published and
unpublished trials (Hemminki 2000, MacLean et al 2003, Ross
et al 1997, Wallace et al 2006, Whittington et al 2004)
Differences in published versus
unpublished data
•
•
Case reports
• Unpublished case reports will generate a different picture of
the relative frequencies of specific adverse effects
Trials
• No clear evidence that data on adverse effects from
published and unpublished trials differed
• Inclusion of unpublished data could provide more precise
estimates of adverse effects
Future
•
•
•
More Reviews
• Increasing number of reviews including adverse effects
either as secondary outcome (in addition to
effectiveness) or as primary outcome
Better Reporting
• CONSORT
Help and support
• Cochrane Adverse Effects Methods Group
http://aemg.cochrane.org/
• Discussion List
http://lists.cochrane.org/mailman/listinfo/aemg
Guidance
•
•
CRD’s Guidance
• Systematic Reviews: CRD’s guidance for undertaking
reviews in health care. 2009
http://www.york.ac.uk/inst/crd/index_guidance.htm
Cochrane Handbook
• Higgins JPT, Green S (editors). Cochrane Handbook for
Systematic Reviews of Interventions Version 5.0.2 [updated
September 2009]. The Cochrane Collaboration, 2009.
Available from www.cochrane-handbook.org
Other Guidance
•
•
Loke YK, Price D, Herxheimer A. Systematic reviews of adverse
effects: framework for a structured approach. BMC Med Res
Methodol 2007;7:32
Chou R, Aronson N, Atkins D, Ismaila AS, Santaguida P, Smith
DH, Whitlock E, Wilt TJ, Moher D. AHRQ series paper 4:
assessing harms when comparing medical interventions: AHRQ
and the effective health-care program J Clin Epidemiol 2010
May;63(5):502-12
Included Papers
•
•
•
•
•
•
•
•
•
Bennett CL, Nebeker JR, Lyons EA, Samore MH, Feldman MD, McKoy JM, et al. The Research on
Adverse Drug Events and Reports (RADAR) project. JAMA. 2005:293:2131-40.
Cosmi B, Castelvetri C, Milandri M, Rubboli A, Conforti A. The evaluation of rare adverse drug events in
Cochrane reviews: the incidence of thrombotic thrombocytopenic purpura after ticlopidine plus aspirin for
coronary stenting. 8th Annual Cochrane Colloquium Abstracts, October 2000 Cape Town, Africa.
Hemminki E, McPherson K. Value of drug-licensing documents in studying the effect of postmenopausal
hormone therapy on cardiovascular disease. Lancet 2000:355:566-9.
Loke YK, Derry S, Aronson JK. A comparison of three different sources of data in assessing the
frequencies of adverse reactions to amiodarone. Br J Clin Pharm 2004:57:616-21.
MacLean CH, Morton SC, Ofman JJ, Roth EA, Shekelle PG, Southern CE-BPC. How useful are
unpublished data from the Food and Drug Administration in meta-analysis? J Clin Epidemiol 2003:56:44-51.
Ross SD, Kupelnick B, Kumashiro M, Arellano FM, Mohanty N, Allen IE. Risk of serious adverse events in
hypertensive patients receiving isradipine: a meta-analysis. J Hum Hypertens 1997:11:743-51.
Tramer MR, Moore RA, McQuay HJ. Propofol and bradycardia: causation, frequency and severity. Br J
Anaesth 1997:78:642-51.
Wallace AE, Neily J, Weeks WB, Friedman MJ. A cumulative meta-analysis of selective serotonin reuptake
inhibitors in pediatric depression: did unpublished studies influence the efficacy/safety debate? J Child
Adolesc Psychopharmacol 2006:16:37-58.
Whittington CJ, Kendall T, Fonagy P, Cottrell D, Cotgrove A, Boddington E. Selective serotonin reuptake
inhibitors in childhood depression: systematic review of published versus unpublished data. Lancet
2004:363:1341.
Publications
•
•
•
•
•
•
Golder S, Loke YK, McIntosh HM. Room for improvement? A survey of the methods
used in systematic reviews of adverse effects. BMC Medical Research Methodology
2006;27;6:3.
Golder S, Loke YK. Is there evidence for biased reporting of published adverse
effects data in pharmaceutical industry-funded studies? British Journal of Clinical
Pharmacology 2008:66(6)767-773.
Golder S, Loke YK. Search strategies to identify information on adverse effects: a
systematic review. Journal of the Medical Library Association 2009:97(2):84–92.
Golder S, Loke YK, Bland M. Unpublished data can be of value in systematic
reviews of adverse effects: methodological overview. Journal of Clinical
Epidemiology 2010;63(10):1071-1081.
Golder S, Loke YK. Sources of information on adverse effects: a systematic review.
Health Information and Libraries Journal 2010;27(3):176-90.
Loke YK, Golder SP, Vandenbrouche JP. Comprehensive evaluations of the
adverse effects of drugs: importance of appropriate study selection and data
sources. Therapeutic Advances in Drug Safety. 2011 [in press]
Industry Data
•
•
Raw data from industry
• No evidence that the numbers of adverse effects from
industry and non-industry funded studies differed.
Interpretation and conclusion of industry studies
• Bias may be introduced in the interpretation and conclusions
of industry funded studies
Sources
•
•
MEDLINE
• MEDLINE retrieved the highest number of relevant
references in only four out of thirteen cases:
• One compared MEDLINE with IPA
• One compared MEDLINE with reference books
• Two searched for non-drug interventions
EMBASE vs MEDLINE
• In eight out of ten cases searching EMBASE retrieved more
relevant references than MEDLINE.
Sources
•
•
•
Derwent Drug File vs EMBASE vs MEDLINE
• In four out of five cases searching Derwent Drug File
retrieved more relevant references than EMBASE or
MEDLINE
Industry Submissions
• In two out of four cases data retrieved from industry
submissions retrieved the highest number of relevant
records and in each case many records were unique
International Pharmaceutical Abstracts (IPA)
• Searching IPA retrieved either the lowest or joint lowest
number of relevant records in five out of seven cases
Search strategies
•
Evaluated generic adverse effects search strategies
• Badgett et al 1999
• (ae or co or po or de).fs or CASE REPORT/ and
HUMAN/
•
Golder et al 2006
• (ae or co or de).fs or (safe or safety or side effect* or
undesirable effect* or treatment emergent or tolerability
or toxicity or adrs or (adverse adj2 (effect or effects or
reaction or reactions or event or events or outcome or
outcomes)).ti,ab
Search strategies
•
Evaluated searches for named adverse effects
• Wieland et al 2005
• “human” [MESH] AND journal article [pt] AND breast
neoplasms [majr:noexp] AND (risk [mh:noexp] OR risk
factors [mh:noexp] OR follow-up studies [mh:noexp] OR
odds ratio [mh:noexp])
Search strategies
•
•
Be careful of large numbers of results
• High sensitivity can be achieved but with low precision
• Search strategies which have been evaluated, achieved
sensitivity 97% to 100%, with precision 0.9% to 2.8% (NNR:
111 to 36)
Don’t rely on electronic searches alone
• Even using generic adverse effects free text and indexing
terms and synonyms for named adverse effect likely to miss
relevant articles
Other factors
•
•
•
•
Author affiliation
• Surgeons and those working in screening reported less
complications
Impact factor
• High impact factor journals reported higher rates of adverse
effects
Year of publication
• No significant difference
Country Setting
• No significant difference
Study designs
•
•
•
•
RCTs less heterogeneous but smaller number of participants
Most confidence intervals of the incidence or risk ratio of adverse
effects overlap between study designs
Most study designs agree, in terms of finding an increase, decrease or
no difference, in adverse effects
No consistent pattern of reporting between study designs
• Observational studies may not pose a major threat of bias in the
reporting of adverse effects data but caution is still needed