Transcript External validity or applicability

AIFA EUROPEAN CONFERENCE ON CLINICAL
RESEARCH FOR DECISION MAKING
March 30, 2007
External validity of clinical
trials and transferability of
their results to medical practice
Luigi Pagliaro, Professor of Medicine,
University of Palermo, Palermo, Italy
1
External validity of clinical trials and transferability
of their results to medical practice
1.External validity or
applicability
2. Applicability: information needed
3. Problems of applicability
4. Help for evaluating applicability
Final remarks
References
2
External validity or applicability
In an RCT:
 Internal validity refers to the
comparability between the
intervention (I) and the control
group (C), and to the care for
minimizing the risk of bias
 External validity refers to the
availability of information to
guide the applicability of the
trial results to the target
population in a specific setting (1)
I. group
C. group
Trial results
External validity,
applicability
Target population
and setting
3
1. External validity or applicability
 There is consensus that RCTs with adequate internal
validity most reliably assess the efficacy of clinical
treatments (2)
 The statistical efficiency of RCTs is maximized in
the megatrials that enroll very large numbers of pts
belonging to different categories to detect
“moderate but still worthwhile treatment effects”
on major outcomes (3)
 A fundamental idea underlying the logic of the
megatrials is that the direction (although not the
size) of a treatment effect is similar throughout
the different categories of pts (4)
4
1. External validity or applicability
“The modest benefit ascribed to many treatments in
clinical trials can be misleading because modest average
effects may reflect a mixture of substantial benefit for
some, little benefit for many, and harm for a few”
From 5
5
1. External validity or applicability
Megatrials are the
best method to
answer the question:
“can this treatment
work”? (6)





RCT







Individual
pts
……But the applicability of the treatment to
the practice must take into account the
characteristics of the individual patients (7)
6
External validity of clinical trials and transferability
of their results to medical practice
1.External validity or applicability
2.Applicability: information
needed
3.Problems of applicability
4.Help for evaluating applicability
Final remarks
References
7
2 Applicability: information needed (from 1, modified)
1. Characteristics
of randomised pts
2. Setting
Eligibility; inclusion & exclusion
criteria; ratio of randomised
pts to eligible non-randomised
pts; baseline risk; stage in the
natural history and severity of
disease; comorbidity
Recruitment from primary,
secondary or tertiary care;
characteristics & expertise of
participating centres and
clinicians
8
2 Applicability: information needed (from 1, modified)
3. Differences
of trial protocol
vs practice
Doses and administration of the
trial treatment; adequacy of nontrial treatments; therapeutic or
diagnostic advances since the trial
4. Outcome
measures and
follow-up
Clinical relevance of the end points
and effect size; effect of the
intervention on the most relevant
components of a combined end point
5. Adverse
effects of
treatment
Rates of adverse effects and of
treatment discontinuations;
were pts at risk of complications
excluded from the trial?
9
2 Applicability: information needed (from 1, modified)
 Many RCTs don’t report the information needed to
evaluate the applicability of their results to the
practice, and/or they exclude specific subgroups of
pts that could benefit from the treatment (8-11)
 Lack of this information is one of the main
reasons for the underuse in practice of treatments
that were beneficial in trials (1), and/or of the use
of a treatment in the “wrong” patients,sometimes
with harmful consequences
10
External validity of clinical trials and transferability
of their results to medical practice
1. External validity or applicability
2. Applicability: information needed
3.Problems of applicability
4. Help for evaluating applicability
Final remarks
References
11
3. Problems of applicability
RCTs lacking information needed to apply the results
in groups of patients; some examples:
 Treatment for patients with hypertension and
comorbidities
 Peg-interferon in HCV-related cirrhosis without
knowledge of esophagogastric varices
 Tolvaptan for hyponatremia in cirrhosis
 Warfarin for patients requiring caution
12
3. Problems of applicability
Inappropriate use of trial results to “wrong”
patients and risk of harmful consequences;
some examples
 Endarterectomy for asymptomatic carotid
stenosis in centres and surgeons with lesser
expertise than those participating in the trial
 Spironolactone prescribed to pts with heart
failure more vulnerable to hyperkalemia than
those in the trial
13
3. Problems of applicability
Comorbidity in hypertensive pts in primary care (12)
RCT
Appel & al
Hansson & al
Wing & al
ALLHAT, pts
≥ 55-year-old
Sacks & al
Main comorbidities excluded
Renal damage &/or diabetes
None reported
Plasma creatinine >2.5 mg/dl
History of HF or EF < 35%
Heart disease, diabetes+insulin,
renal failure
Comorbidities were present in 89-100% of the
Fortin’s pts potentially eligible to each RCT
14
3. Problems of applicability
Lack of information
Heathcote EJ & al. Peginterferon alfa-2° in
patients with chronic hepatitis and cirrhosis.
N Engl J Med 2000; 343: 1673-80 (13)
 The paper does not report whether patients with
esophago-gastric varices were included in the
study
 This information would have been of help for
decision making: esophagogastric varices
(prevalence about 40% in Child A cirrhosis, target
of the study) suggest an advanced stage of disease
(14), with a lower probability of interferon
response (15)
15
3. Problems of applicability
Lack of information
Schrier RW & al (SALT trials). Tolvaptan, a selective oral
vasopressin V2-receptor antagonist, for hyponatremia.
N Engl J Med 2006; 355: 2099-112 (16); letters: 2007; 356: 961-3
 The trial excluded cirrhotics with Child-Pugh
score >10
 The information about these pts would have been of
paramount relevance: at least 50% of cirrhotics who
develop significant hyponatremia have Child-Pugh
score >10 (17, 18)
 Only 67 cirrhotics were treated with Tolvaptan
 The trial cannot provide any information on the risk
of adverse events of the drug in cirrhotics
16
3. Problems of applicability
Lack of information
Schrier RW & al (SALT trials). Tolvaptan, a selective oral
vasopressin V2-receptor antagonist, for hyponatremia.
N Engl J Med 2006; 355: 2099-112 (16); letters: 2007; 356: 961-3
 The trial excluded cirrhotics with Child-Pugh
score >10; only 67 cirrhotics received Tolvaptan
 The information about these pts would have been
of paramount relevance: at least 50% of
cirrhotics who develop significant hyponatremia
have Child-Pugh score >10 (17, 18)
 The trial cannot provide any information on the
risk of adverse events of the drug in cirrhotics
17
3. Problems of applicability
Lack of information
Hart RG & al. Antithrombotic therapy to prevent
stroke in pts with atrial fibrillation: a meta-analysis.
Ann Intern Med 1999; 131: 492-501 (19)
 Exclusion criteria of the RCTs are often ill-defined
(eg: “medical disorders” [20]; “contraindications for
Warfarin or aspirin therapy” [21]), and information
on co-treatments for pts with comorbidities is
generally omitted
 Clinicians may be uncertain about the risk/benefit
of Warfarin in a substantial number of pts: a
reason for its underuse (30-60% [22])?
18
3.1 External validity and applicability to subgroups
and individuals
Heterogeneity X2: p < 0.0001
Baseline risk
Endoscopic sclerotherapy
for prevention of first
variceal bleeding in
cirrhosis: only RCTs with
bleeding rate > 40% in
untreated controls show a
significant benefit (data
adjusted for the duration
of follow up); data
subsequently confirmed (2)
1. From: Pagliaro L & al. Ann Intern Med 1992; 117: 59-70
2. D'Amico G, Hepatology 1995; 22: 332-54
19
3. Problems of applicability
Inappropriate use of RCT results
 The ACAS (Asymptomatic Carotid Artery Surgery)
trial of carotid endarterectomy vs medical
treatment rejected 40% of applicants,subsequently
barring those with adverse operative outcomes (1)
 However: in USA, most carotid endarterectomies are
performed by surgical teams whose complication rates
would have rendered them ineligible for ACAS (1,23):
results of the ACAS trial and of 8 published series
Stroke & death, %
(CI)
ACAS
8 series
1.50
(0.6-2.4)
4.30
(3.5-5.2)
20
3. Problems of applicability
Inappropriate use of RCT results
 The RALES trial (24): Spironolactone significantly
reduced all-cause & cardiac mortality in pts with
severe congestive heart failure (NYHA class III-IV)
 Pts with serum creatinine concentration > 2.5 mg/dl
were excluded
 Mean age 65 years; females 27%; proportion of
diabetics not reported
 Mean doses of spironolactone 26 mg/day; pts cotreated with β-blockers 10%
No hyperkalemia-related deaths in the trial; however,
in the years following the publication of the RALES: 21
3. Problems of applicability
Inappropriate use
of RCT results
 After the RALES
publication, spironolactone
prescriptions greatly
increased, even for pts more
vulnerable to hyperkalemia
than those in the trial (eg for
elderly, diabetics, and for pts
co-treated with β-blockers),
or for pts excluded from the
trial (ie HF in NYHA class I
and II; mild renal failure);
higher doses were often given
(25, 26)
Prescriptions of
Spironolactone
per 1000 pts
Hyperkalemiarelated deaths per
1000 pts
RALES
RALES
22
External validity of clinical trials and transferability
of their results to medical practice
1. External validity or applicability
2. Applicability: information needed
3. Problems of applicability
4.
Help for evaluating applicability
Final remarks
References
23
4.Help for evaluating applicability
Interpretive medicine (27, 28)
The drug X is
effective in
the disease X1
Evidence from an
unbiased RCT
An approach
requiring time and
competence
The drug X can work in
my pts with the disease
X1, provided that:
They have the
characteristics a, b, c….m
However, there is no
evidence to support the
drug X if they have the
characteristics n, o, p…..z24
4.Help for evaluating applicability
Guidelines
Guidelines built on the global
evidence of many RCTs may
fill in the gaps of information
of the individual RCTs
100
72
80
84
60
40
Individual RCTs
Pre-appraised summaries
Guidelines
20
5
0
% preferences of 302 GP
(From 29)
25
4.Help for evaluating applicability
Subgroup analysis
A subgroup analysis can be worthwhile when
established a priori from pathophyisiologic principles
in trial populations probably heterogeneous (30,31),
eg:
 Potential heterogeneity in risks of treatment or in
risk without treatment, eg baseline risk
 Potential heterogeneity of treatment effect related
to pathophysiology, eg multiple pathologies underlying
a clinical syndrome; or genetic variation
 Underuse of treatment in clinical practice due to
uncertainty about benefit, eg underuse in specific
groups of pts such as in elderly
From 31 26
4.Help for evaluating applicability
Subgroup analysis
Pharmacogenomic
polymorphism
Conclusions. “VKORC1
haplotypes can be used to
stratify patients into low-,
intermediate-,and highdose warfarin groups and
may explain differences in
dose requirements among
pts of different
ancestries”(32)
{VKORC1: Vit K epoxide
reductase complex 1}
27
4.Help for evaluating applicability
Subgroup analysis
Pharmacogenomic
polymorphism
Conclusions: in pts
with IBD, very
high TPMT activity
predicts treatment
failure (33)
{TPMT: Thiopurine
methyltransferase}
28
Some final remarks
1
 External validity refers to the information
needed to promote the applicability of trial
results to clinical practice
 Main issues of external validity are the
characteristics of randomised pts, comorbidities
and co-treatments, outcome measures, and setting
 Information gaps due to inadequate external
validity may lead to clinical decisions unsupported
by adequate knowledge on the most appropriate
applicability of the treatments
29
Some final remarks
2
 Overviews and guidelines gathering the global
evidence of individual RCTs and analysis of
predefined subgroups may aid the transferability
of clinical research to practice
 A more effective remedy would be the inclusion
of the most relevant issues of external validity in
the design and reports of the RCTs submitted for
approval or publication
30
References
1. Rothwell PM. Lancet 2005; 365: 82-93
2. Guyatt GH & al for the EBM Working Group. JAMA 1993; 270: 2958-60
3. Collins R & al. In: Maynard A & Chalmers I Eds. Non-random reflections
on Health Services Research. BMJ Publ Group, 1997, p.197-230
4. Peto R. J Clin Epidemiol 1995; 48: 23-40
5. Kravitz RL & al. Milbank Quarterly 2004; 82: 661-87
6. Haynes B. BMJ 1999; 319: 652-3
7. Rothwell PM & al. Lancet 2005; 365: 256-65
8. Hejat A& al. Arch Intern Med 2002; 162: 1682-88 (HF)
9. Herland K & al. Respir Med 2005; 99: 11-19 (COPD)
10. Uijen AA & al. J Clin Epidemiol 2007; 60: 330-5 (Hypertension)
11. Van Spall HGC & al. JAMA 2007; 297: 1233-40 (Eligibility)
12. Fortin M, & al. Ann Fam Med 2006; 4: 104-108
13. Heathcote EJ & al. N Engl J Med 2000; 343: 1673-80
14. Pagliaro L & al. Portal hypertension in cirrhosis: natural history. In:
Portal hypertension. Pathophysiology and treatment, J Bosch & RJ
Groszmann Eds. Blackwell Scientific Publ, 1994: p.72-92
15. Poynard T & al Gastroenterology 2002; 122: 1303-13
31
16. Schrier RW & al. N Engl J Med 2006; 355: 2099-2112
References
17. Angeli P & al. Hepatology 2006; 44: 1535-42
18. Biggins SW & al. Hepatology 2005; 41: 32-9
19. Hart RG & al. Ann Intern Med 1999; 131: 492: 501
20. Ezekowitz MD & al. N Engl J Med 1992; 327: 1406-12
21. GullovAL & al. Arch Intern Med 1998; 158: 1913-21
22. Choudhry NK & al. BMJ 2006; 332: 141-3
23 Hartling L & al. Ann Intern Med 2005; 142: 1100-11
24. Pitt B & al (RALES). N Engl J Med 1999; 341: 709-17
25 Juurlink DN & al. (Ontario Study) N Engl J Med 2004; 351: 543-51
26. Bozkurt B & al. JACC 2003; 41: 211-4
27. Horton R. Stat Med 2000; 19: 3149-64
28. Horton R. Can Med Ass J (CMAJ) 1998; 158: 245-9
29. Guyatt GH & al. BMJ 2000; 320: 954-5
30. Feinstein AR. J Clin Epidemiol 1998; 51: 297-99
31. Rothwell PM, Lancet 2005; 365: 176-86
32. Rieder MJ & al. N Engl J Med 2005; 352: 2285-93
33. Ansari A& al. Aliment Pharmacol Ther 2002; 16: 1743-50
32
stop
33