Transcript Slide 1

Methodologic Challenges:
Appraisal of Evidence
Ralph M. Meyer
NCIC Clinical Trials Group
and Queen’s University
Appraisal of Clinical Trials
• 101:
Some Basics
• 202:
Strategic Principles of Trial Design
• 303:
The Interim Analysis
• 404:
Biomarkers
Beyond: Some issues of drug development
101
Some Basics
Some Basics
• Were the patients really randomized?
• Were clinically relevant outcomes reported?
• Is the population recognizable?
• Was there clinical + statistical significance?
• Is the intervention feasible?
• Were all accounted for?
Sackett et al, 1985 (1st Ed)
Guyatt, JAMA 1993
Some Basics
Beware of:
• The incomplete randomization
FFTF
Diehl, NEJM 2003
Some Basics
Beware of:
• The incomplete randomization
• Surrogate outcomes
Hierarchy of Outcomes
Major
Survival
QoL
Economic
Surrogate
FFP
Hospitalization
Response
Toxicity
Meyer, Kouroukis, Evid-based Oncol: 2001
Outcome Measures for Research Trials
Basic Research
Investigators
Investigators
Phase I Trials
Case
Reports
Phase II Trials
Outcomes
Pharmacokinetics
Toxicity
Response
Optimum Dose
Practitioners
Outcomes
Response
FFPD
Toxicity
Reviews
Phase III Trials,
Systematic Reviews
Outcomes
FFPD
Survival
Quality of Life
Toxicity
Practitioners
Haynes, Ann Int Med 1990
Meyer, Kouroukis; Evid-based Oncol, 2001
Some Basics
Beware of:
• The incomplete randomization
• Surrogate outcomes
• QoL – small differences / no response
• Control arm interventions
• Over and underpowering
202
Strategic Principles of Trial Design
Strategic Designs of Phase III Trials
• Explanatory vs. Pragmatic
• Large Simple Trials
• Non inferiority Trials
Explanatory vs. Pragmatic
Explanatory vs. Pragmatic Trials
Explanatory
• Tests a biologic principle / causal effect
• Emphasize efficacy
Pragmatic
• Tests a treatment policy
• Emphasize effectiveness
Example 1
observe
RT
Control Group
30 days
Expt’l Group
RT
treat
The explanatory trial:
Does 30 days of a radiosensitizer have a biologic benefit?
Schwartz + Lellough, J Chron Dis, 1967
Example 2
Control Group
RT
30 days
Expt’l Group
RT
treat
The pragmatic trial:
Does 30 days of a radiosensitizer improve health outcomes?
Schwartz + Lellough, J Chron Dis, 1967
Large Simple Trials
Large Simple Trials
Principles:
“The real differences between two treatments
in some important outcome will probably not
be large, but even a moderate difference in
an important outcome may be worthwhile”
Peto, Collins, Gray, J Clin Epi, 1995
Large Simple Trials
Principles / Implications:
• Seeking large effect sizes is impractical
• If small / moderate effect sizes are sought,
the experimental design must get it right:
– Minimize bias
– Minimize random error
• ergo, large sample size
N = 17,187
ISIS-2, Lancet 1988
Large Simple Trials
Principles / Implications:
• May be used to compare existing standards
• May be used to confirm a meta-analysis
• May not test a paradigm change
• Are subject to biomarker qualification
Large Simple Trials
Beware of:
• Very small differences / large NNTs
• Heterogeneous populations in an era of:
- targeted therapy
- biomarkers
Superiority vs. Non-inferiority
Superiority vs. Non-inferiority
A new treatment is:
• ‘as good’ at disease control and is:
– Less toxic
– Associated with a better QoL
– More cost effective
– More convenient
Superiority vs. Non-inferiority
Key Principles:
• Include superiority for a 2o outcome
• Define the non-inferiority boundary
The benchmark will be the upper 95% CI
• Be better than ‘putative placebo’
• Include an as-treated analysis
Kaul, Ann Int Med 2006
Treatment Differences in Noninferiority Trials
Piaggio, JAMA 2006
The Putative Placebo
Kaul, Ann Int Med 2006
Superiority vs. Non-inferiority
Beware of:
• A wolf in sheep’s clothing
(superiority trial that fails to meet endpoint)
• The lack of a superior 2o outcome
303
The Interim Analysis
Interim Analysis
• Trials test hypothesis; equipoise exists
• Cumulative data address the hypothesis
• These data can confirm or reject the hypothesis
• If conclusively addressed, it would be both
unethical and an unwise use of resources to
continue to conduct the trial
• Interim analysis are therefore appropriate
• The issue is methodological soundness
Interim Analysis
Issues of methodologic soundness:
• There should be sufficient events
• There should be predefined boundaries
• The boundaries should be based on sound
application of statistical principles
e.g., O'Brien-Fleming boundaries
• Data should be independently reviewed (DSMC)
• Follow-up should continue
Jennison and Turnbull
Interim Analysis
Beware of:
• Repetitive reporting
• No statement about boundaries
• Lack of an independent DSMC
• Lack of data cleaning processes
• Results that are ‘too good’ (but be careful)
404
Biomarker Development
Some Definitions
Biomarker (Biological Marker):
• A characteristic that is objectively measured and
evaluated as an indicator of normal biological
processes, pathogenic processes or pharmacologic
responses to a therapeutic intervention
NIH Biomarkers Definition Working Group,
Clin Pharmacol Ther, 2001
Potential Role of Biomarkers
• Define causation
• Early detection / screening
• Assist in making a diagnosis
• Define a therapeutic target
• Facilitate anti-tumour response assessment
• Influence getting therapy, through prognosis
• Determine who gets which therapy, through prediction
• Define details of intervention (e.g., dose)
Some Definitions
Prognostic Marker:
• Identify patients with differing risks of specific
outcomes, such as progression or death
Predictive Marker:
• Predicts the differential efficacy of a particular
therapy based on the marker status
Sargent, J Clin Oncol, 2005
Predictive Markers
Confirmation of a predictive marker follows the same
principles as confirming best therapy:
An RCT is required
Principles are aligned with those of a subset analysis
Register
Indirect
Test
Biomarker
Biomarker
-’ve
Rx A
Biomarker
+’ve
R
R
HR
HR
Rx B
Rx A
Rx B
Sargent, J Clin Oncol, 2005
Register
Indirect
Test
Biomarker
-’ve
R
Biomarker
Role of biomarker
can be tested
through statisitical
interaction
HR
Rx A
Biomarker
+’ve
R
HR
Rx B
Rx A
Rx B
Sargent, J Clin Oncol, 2005
Register
Indirect
Test
The statistical test
for interaction is
crucial: Rx B may
just be better
therapy
Biomarker
Biomarker
-’ve
Rx A
Biomarker
+’ve
R
R
HR
HR
Rx B
Rx A
Rx B
Sargent, J Clin Oncol, 2005
NCIC CTG CO.17
HR = .77
P=.005
Med. estimates 6.1 vs.
4.6 mos
Jonker, NEJM 2007
Laurent-Puig, Clin Cancer Res 2009
Karapetis, NEJM 2008
K-ras is not a prognostic
marker
Karapetis, NEJM 2008
Overall Survival
HR = .98
HR = .55
Test for interaction P < 0.001
K-ras is a predictive marker
Karapetis, NEJM 2008
The statistical test
for interaction is
crucial: Rx B may
just be better
therapy
Register
Indirect
Test
Biomarker
(P < 0.001)
K-ras wild
type
R
Rx A
K-ras
mutant
HR = .55
Rx B
HR = .98
Rx A
R
Rx B
Sargent, J Clin Oncol, 2005
Register
Direct Test
Biomarker
Rx is
standard
Rx A
R
Biomarker
determines
Rx
Biomarker
–’ve
Biomarker
+’ve
Rx A
Rx B
Sargent, J Clin Oncol, 2005
Carde, J Clin Oncol: 1993
The biomarker
Carde, J Clin Oncol: 1993
P = 0.24
Carde, J Clin Oncol: 1993
Biomarkers and Guidelines
Beware of:
• Overuse of biologic plausibility
• Data for non-randomized trials
• Multiple comparisons
• Confusing prognosis with prediction
• Using P values to define magnitude of benefit
• Omitting tests for interaction
Biomarkers and the Meta-analysis
• A meta-analysis may detect a small
benefit in a heterogeneous population
• A biomarker may define a population for
whom the benefit is most important
Drug Development and Guidelines
Drug Development and Guidelines
Major Points:
• Complex topic
• Distinguish drug from policy development
• Rigor is not necessarily generalizable
• Main issue is the ‘phase III failure’
• Developmental designs should reduce this risk
Progression of Trials
Major Exit Points
Preclinical
Phase 1
Phase 2
Phase 3
Drug Development and Guidelines
Implications of Phase III failure
• Time
• Patient care
• Financial implications
• Phase II design must minimize phase III failure
Progression of Trials
Preclinical
Phase 1
Phase 2
Single Arm
Randomized
Phase 3
Drug Development and Guidelines
Purpose of a Phase II Trial:
• Estimate efficacy and toxicity
• Inform phase III design
• Not practice policy
• Objectives facilitated by randomization
Drug Development and Guidelines
Purpose of a Phase III Trial:
• Depends: explanatory vs pragmatic
• Causal relationship vs. policy
• Continuum, not ordinal
Progression of Trials
Preclinical
Phase 1
Phase 2
Phase 3
Explanatory
Pragmatic
Progression of Trials: Phase 3
EARLY
LATE
Patients
Metastatic
Adjuvant
Design Principles
Explanatory
Pragmatic
Regulatory Approval
Primary
Secondary
Data Collection
Detailed
Less Detailed (?)
Progression of Trials: Phase 3
EARLY
LATE
Emphasis of biologic POP
Emphasis of effectiveness
Lesser direct relevance to
health care delivery
Direct relevance to health care
delivery is essential
Correlative biology may
emphasize tumour factors
Correlative biology may
emphasize patient factors
Some NCIC CTG Trials:
Explanatory vs. Pragmatic (1)
TRIAL
SS
HR
P
Absolute Difference*
PA.3
569
.82
.038
11 days (6.24 vs. 5.9 mos)
BR.21
731
.73
.001
60 days (6.7 vs. 4.7 mos)
CO.17
572
.68 <.001
45 days (6.1 vs. 4.6 mos)
* Median overall survival
Some NCIC CTG Trials:
Explanatory vs. Pragmatic (2)
TRIAL
SS
HR
P
Absolute Difference
PA.3
569
.82
.038
11 days (6.24 vs. 5.9 mos)
BR.21
731
.73
.001
60 days (6.7 vs. 4.7 mos)
CO.17
572
.68 <.001
45 days (6.1 vs. 4.6 mos)
• Two trials were published in NEJM
• The 3rd was an ASCO plenary paper
• All 3 had important correlative studies
– Two of these were NEJM publications
Drug Development and Guidelines
Beware of:
• Misuse of randomized phase II for policy
• Misuse of explanatory trial for policy
• Over-interpretation of Hazard Ratios
Methodologic Challenges:
Appraisal of Evidence
Some conclusions:
• Trial design / conduct / analysis can be complex
• It is unlikely that there will be no blemishes
• Goal is to use best evidence …
• … and understand potential limitations