Transcript Sequential Testing Approaches for Clinical Trials

Gatekeeping Testing Strategies in
Clinical Trials
Alex Dmitrienko, Ph.D.
Eli Lilly and Company
FDA/Industry Statistics Workshop
September 2004
Outline
Gatekeeping strategies
•
account for the hierarchical structure of multiple analyses
(comparisons) and preserve the overall false positive rate
Clinical trial examples
•
•
registration trials with multiple primary and secondary
endpoints (trials in patients with depression and acute
lung injury)
dose-finding trials (trial in patients with hypertension)
Slide 2
Primary versus secondary findings
Findings with respect to secondary endpoints
provide much useful information
•
useful to prescribing physicians and patients
FDA guidance “Clinical studies section of labeling
for prescription drugs and biologics”
•
“The CLINICAL STUDIES section should present those
endpoints that are essential to establishing the
effectiveness of the drug (or that show the limitations of
effectiveness) and those that provide additional useful
and valid information about the activities of the drug”
Slide 3
Primary versus secondary findings
Dilemma
•
•
regulatory agencies and pharmaceutical companies have
long debated what secondary findings should be included
in the product label
regulatory agencies are concerned that pharmaceutical
companies tend to present favorable data and ignore
unfavorable data
Gatekeeper strategies offer one potential solution
to the dilemma
Slide 4
Types of gatekeeping strategies
Two types of gatekeeping strategies
•
•
sequential strategies have been used for 10 years
parallel strategies were introduced by Dmitrienko, Offen
and Westfall (2003)
Introduce general gatekeeping framework
•
•
•
based on the closed testing principle (Marcus et al, 1976)
focus on strategies derived using Bonferroni’s test
easily extended to more powerful tests that account for
the correlation among the endpoints (Dunnett’s test,
resampling tests)
Slide 5
Example 1: One primary endpoint
Depression trial
•
•
Experimental drug is compared to placebo
Single primary endpoint
– 17-item Hamilton depression rating scale (HAMD 17 score)
•
•
Trial is declared successful if the drug is superior to
placebo
Two important secondary endpoints
– response rate based on the HAMD 17 score
– remission rate based on the HAMD 17 score
•
Can the secondary findings be included in the product
label?
Slide 6
Sequential gatekeeping strategy
Family 1
(gatekeeper)
A1: HAMD17
Family 2
A2: Response rate
A3: Remission rate
Step 1: Perform the primary analysis
Step 2: Perform the secondary analyses with an adjustment
for multiplicity if the primary analysis yielded a significant
result
Slide 7
Sequential gatekeeping strategy
Endpoint
Primary: HAMD 17
Secondary: Response rate
Secondary: Remission rate
Raw p Adjusted p
0.046
0.046
0.048
0.048
0.021
0.042
Primary analysis: No adjustment for multiplicity
Secondary analyses: Stepwise Holm’s test
All primary and secondary findings are significant at 5%
level
Slide 8
Example 2: Multiple primary endpoints
Clinical trial in patients with acute lung injury
•
•
Experimental drug is compared to placebo
Two primary endpoints
– number of days patients are off mechanical ventilation (vent-free days)
– 28-day all-cause mortality rate
•
•
Trial is declared successful if the drug is superior to
placebo with respect to either endpoint
Two important secondary endpoints
– number of days patients are out of ICU (ICU-free days)
– overall quality of life at the end of the study
•
Can the secondary findings be included in the product
label?
Slide 9
Parallel gatekeeping strategy
Family 1
(gatekeeper)
Family 2
A1: Vent-free days
A3: ICU-free days
A4: Quality of life
A2: Mortality
Step 1: Perform the primary analyses with an adjustment for multiplicity
Step 2: Perform the secondary analyses with an adjustment for
multiplicity if at least one primary analysis yielded a significant result
Slide 10
Parallel gatekeeping strategy
Family 1
(gatekeeper)
Family 2
A1: Vent-free days
A3: ICU-free days
A4: Quality of life
A2: Mortality
Step 1: Perform the primary analyses with an adjustment for multiplicity
Step 2: Perform the secondary analyses with an adjustment for
multiplicity if at least one primary analysis yielded a significant result
Slide 11
Parallel gatekeeping strategy
Family 1
(gatekeeper)
Family 2
A1: Vent-free days
A3: ICU-free days
A4: Quality of life
A2: Mortality
Step 1: Perform the primary analyses with an adjustment for multiplicity
Step 2: Perform the secondary analyses with an adjustment for
multiplicity if at least one primary analysis yielded a significant result
Slide 12
Statistical details
Sequential families of null hypotheses
•
Null hypotheses H11, H12, H21, H22 grouped into two
families F1={H11, H12} and F2 ={H21, H22}
Parallel gatekeeping testing procedure
Overall Type I error rate is no greater than  (e.g., 0.05)
2. The null hypotheses in F2 can be tested if at least one
hypothesis in F1 has been rejected [parallel testing]
3. Adjusted p-values for the hypotheses in F1 do not
depend on the p-values associated with the hypotheses
in F2 [primary analyses cannot depend on secondary
analyses]
1.
Slide 13
Closed testing
Closed family of hypotheses
•
Consider all 15 intersections of the null hypotheses
– {H11 or H12 or H21 or H22}, {H11 or H12 or H21}, etc
•
•
Specify a test for each intersection hypothesis that
controls the Type I error rate, e.g., Bonferroni test
Establish implication relationships
– Intersection hypothesis {H11 or H12} implies H11 and H12, etc
•
Tests for original hypotheses
– Reject a null hypothesis if all intersection hypotheses implying it have
been rejected
Slide 14
Decision matrix
Intersection hypothesis
Rejection rule
H11 or H12 or H21 or H22
H11 or H12 or H21
H11 or H12 or H22
H11 or H12
H11 or H21 or H22
H11 or H21
H11 or H22
H11
H12 or H21 or H22
H12 or H21
H12 or H22
H12
H21 or H22
H21
H22
Z11>z/2 or Z12>z/2
Z11>z/2 or Z12>z/2
Z11>z/2 or Z12>z/2
Z11>z/2 or Z12>z/2
Z11>z/2 or Z21>z/4 or Z22>z/4
Z11>z/2 or Z21>z/2
Z11>z/2 or Z22>z/2
Z11>z/2
Z12>z/2 or Z21>z/4 or Z22>z/4
Z12>z/2 or Z21>z/2
Z12>z/2 or Z22>z/2
Z12>z/2
Z21>z/2 or Z22>z/2
Z21>z
Z22>z
Slide 15
Parallel gatekeeping strategy
Two scenarios
•
•
significant improvement in the mean number of ventilatorfree days and 28-day all-cause mortality
significant improvement in 28-day all-cause mortality but
not in mean number of ventilator-free day
Weighted analyses
•
primary endpoints are unequally weighted to reflect their
relative importance (likelihood of success)
– weight=0.9 for number of ventilator-free days
– weight=0.1 for 28-day all-cause mortality
Slide 16
Parallel gatekeeping strategy
Scenario 1
•
significant improvement in the mean number of ventilatorfree days and 28-day all-cause mortality
Endpoint
Primary: Vent-free days
Primary: Mortality
Secondary: ICU-free days
Secondary: Quality of life
Raw p Adjusted p
0.024
0.027
0.003
0.030
0.026
0.029
0.002
0.027
Primary analyses: Bonferroni’s test
Secondary analyses: Stepwise Holm’s test
All primary and secondary findings are significant at 5%
level
Slide 17
Parallel gatekeeping strategy
Scenario 2
•
significant improvement in 28-day all-cause mortality but
not in mean number of ventilator-free day
Endpoint
Primary: Vent-free days
Primary: Mortality
Secondary: ICU-free days
Secondary: Quality of life
Raw p Adjusted p
0.084
0.093
0.003
0.030
0.026
0.093
0.002
0.040
Primary analyses: Bonferroni’s test
Secondary analyses: Stepwise Holm’s test
The primary mortality analysis and secondary quality of
life analysis are significant at 5% level
Slide 18
Example 3: Dose-finding study
Clinical trial in patients with hypertension
•
Four doses of an experimental drug are compared to
placebo
– doses are labeled as D1, D2, D3 and D4
•
Primary endpoint
– reduction in diastolic blood pressure
•
Objectives of the study
– find the doses with a significant reduction in diastolic blood pressure
compared to placebo
– study the shape of the dose-response curve
Slide 19
Example 3: Dose-finding study
Family 1
(gatekeeper)
Family 2
(gatekeeper)
A1: D4 vs. P
A3: D2 vs. P
Family 3
Pairwise comparisons
A2: D3 vs. P
A4: D1 vs. P
Step 1: Compare doses D3 and D4 to placebo
Step 2: Compare doses D1 and D3 to placebo if at least one
comparison at Step 1 is significant
Step 3: Perform various pairwise dose comparisons if at least one
comparison at Step 2 is significant
Slide 20
Example 3: Dose-finding study
Family 1
(gatekeeper)
Family 2
(gatekeeper)
A1: D4 vs. P
A3: D2 vs. P
Family 3
Pairwise comparisons
A2: D3 vs. P
A4: D1 vs. P
Step 1: Compare doses D3 and D4 to placebo
Step 2: Compare doses D1 and D3 to placebo if at least one
comparison at Step 1 is significant
Step 3: Perform various pairwise dose comparisons if at least one
comparison at Step 2 is significant
Slide 21
Example 3: Dose-finding study
Family 1
(gatekeeper)
Family 2
(gatekeeper)
A1: D4 vs. P
A3: D2 vs. P
Family 3
Pairwise comparisons
A2: D3 vs. P
A4: D1 vs. P
Step 1: Compare doses D3 and D4 to placebo
Step 2: Compare doses D1 and D3 to placebo if at least one
comparison at Step 1 is significant
Step 3: Perform various pairwise dose comparisons if at least one
comparison at Step 2 is significant
Slide 22
Example 3: Dose-finding study
Mean reduction in DBP (mmHg)
10
5
0
0
1
2
Dose
3
4
Slide 23
Parallel gatekeeping strategy
Comparison
Raw p
Adjusted p
Gatekeeping
Holm
procedure procedure
0.0016
0.0055
0.0269
0.0673
0.0394
0.0787
D4 vs. P
D3 vs. P
D2 vs. P
0.0008
0.0135
0.0197
D1 vs. P
D4 vs. D1
0.7237
0.0003
1.0000
0.0394
1.0000
0.0021
D4 vs. D2
D3 vs. D1
0.2779
0.0054
1.0000
0.0394
0.8338
0.0324
D3 vs. D2
0.8473
1.0000
1.0000
Dunnett
procedure
0.0030
0.0459
0.0656
0.9899
Doses D2, D3 and D4 are significantly different from
placebo at 5% level
Slide 24
Summary
Gatekeeping strategies can be successfully used in
•
•
pivotal trials with multiple primary and secondary
endpoints
dose-finding studies
Registration trials
•
a priori designation of gatekeeping strategy allows
additional data useful to physician and patient to be
presented in the product label
Dose-finding studies
•
efficient tests of dose-response relationship
Slide 25
Extensions
More powerful gatekeeping tests
•
•
based on more powerful tests, e.g., Simes test
based on tests accounting for the correlation among the
endpoints (exact parametric tests such as Dunnett’s test
and approximate resampling-based Westfall-Young tests)
Software implementation
•
SAS programs for gatekeeping tests can be found in
Dmitrienko, Molenberghs, Chuang-Stein, Offen. (2004).
Analysis of Clinical Trials: A Practical Guide. SAS
Publishing, Cary, NC.
Slide 26
References
Papers
•
•
•
Dmitrienko, Offen, Westfall. (2003). Gatekeeping
strategies for clinical trials that do not require all primary
effects to be significant. Statistics in Medicine. 22, 23872400.
Marcus R, Peritz E, Gabriel KR. (1976). On closed testing
procedure with special reference to ordered analysis of
variance. Biometrika. 63, 655-660.
Westfall, Krishen. (2001). Optimally weighted, fixed
sequence, and gatekeeping multiple testing procedures.
Journal of Statistical Planning and Inference. 99, 25-40.
Slide 27