Transcript Definition of treatment allocation

Uses and Abuses of (Adaptive)
Randomization:
An Industry Perspective
Benjamin Lyons, Ph. D. and
Akiko Okamoto, Sc.D.
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
1
Outline
•
•
Adaptive vs. Static Randomization
Implementation Challenges
–
–
–
•
Errors by Investigators
Errors in Algorithm
Errors related to Drug Supply
Conclusion
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
2
Adaptive vs. Static
Randomization
• Static randomization requires that one
randomization list is generated at the start of the
trial.
• Adaptive (Dynamic) randomization algorithms
(e.g., Urn model) assign treatments based on
patient characteristics and previous treatment
assignments.
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
3
Covariate Adaptive
Randomization
• Treatment assignment of the (n+1)st patient may
depend upon the previous first n patients.
• Usual mechanism is a balance function that is
minimized by assigning the new patient to a
certain treatment.
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
4
Why use adaptive
randomization?
• Treatment balance “required” within each level of
stratification factors.
• For small trials with many stratification factors
static-stratified randomization will not insure
balance within each strata or overall.
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
5
Why avoid adaptive
randomization
• May be hard to interpret using standard theory
(see recent CPMP guidelines on adjustments for
baseline covariates).
• Many chances to make errors.
• Implications of some errors on inference are not
easy to understand in the context of standard
theory.
• Some errors may put trial validity at risk.
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
6
Implementation Challenges
Three types of errors:
– Errors by investigators;
– Errors in the algorithm;
– Errors caused by a faulty drug supply method.
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
7
Example 1: Site Error
• Site enters the wrong strata level for a patient.
• Site assigns the wrong medication kit and perhaps
treatment to patient.
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
8
Response
• Do we update the balance function by altering the
assignment weights to reflect error?
• If corrected there are three categories of balance
functions:
– randomized before the error;
– randomized after the error but before the correction;
– randomized after the correction.
• If not corrected there are only two.
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
9
Analysis
• How do you report this?
• Are the pre-specified test statistics asymptotically
valid?
• For stratification error is there a sensitivity
analysis?
• How should you incorporate into a permutation or
or resampling procedure?
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
10
Prevention
• Site training.
• Train sponsor staff on how to react to the error.
• Giving IVRS vendor staff explicit instructions on
who decides to update the algorithm.
• Is it sound to alter the algorithm for a few minor
errors?
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
11
Example 2: Algorithm Error
• Specification is correct for 1:1 assignment as
indicated by simulation in SAS.
• Actual code to calculate assignment written in an
SQL program.
• Validation of SQL program did not include any
simulation.
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
12
Result
• Error in SQL program detected after 50%
enrollment. Balance is 2:1.
• Program fixed so that the balance at the end of the
trial is 1:1.
• Probability of treatment assignment correlated
with date of trial entry.
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
13
Analysis
• Is this trial randomized?
• Are the standard test statistics asymptotically
valid.
• How should we account for the error in any
permutation test?
• Should the trial results be reported at all?
• Could entry time be correlated with patient
characteristics and hence outcome?
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
14
Prevention
• Validate the actual software that produces the
assignment through simulation prior to roll out.
• Check balance results frequently during the trial.
• Vendor must have a responsible/trained statistician
who understands the issues.
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
15
Example 3: Drug Supply
• Supply at sites is not adequate.
–
–
–
–
Lack of study drug.
Drug not re-supplied often enough.
High enrollment in short periods.
Uneven enrollment by site.
• In some cases all treatment arms are not available
when a subject is randomized.
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
16
Response
• System provides “over rides” or “forced
randomizations”:
– the patient is assigned to available treatment
regardless of what the algorithm says.
• Adaptive algorithm is ignored for this patient.
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
17
Result
• Trial should be balanced if only a few
occurrences.
• “Forced” assignment included in the balance
function.
• The algorithm has not been implemented as stated
in the protocol and the report.
• Are subsequent randomizations that used the
faulty balance function valid?
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
18
Analysis
• Are the standard test statistics asymptotically
valid?
• How does a permutation test account for the ‘over
rides’?
• How many “forced” assignments before the entire
randomization is suspect?
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
19
Prevention
• Supply trials with dynamic randomization
centrally with one kit going to each site after each
randomization.
OR
• Have abundant supply at all sites.
OR
• Do not allow forced randomization, turn patients
away if all arms not available.
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
20
Simulation
• 171 Subjects.
• Two treatment Arms: A and B.
• 4 Strata: Site (16) and three prognostic factors
(2,2, and 4 levels).
• Randomization by Biased Coin.
• Entry time , stratification and response based on
CNS trial.
• Assignment is altered in 10,000 replications.
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
21
Supply Algorithm
• Each site began with 4 kits: 2 A and 2 B.
• Re-supplied in 1 “day” with two kits when one
arm is empty.
• Patients may enter with only 1 arm available.
• If arm assigned by IVRS was missing then the
remaining treatment was given.
• Drug supply is part of the simulation.
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
22
Results
• For 10000 “trial” simulations
–
–
–
–
Average of 5 “forced randomization: per trial;
T-statistic calculated for each “trial”;
Distribution similar to the theoretical.
Supply error has no effect.
Nominal Level
2.50%
Critical value
-1.96
Resampling %
1.80%
FDA/Industry Workshop
September, 19, 2003
20% 50% 80% 97.50%
-0.84
0 0.84
1.96
18% 50% 81% 98.40%
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
23
Conclusion
•
•
•
•
•
Adaptive Randomization is more difficult to
execute then static randomization.
There are several sources of error.
Result of errors are poorly understood.
Some errors may be “minor” errors.
Using Adaptive randomization adds costs and
risk to running a trial.
FDA/Industry Workshop
September, 19, 2003
Johnson & Johnson Pharmaceutical
Research and Development L.L.C.
24