Transcript Risk Analysis in Drug Approval

Thoughts on the Use of Decision
Analysis in the Review of New Drug
Applications
October 3, 2007
Todd Durham
Outline
NDAs and the nature of the decision
 Potential benefits and challenges of
decision analysis
 An illustration
 Learning and opportunities

Mental Exercise

Imagine that tomorrow you are diagnosed with
a disease from which you will die in exactly 7
days.
 If you could take a pill:



That would definitely cure you from this disease,
how much would you pay for it?
That would give you a 25% chance of a cure, how
much would you pay for it?
That would give you a 25% chance of a cure, how
much risk (s%) of a debilitating stroke would you
accept?
New Drug Applications

Marketing applications for new drugs
 FDA reviewed between 20-30 NDAs (for
NMEs) per year between 2001-2003 (FDA,
Critical Path, 2004)
 Data submitted with a NDA




Human evidence of benefit
Human evidence of risk
Manufacturing controls
Animal data on toxicology and carcinogenicity
Objective in Reviewing a NDA
Decide if a drug’s benefits outweigh its
risks
 Evolved historically with various changes
in the law to:

Avoid misleading doctors or consumers
 Keep dangerous drugs out of the system


What does the law really say?
“Substantial Evidence” from FDC Act of 1962
Substantial evidence was defined in section 505(d) of the
Act as “evidence consisting of adequate and wellcontrolled investigations, including clinical investigations,
by experts qualified by scientific training and experience
to evaluate the effectiveness of the drug involved, on the
basis of which it could fairly and responsibly be
concluded by such experts that the drug will have the
effect it purports or is represented to have under the
conditions of use prescribed, recommended, or
suggested in the labeling or proposed labeling thereof.”
(FDA, Clinical Evidence of Effectiveness, 1998)
Sufficient Criteria for Demonstration of Efficacy

Choice of Primary Endpoint




Results for Primary Endpoint



Reliably measures a clinically relevant characteristic
Statistically sensitive to treatment
Identified a priori (with corresponding analysis methods)
Treatment effect is “statistically significant” in at least two studies
Magnitude of treatment effect (Δ) is clinically relevant
Results for Secondary Endpoints

Results from secondary endpoints further describe the relevance
of Δ (primary endpoint) if results from primary endpoint in the
same study are statistically significant
The Case of Carvedilol
“… the usual two-study FDA paradigm does not make
sense under all situations. This much is clear. But I would
also suggest, as stated above, that experience has
shown the paradigm to be a very useful guideline;
exceptions should therefore be relatively unusual, and,
when in doubt; one should err on the side of
conservatism. Nevertheless, it strikes me as absurd in
extreme cases to insist that if one does not meet the
original primary end point in a study, that conclusions can
never be definitive but only hypothesis generating.”
(Fisher, 1999)
Criteria Used in Reviewing a NDA

Benefit





Safety




Quantity of evidence
Quality of evidence
Typically restricted to one or a few “endpoints”
Leads to a labeled claim consistent with results
From any number of reported adverse events
Cardiac safety studies (e.g., QTc)
Potentially animal studies (e.g., risk to fetus)
Manufacturing
Decision to be Made
Approve the new drug
 Reject the new drug
 Ask the sponsor for more information
(“approvable”)

Influences on the Decision





Statistical robustness of the apparent benefit,
with appropriate statistical control of the false
positive rate
Clinical relevance of the benefit
Excess safety risks, with no control of the false
positive rate
Severity of the disease
Availability of other treatments
When a Drug is Approved

Can be legally marketed in the U.S.





Safety will continue to be monitored


Doctors have a prescribing option
Patients have a treatment option
Pharmaceutical companies make revenue
Need for education all around
Surveillance has less rigor than RCTs
May be studied further


Expand the label
Clarify the role of the new drug or its effects
When a Drug is “Approvable”

Can not be legally marketed in the U.S.
Doctors can not prescribe it
 Patients can not take it


May be studied further
Pharmaceutical companies spend more
money on research
 Time for further research and submission

When a Drug is Rejected
Sponsor may withdraw application
 Can not be legally marketed in the U.S.

Doctors can not prescribe it
 Patients can not take it

“Easy” Approval Decisions

A lot of evidence of clear benefit



Clinically relevant
Statistically robust (very unlikely due to chance)
At least moderately sized safety database
reflects reasonable risks
 No evidence of toxic or carcinogenic effects
 No other available treatments or just a few
treatments with some toxicities
“Easy” Rejection Decisions
Obvious hazards with little benefit
 Poor manufacturing controls

Decisions are Much Harder When

Mixed results for benefit
Drug which has been shown to have a
benefit in some populations but not others.
 A lot of studies, only a few of which were
successful.
 Statistical criteria for “success” are not met.


Some significant trade-offs must be
reckoned with.
Made Even More Difficult

Changing landscape





Regulatory standards (e.g., emerging concerns)
Medical advances
Changing standard of care
Ex-US medical care
External pressures



Congress
Patient advocates
Pharmaceutical industry
Benefits of a Decision Analysis

Transparency of the decision



Role of uncertainties


Many objectives possible (identified, weighting)
Influences for all stakeholders
Which ones make the most difference?
Model that can be applied to many products in
the same therapeutic area, but evolve over
time.
 Dissection of the problem  greater
understanding
Transparency
Patients
 To pharmaceutical company
 Within the FDA
 Congress

Role of Uncertainties

How much do the following uncertainties bear
on the consequences?





Quality or quantity of evidence of benefit
Medical need, population affected
Available therapies
How many patients will actually use the treatment?
Don’t need to be accurate but having a grasp
on the range of uncertainties can still be
instructive (through tornado diagrams)
An Evolving Model
Changes in medicine
 Changes in how medical expenses are
reimbursed
 Changes in societal priorities or norms

Dissection of the Problem

Factors which most influence the best
decision can lead to new priorities
Role of available therapy  compare the
new treatment to available therapy
 Quantity of evidence  additional
information
 The safety/benefit tradeoff  patient
involvement


Insensitivity of the model to various
uncertainties can make decisions easier
Challenges of DA for this Application

How to define the safety risks?
All of them?
 Control of false positive rate?


How to assess the consequences
By whom?
 Using what measure?

Consequences
Time
 Money
 Human lives
 Unwanted events
 Quality-adjusted life years
 Credibility / trust (how to value?)
 Quality of information (what is its value?)

Basic Decision Tree
Consequences
C1
C2
C3
Approved
NDA Decision
Approvable
Rejected
Waiting for More Information
Success
Approved
Yes
NDA Decision
Approvable
Rejected
Outcome
Failure
New Study?
No
Presumably, success would lead to a greater chance of regulatory approval,
but what are the consequences of having made this decision to wait for
more information?
Illustration: Serious Diagnosis
Advanced cancer that affects 50,000
individuals per year
 Current expected life-span (median) is
20 months from diagnosis.
 The one available treatment is not
tolerated well such that most patients
choose not to take it.

Loosely adapted from story in New York Times, 2007.
Results from Clinical Trials

New treatment compared to placebo
 Efficacy:




Treatment effect is ~4 months of survival (benefit)
in two studies.
In one study survival had a nominal p-value
<=0.050, but it was a secondary endpoint.
Primary endpoint was stopping progression of
cancer (failed in both studies).
Safety:


Most common side effect is flu-like symptoms
1-2% chance of a stroke from new treatment
Considerations

DA could address the consequences of a world with
(now or later) and without the new treatment




Was survival a false positive finding?



Lives lost in a period of time
New strokes in a period of time
Bouts of flu-like symptoms
Zero survival benefit
What to do with the conventional hypothesis testing
interpretation?
Won’t the benefit depend on how many patients might
use the new treatment?
Could this Ever Be Applied?

Modest proposals:




More difficult proposal:


FDA could conduct an exercise by writing out an influence
diagram for approval decisions in one therapeutic area.
Carry out research on how to best communicate risk to
patients (both benefit and safety).
Increased emphasis on risk communication to patients.
Steiner, 1999 has tremendous insight on the topic.
Conduct focus groups with patients to examine ability to elicit
their trade-offs. Howard has written on ways to value life
and other outcomes.
Fantasy-land proposal:

Make all drugs available for marketing and change the
regulatory paradigm such that regulators verify accuracy of
labeling and educate doctors and the public.
Learning from Experience

Unexpected clarity, almost profound new
understanding of the decision to be made.
 Ability to proceed without regret knowing the
problem had been understood as best as
humanly possible.
 Training is important. Even highly intelligent
people do a poor job of estimating uncertain
quantities.
Illustration: What if…
The benefit was only 0-4 months of
survival, with a great deal of skepticism
that 4 months from the trials was “real”?
 Some patients might trade the chance of
a stroke for a chance at an extra month
or two of life.

But they can’t make this choice unless the
drug is made available to them.
 We won’t know unless we ask.

References
US Department of Health and Human Services, Food and Drug Administration, 2004.
Challenge and opportunity on the critical path to new medical products. Available from
www.fda.gov.
US Department of Health and Human Services, Food and Drug Administration, 1998,
Providing clinical evidence of effectiveness for human drug and biological products. Available
from www.fda.gov.
Fisher L. Carvedilol and the Food and Drug Administration (FDA) Approval Process: The
FDA Paradigm and Reflections on Hypothesis Testing. Controlled Clinical Trials 1999;20:16–
39.
Steiner J. Talking About Treatment: The Language of Populations and the Language of
Individuals. Annals of Internal Medicine 1999; 130,7: 618-622.
Howard RA. On Making Life and Death Decisions. Readings on the Principles and
Applications of Decision Analysis. Howard RA and Matheson JA, editors. 1989. Strategic
Decisions Group.
Howard RA. On Fates Comparable to Death. Readings on the Principles and Applications
of Decision Analysis. Howard RA and Matheson JA, editors. 1989. Strategic Decisions
Group.
Andrew Pollack, “Panel Endorses New Anti-Tumor Treatment,” The New York Times (March
30, 2007).