Transcript ACCworkshop06.09

Using Clinical Trial Data to
Construct Policies for Guiding
Clinical Decision Making
S. Murphy & J. Pineau
American Control Conference Special Session
June, 2009
Outline
Long Term Goal: Improving Clinical Decision Making
Using Data
– Sequential Clinical Decision Making
– Clinical Trials
– Challenges
• Incomplete, primitive, mechanistic models
• Measures of Confidence
– Illustration
2
3
Critical Decisions
• Which treatments should be offered first?
• How long should we wait for these
treatments to work?
• How long should we wait before offering a
transition to a maintenance stage?
• Which treatments should be offered next?
• All of these questions relate to the
formulation of a policy.
4
Examples of Clinical Trials
• Sequenced RBT: Goal is to improve
neonatal outcomes
• STAR*D: Goal is to achieve depression
remission.
5
Jones’ Study for Drug-Addicted
Pregnant Women
rRBT
2 wks Response
Random
assignment:
tRBT
Random
assignment:
tRBT
tRBT
Nonresponse
eRBT
Random
assignment:
2 wks Response
aRBT
Random
assignment:
rRBT
rRBT
Random
assignment:
Nonresponse
tRBT
rRBT
6
STAR*D
Preference
Stage 1
Treatment
Intermediate
Outcome
Preference
Stage 2
Treatment
Remission
Continue
on Present
Treatment
Bup
Switch
R
Ven
Ser
MIRT
Switch R
+ Bup
Augment R
No
Remission
NTP
+ Bus
+LI
Augment R
+THY
7
Challenges
• Incomplete Mechanistic Models
– non-causal “associations” in data occur due to
the unknown causes of the observations
• Small, Expensive, Data Sets with High
Noise to Signal Ratio
– Measures of confidence are essential
8
Conceptual Structure in the Behavioral
Sciences (clinical trial data)
Unknown
Causes
Observations
Unknown
Causes
Action
Stage 1
Observations
Action
Stage 2
Reward
Stage 2
9
Unknown, Unobserved Causes
(Incomplete Mechanistic Models)
Maturity/
Decision
to join "Adult"
Society
Unknown
Causes
+
-
Binge Drinking
Yes
Counseling on
Health
Consequences
Yes/No
-
Binge Drinking
Yes/No
Time 2
Sanctions
+ counseling
Yes/No
Functionality
Time 3
10
Unknown, Unobserved Causes
(Incomplete Mechanistic Models)
• The problem: Even when treatments are
randomized, non-causal associations occur in the
data.
• Solutions:
– Recognize that parts of the transition probabilities
(“system dynamics”) can not be informed by domain
expertise as these parts reflect non-causal associations
– Or use methods for constructing policies that “average”
over the non-causal associations between action and
11
cost or reward.
Measures of Confidence
• We would like measures of confidence for
the following:
– To assess if there is sufficient evidence that a
particular observation (e.g. output of a
biological test) should be part of the policy.
– To assess if there is sufficient evidence that a
subset of the actions lead to lower cost than the
remaining actions.
(reward=-cost) 12
Measures of Confidence
• Traditional methods for constructing
measures of confidence require
differentiability (if frequentist properties are
desired).
• Optimal policies are constructed via nondifferentiable operations (e.g.
minimization/maximization).
13
STAR*D
Preference
Stage 1
Treatment
Intermediate
Outcome
Preference
Stage 2
Treatment
Remission
Continue
on Present
Treatment
Bup
Switch
R
Ven
Ser
MIRT
Switch R
+ Bup
Augment R
No
Remission
NTP
+ Bus
+LI
Augment R
+THY
14
STAR*D
• Stage 1 Observation:
• QIDS: low score is desirable
• Preference for type of Stage 1 treatment: Switch or Augment
•
Stage 1Treatment Action: If Stage 1 preference is Switch then randomize
switch to either Ser, Bup or Ven; if Stage 1 preference is Augment then
randomize to augment with Bup or Bus.
•
Stage 2 Observation:
• QIDS: low score is desirable
• Preference for type of Stage 2 treatment: Switch or Augment
•
Stage 2 Treatment Action: If Stage 2 preference is Switch then randomize
switch to either Mirt or Ntp: if Stage 2 preference is Augment then randomize
to augment with Li or Thy
• Patients exit to follow-up if remission is achieved (QIDS ≤ 5).
15
Construct the policy to minimize cost
(or maximize reward)
•Cost: minimum of time to remission and 30 weeks.
•Construct policy so as to minimize average cost
16
Algorithm
• Fitted Q-iteration with linear function approximation.
One estimates the “state-action cost” function at stages
1,2 via a linear model.
•Use voting across bootstrap samples (approximate
double bootstrap) to assess confidence that a particular
action is best.
(cost=-value=-benefit-to-go)
17
18
Conclusion for Stage 1(level 2)
• If QIDS is >13 then both Ven and Bup are best
treatment actions
• If QIDS is <9 then Ser is best treatment action.
• If QIDS is around 10-13 then no real winner(s).
19
Discussion
If modern control methods are to be used with clinical
trial data then these methods
•must accommodate the existence of unknown,
unobserved variables influencing observations at
multiple stages,
•should provide measures of confidence and
•must be combined with modern missing data methods.
20
This seminar can be found at:
http://www.stat.lsa.umich.edu/~samurphy/
seminars/ACC06.09.ppt
Email me with questions or if you would like a
copy!
[email protected]
21
The Problem
• Many patients dropout of the study.
Stage 1
Stage 2
Remit
383
36
Move to
next stage
Dropout
456
260
362
160
Sum
1201
456
22