Transcript Assessing the Total Effect of Time

Dynamic Treatment Regimes,
STAR*D & Voting
D. Lizotte, E. Laber & S. Murphy
ENAR
March 2009
Dynamic treatment regimes are individually tailored
treatments, with treatment type and dosage changing
according to patient outcomes. Operationalize clinical
practice.
k Stages for one individual
Observation available at jth stage
Action at jth stage (usually a treatment)
2
k=2 Stages
Goal: Construct decision rules that input information
available at each stage and output a recommended
decision; these decision rules should lead to a maximal
mean Y.
Y is a known function of
The dynamic treatment regime is the sequence of two
decision rules:
3
Optimal Dynamic Treatment
Regime
satisfies
4
Data for Constructing the Dynamic
Treatment Regime:
Subject data from sequential, multiple assignment,
randomized trials. At each stage subjects are
randomized among alternative options.
Aj is a randomized treatment with known randomization
probability.
5
STAR*D
Preference
Stage 1
Treatment
Intermediate
Outcome
Preference
Stage 2
Treatment
Remission
Continue
on Present
Treatment
Bup
Switch
R
Ven
Ser
MIRT
Switch R
+ Bup
Augment R
No
Remission
NTP
+ Bus
+LI
Augment R
+THY
6
STAR*D Analyses
• X1 includes site, preference for future
treatment and can include other baseline
variables.
• X2 can include measures of symptoms
(Qids), side effects, preference for future
treatment
• Y is (reverse-coded) the minimum of the
time to remission and 30 weeks.
7
Regression-based methods for
constructing decision rules
•Q-Learning (Watkins, 1989) (a popular method from
computer science)
•Optimal nested structural mean model (Murphy, 2003;
Robins, 2004)
• The first method is equivalent to an inefficient version of
the second method, when using linear models, each stages’
covariates include the prior stages’ covariates and the actions
are centered to have conditional mean zero.
8
A Simple Version of Q-Learning –
There is a regression for each stage.
• Stage 2 regression: Regress Y on
obtain
to
• Stage 1 regression: Regress
obtain
to
on
9
for patients entering stage 2:
•
is the average outcome conditional on patient history (no
remission in stage 1; includes past treatment and variables
affected by stage 1 treatment).
•
is the estimated average outcome assuming the “best”
treatment is provided at stage 2 (note max in formula).
•
is the dependent variable in the stage 1 regression for patients
moving to stage 2
10
A Simple Version of Q-Learning –
• Stage 2 regression, (using Y as dependent variable)
yields
• Stage 1 regression, (using
yields
as dependent variable)
11
Decision Rules:
12
Measures of Confidence
• Classical
– Confidence/Credible intervals and/or pvalues concerning the β1, β2.
– Confidence/Credible intervals concerning
the average response if
is used in
future to select the treatments.
13
A Measure of Confidence
for use in
Exploratory Data Analysis
• Voting
– Estimate the chance that a future trial would
find a particular stage j treatment best for a
given sj. The vote for treatment aj* is
14
A Measure of Confidence
for use in
Exploratory Data Analysis
Voting
– If stage j treatment aj is binary, coded in
{-1,1}, then
15
The Vote: Intuition
If
has a normal distribution with
variance matrix
then
is
16
Bootstrap Voting
An inconsistent bootstrap vote estimator of
is
17
Bootstrap Voting
A consistent bootstrap vote estimator of
is
where
is smooth and
18
What does the vote mean?
•
is similar to 1- pvalue for the
hypothesis
in that it
converges, as n increases, to 1 or 0 depending
on the sign of
• If
then the limiting distribution is
not uniform; instead
converges to a
constant.
19
STAR*D
Regression formula at stage 2:
20
21
STAR*D
Regression formula at stage 1:
22
STAR*D
Decision Rule for subjects preferring a switch at stage 1
• if
offer VEN
• if
offer SER
• if
offer BUP
23
STAR*D
Level 2, Switch
24
25
26
Truth in Advertising:
STAR*D
Missing Data + Study Drop-Out
•
•
•
•
1200 subjects begin level 2 (e.g. stage 1)
42% study dropout during level 2
62% study dropout by 30 weeks.
Approximately 13% item missingness for
important variables observed after the start
of the study but prior to dropout.
27
Truth in Advertising:
STAR*D
Multiple Imputation within Bootstrap
• 1000 bootstrap samples of the 1200 subjects
• Using the location-scale model we formed
25 imputations per bootstrap sample.
• The stage j Q-function (regression function)
for a bootstrap sample is the average of the
25 Q-functions over the 25 imputations.
28
Discussion
• We consider the use of voting to provide a measure
of confidence in exploratory data analyses.
• Our method of adapting the bootstrap voting requires
a tuning parameter. It is unclear how to best select
this tuning parameter.
• We ignored the bias in estimators of stage 1
parameters due to the fact that these parameters are
non-regular. The voting method should be combined
with bias reduction methods.
29
This seminar can be found at:
http://www.stat.lsa.umich.edu/~samurphy/
seminars/ENAR2009.ppt
Email me with questions or if you would like a
copy!
[email protected]
30