Transcript SMART Experimental Designs for Developing Adaptive

Treatment Effect Heterogeneity
&
Dynamic Treatment Regime
Development
S.A. Murphy
Dynamic treatment regimes (DTRs) are
individually tailored treatments, with treatment
type and dosage changing according to
individual outcomes.
***utilize treatment effect heterogeneity to
individualize treatment***
2
Example of a DTR
•Adaptive Drug Court Program for drug
abusing offenders.
•Goal is to minimize recidivism and drug
use.
•Marlowe et al. (2008, 2009, 2011)
3
Adaptive Drug Court Program
non-responsive
low risk
As-needed court hearings
+ standard counseling
As-needed court hearings
+ ICM
non-compliant
high risk
non-responsive
Bi-weekly court hearings
+ standard counseling
Bi-weekly court hearings
+ ICM
non-compliant
Court-determined
disposition
4
Treatment Effect Heterogeneity
• Focus on Theory: Used to deepen understanding
of underlying causal, mechanistic structure
• Focus on Practice: Used to improve decision
making in practice
– For Whom, When, and in Which Context, might a
specific treatment be most useful?
– This is our focus today
Treatment Effect Heterogeneity
&
DTR Development
• Take Advantage of Treatment Effect
Heterogeneity in Design of Intervention Trial
– Embedded tailoring variables
– Part of “treatment action”
• Take Advantage of Treatment Effect
Heterogeneity in Design of the DTR.
– Data analyses
Pelham ADHD Study
Continue, reassess monthly;
randomize if deteriorate
Yes
8 weeks
Begin low-intensity
BMOD
AssessAdequate response?
No
Augment with other treatment
Random
assignment:
Intensify Current Treatment
Random
assignment:
Continue, reassess monthly;
randomize if deteriorate
8 weeks
Begin low dose
Med
Intensify Current Treatment
AssessAdequate response?
Random
assignment:
No
Augment with other Treatment
Txt Effect Heterogeneity 
Embedded Tailoring Variable
• Embedded Tailoring Variables: (a) Teacher
reported Impairment Scale, (b) Teacher
reported individualized list of target behaviors
• Non-response is assessed at 8 weeks and every
4 weeks thereafter.
Txt Effect Heterogeneity 
Embedded DTRs
4 Embedded DTRs
1) Start with BMOD; only if nonresponse
criterion reached, augment with MED
2) Start with BMOD; only if nonresponse
criterion reached, intensify BMOD
3) Start with MED; only if nonresponse criterion
reached, augment with BMOD
4) Start with MED; only if nonresponse criterion
reached, intensify MED
Oslin Alcoholism Trial
NTX
8 wks Response
Random
assignment:
Early Trigger for
Nonresponse
Random
assignment:
TDM + NTX
CBI+MM
Nonresponse
CBI +NTX+MM
Random
assignment:
8 wks Response
NTX
Random
assignment:
TDM + NTX
Late Trigger for
Nonresponse
Random
assignment:
Nonresponse
CBI +MM
CBI +NTX+MM
Txt Effect Heterogeneity 
Embedded Tailoring Variable &
Embedded DTR
• Embedded Tailoring Variable: heavy
drinking days (HDD)
• First randomization is between treatment
actions: move to stage 2 if 2 HDDs versus
move to stage 2 if 5 HDDs
• 8 Embedded DTRs
A Data Analysis Method for Utilizing Treatment
Effect Heterogeneity to Construct a “More
Deeply Tailored” DTR: Q-Learning
Subject data from sequential, multiple assignment,
randomized trials. At each stage subjects are
randomized among alternative options.
Aj is a randomized action with known randomization
probability. Binary actions with P[Aj=1]=P[Aj=-1]=.512
Dynamic Treatment Regime (DTR)
• The DTR is given by a sequence of decision
rules, one per stage of treatment (here 2
stages)
DTR= f d1 (X 1); d2 (X 1; A 1; X 2 )g
• Goal: Construct f d1 (X 1); d2 (X 1; A 1; X 2 )g
for which the expected outcome E d1 ;d2 [Y ] is
maximal.
13
Q-Learning
• Q-Learning (Watkins, 1989; Ernst et al., 2005;
Murphy, 2005) (a popular method from
computer science)—generalizes regression to
multiple stages
• Q-Learning uses dynamic programming
arguments combined with linear regression
estimation of conditional means.
14
Simple Version of Q-Learning –
There is a regression for each stage.
• Stage 2 regression: Regress Y on
to obtain
• Stage 1 regression: Regress
to obtain
on
15
for subjects entering stage 2:
•
is the predicted end of stage 2 response when the
stage 2 treatment is equal to the “best” treatment.
•
is the dependent variable in the stage 1 regression
for patients moving to stage 2
16
A Simple Version of Q-Learning –
• Stage 2 regression, (using Y as dependent variable)
yields
• Arg-max over a2 yields
17
A Simple Version of Q-Learning –
• Stage 1 regression, (using
yields
as dependent variable)
• Arg-max over a1 yields
18
Pelham ADHD Study
Continue, reassess monthly;
randomize if deteriorate
Yes
8 weeks
Begin low-intensity
BMOD
AssessAdequate response?
No
Augment with other treatment
Random
assignment:
Intensify Current Treatment
Random
assignment:
Continue, reassess monthly;
randomize if deteriorate
8 weeks
Begin low dose
Med
Intensify Current Treatment
AssessAdequate response?
Random
assignment:
No
Augment with other Treatment
ADHD Example
• (X1, A1, R1, X2, A2, Y)
– Y = end of year school performance
– R1=1 if early responder; =0 if early non-responder
– X2 includes the month of non-response, M2, and a
measure of adherence in stage 1 (S2 )
– S2 =1 if adherent in stage 1; =0, if non-adherent
– X1 includes baseline school performance, Y0 ,
whether medicated in prior year (S1), ODD (O1)
– S1 =1 if medicated in prior year; =0, otherwise.
20
ADHD Example
• Stage 2 regression for Y:
(1; Y0 ; S1 ; O1 ; A 1 ; M 2 ; S2 )®2 +
A 2 (¯21 + A 1 ¯22 + S2 ¯23 )
• Stage 1 outcome: R1 Y + (1 ¡ R1 ) Y^
Y^ = (1; Y0 ; S1 ; O1 ; A 1 ; M 2 ; S2 ) ®
^2+
maxa2 ( ¯^21 + A 1 ¯^22 + S2 ¯^23 )a2
21
Dynamic Treatment Regime Proposal
IF medication was not used in the prior year
THEN begin with BMOD;
ELSE select either BMOD or MED.
IF the child is nonresponsive and was nonadherent, THEN augment present treatment;
ELSE IF the child is nonresponsive and was
adherent, THEN select intensification of
current treatment.
22
Future Challenges
•High dimensional data; investigators want to
collect real time data
• Feature construction & Feature selection
• Many stages or infinite horizon
This seminar can be found at:
http://www.stat.lsa.umich.edu/~samurphy/
seminars/JSM_Txt_Heterogeneity2012.ppt
23