Transcript Slides - CIMPOD Conference 2016

Practical Applications and
Decisions for Using Propensity
Score Methods
Doug Landsittel, PhD
Professor of Medicine, Biostatistics and Translational Science
Director, Section on Biomarkers and Prediction Modeling
CER Track Director, Institute for Clinical Research Education
Director of Biostatistics, Starzl Transplant Institute
Faculty Member, CER Center and Center for Research on Health Care
1.
2.
Study designs and PCORI Methodology Standards
Propensity Score (PS) methods


3.
Developing a decision tool for observational CER


4.
Estimating the PS
Estimating treatment effect with PSs
Systematic review of studies on statistical properties
Decision Tool for Observational Data Analysis Methods for
CER (DecODe CER)
Challenges in clinical research education

Some ongoing programs and resources




Study design must be considered
All observational studies are not created equal
If data have too many inherent limitations…
consider not doing the study
Propensity scores can also have a role in study
design
“The choice of study
design often has
profound
consequences for the
causal interpretation of
study
results.”


Treatments are assigned by the mechanisms of
routine practice.
The actual treatment assignment process or
mechanism is generally unknown.
Developing a Protocol for Observational Comparative Effectiveness Research:
A User’s Guide. Chapter 2. Study Design Considerations. Page 22.
Objectives
CER
Observational
Existing
Record
sExp:
Registry,
EHRs
Survey
Epidemiologic
Research
Studies
Exp:
Exp:
Cohort or
Case-control
Study
NHANES
Systematic Reviews
and Meta Analysis
Summarize multiple
studies
Not
CER
Experimental
Standard
RCT
Variations
of the RCT
QuasiExperimental
Gold
Standard
Exp:
Pragmatic
&/or Cluster
Randomized
Exp: Pre-post
Intervention or
Systematic
Assignment
Analytical
Techniques
Decision Analysis
Account for stochastic
events, costs, utilities

Methods are part of the ACA



Methodology Committee
Numerous advisory committees
Methodology Report



Methodology Standards
5 cross-cutting standards
6 study design specific standards (including causal inference and
observational designs, that include registries and data networks)
http://www.pcori.org/research-results/research-methodology/pcori-methodology-standards

CI-1: Define analysis population using
covariate histories


CI-2: Describe population that gave rise to the
effect estimate(s)



Specify timing; use data from appropriate intervals
Justify exclusions; describe final analysis population
CI-3: Precisely define timing of the outcome
relative to exposure initiation and duration
CI-4: Measure confounders before exposure
and report on confounders

CI-5: Report the assumptions underlying the
construction of propensity scores


Describe resulting groups in terms of the overlap
and balance of potential confounders
CI-6: Assess the validity of the instrumental
variable report the balance of covariates

Describe how the IV satisfies the 3 key assumptions


Would like to say ‘using treatment A versus
treatment B caused an improvement in y’
Usually limited to saying ‘using treatment A
versus treatment is associated with an
improvement in y’
•
Concept can be illustrated through potential outcomes
•
Two treatments (a=1 and a=2) are available
•
Patients receive treatment 1 or treatment 2 at baseline
•
Observe an outcome at a subsequent time Ya



Potential outcomes: Y1 and Y2
Only Y1 or Y2 could be observed, the other is the
counterfactual outcome
The idea of causal inference is to estimate the
difference in potential outcomes

i.e. causal effect = E(Y1-Y2) or some function of that
difference (e.g. relative risk)


Balance the treatment groups w.r.t. the
propensity for being treated



Emulate a randomized trial
PS = an optimal balancing score
Two steps: 1) estimate the PS, and 2) apply the
PS to estimate treatment effects
Step 1: modeling the probability for treatment
 Step 2: match, stratify, adjust, or weight the data
using its propensity score


Select some probability model, as done for
outcome regression




Example: logistic regression: g(T) = Xβ
PS = predicted probabilities of a specific treatment
given their covariate profile
The propensity score model is often limited to
main effects, but it does not need to be
Objective: predicting the PS, not estimating β’s

Not concerned about 10 observations/variable,
parsimony, or other rules used for fitting the outcome
model

Specifying which variables to use in the PS








Discourage exclusion based on non-significant tests
Encourage inclusion of clinically significant variables
Include both confounders and predictors
Dangerous to exclude potential confounders
May use modern regression methods
May need to consider complexities of high
dimensional data with small treatment counts
Pay attention to missing data
PS may by continuous or >2 categories

Greatly complicates the associated methods



Refer to CI-5 from the PCORI Methodology
Standards
Report on covariate balance via standard
differences before and after PS adjustment
Display PS distributions by treatment group



Useful comparisons are limited to areas of overlap
May prefer matching if PS distributions have poor
overlap
Report statistics on sensitivity to unmeasured
confounding

Adjust for the logit(PS) as a covariate in a
multivariable regression



Simple with some theoretical justification
Simulations seem to show greater bias in practice
Create quintiles based on the PS




Estimate treatment effect within strata
Calculate an overall estimate over strata
Investigate standardized differences within strata
Also referred to as sub-classification

Match 1:m depending on sample sizes




Choose a caliper to limit acceptable matches



Common choice = 0.2×standard deviation
Substantial literature on the matching criteria
Analysis based on the matching strategy


Apply existing methods for greedy matching
Existing algorithms in R and Stata, other software
Larger m increases the utilized sample size but may
worsen the utility of selected matches
e.g. conditional logistic regression
Optimal statistical properties in some scenarios

Basic Ideas:
Reweight the population to resemble groups with
equal propensities for treatment
 If PS = P(Trt A), weight those on Trt A by 1/PS, and
those on Trt B by 1/(1-PS)
 Down-weight observations with expected treatment
 Existing algorithms, as used for survey weighting, in
R and Stata, other software




Check if extremes of the PS distributions
overlap between treatment
May truncate weights if extremely large
Optimal statistical properties in some scenarios

http://idbdocs.iadb.org/wsdocs/getdocument.aspx?docnum=35320229


Compare effectiveness of bare metal stents
(BMS) to covered stents (CS) in common iliac
artery (CIA) interventions for aortoiliac
occlusive disease using 2010-14 data from the
Vascular Quality Initiative (VQI)
Outcome: time to loss of primary patency


Patients had unilateral or bilateral CIA stents



Multivariable Cox regression and PS analysis
1,727 unilateral stents; 85.7% BMS, 14.3% CS
1,101 bilateral stents; 83.3% BMS, 16.7% CS
Results still pending

Evaluation of bilateral stents conceptually
similar to unilateral stents




%CS relatively small compared to %BMS






Bilateral stents impose a specific structure to the data
Decided to match patients not stents (bilateral case)
Decided to separate bilateral and unilateral analyses
Exploring 1:1 and 1:2 matching
Difficult to match with m>2; reduces sample size
Now exploring IPTW (similar distributions of PSs)
Bilateral stents: multiple sources of correlation
Less published on PS properties for survival
The approach matters to the final conclusions

PSs only account for measured confounders


Need to check resulting balance achieved after
applying propensity methods



Sensitivity methods are important
Plots of standardized differences
Balance can get worse if a variable is excluded from
the PS model
Describing the distribution of PSs across
treatment groups yields useful information


Decision Tool for Observational Data Analysis
Methods in CER (DecODe CER)
Educational efforts including the Expanding
National Capacity in PCOR through Training
Program





Motivated by CER course at Pittsburgh
Informed the systematic review of the literature
Conducting simulations to fill in the gaps
Working with an advisory committee and
‘stakeholder co-investigators’ to develop
DecODe CER
Funded by PCORI from 3/2014 through 2/2017
Introduction and motivation for causal inference in observational CER
Summary of methods
Static Treatment
Time-Varying Treatment
(e.g. Propensity Scores)
(e.g. Marginal Structural Models)
Restrictions,
Limitations,
Assumptions,
Minimum
study design
considerations
Questions, initial analyses, diagnostics
Input from the systematic review
User answers questions; conducts analyses
(e.g. distribution of propensity scores,
measures of covariate balance)
Input from stakeholders
and advisory committee
Limit or guide methods under consideration
Final table with pros and cons
for each method and scenario

105 studies of propensity scores utilized





102 studies of other causal inference methods






Covariate adjusted (n=29)
Probability weighting (41)
Matching (46)
Stratification (34)
Doubly robust methods (22)
Instrumental variables (18)
Marginal structural models (35)
Structural nested mean models/G-estimation (19)
Other (29)
39 included PS and other methods

Institute for Clinical Research Education CER
course at the University of Pittsburgh



CER Track for MS and Certificate Program
Workshops and invited talks
Many resources have been developed


PCORI methods curriculum (Johns Hopkins)
UC-Davis, OSU and others have methods courses
 http://ctsa-cermethodscourse.org/cer-lessons/
 http://cph.osu.edu/hopes/cer

Pitt’s CER Center website has other resources
 http://www.healthpolicyinstitute.pitt.edu/cerc/about-
comparative-effectiveness-research-center
•
•
Expanding National Capacity in PCOR through
Training (ENACT) Program
The RFA included the following requirements:
•
Target a particular research community
•
Collaborate with specific program partners
Basic, advanced, and experiential training
•
•
Pittsburgh application focused on collaborations
with Minority-Serving Institutions (MSIs)
•
Partners have been instrumental in all phases
•
Other R25s funded at Albert Einstein, Brown
U., MD Anderson, U. of Washington
Program Partners: 6 Institutions from Research
Centers in Minority Institutions (RCMIs)
• Charles R. Drew University of Medicine and Science
• Howard University
• Meharry Medical College
• Morehouse School of Medicine
• University of Hawai’i at Manoa
• University of Puerto Rico, Medical Science Campus
Developed a fundamental course (=vocabulary ) of
PCOR and an advanced methods and grant writing
course (online through Acatar Learning Environment).



‘Propensity scores’ does not, alone, describe a
specific method
Many other methods utilize the PS, including
doubly robust methods
The PS can be a valuable tool for exploring the
nature of treatment selection

Funded by a Patient-Centered Outcomes Research Institute (PCORI)
Project Award (ME-1306-03827)


Disclaimer: All statements in this report, including its findings and
conclusions, are solely those of the author and do not necessarily represent
the views of the PCORI, its Board of Governors, or Methodology Committee.
Investigative Team
Sally Morton, PhD, Joyce Chang, PhD `
 Margaret Chen, MS, Andrew Topp, PhD Candidate
 Librarians: Ester Saghafi, MEd,MLS, Barb Folb, MM, MLS, MPH, Andrea
Ketchum, MLIS


Stakeholders Co-investigators


Michael Schneider, PhD, DCC, Pam Smithberger, PharmD, MS
Advisory Board

Sharon-Lise Normand, PhD (Harvard), Dylan Small, PhD (Penn), Maria
Glymour, PhD (UCSF), Dominick Esposito, PhD (Mathematica Policy
Institute), Gerard Brennan, PhD, PT (Intermountain Healthcare)
•
•
Funded by the AHRQ (through PCORI trust funds)
Investigative Team and Executive Committee
(Pittsburgh)
•
•
Advisory Board (from the MSIs and Pittsburgh)
•
•
•
•
W. Kapoor, S. Morton, D. Rubio, E. Davis, K. Abebe,
K. McTigue, C. Shen, V. Gilliam, L. Bell
P. Davis, E. Garcia-Rivera, E. Miller, N. Morone, S.M.
Nouraie, C. Pettigrew, M.F. Lima, A. Quarshie, T.B.
Seto, M.A. Shaheen, J. South-Paul, C. Wilkins
ENACT Fellows and students
Other leadership at the MSIs
Dean’s office of the U. of Pittsburgh SOM (A. Levine)
Feel free to contact me at [email protected] or
[email protected].