Transcript ARR 2 pp
Heterogeneity is not always noise Frank Davidoff 29 March 2012 Heterogeneity is not always noise Heterogeneity Composition from diverse elements or parts; multifarious composition Oxford English Dictionary 2 Heterogeneity is not always noise The Heterogeneity Problem Heterogeneity: You can’t live with it, and you can’t live without it 3 Heterogeneity is not always noise Today’s territory • How heterogeneity interferes with causal inference in clinical science • How heterogeneity also deepens our knowledge • How the effects of heterogeneity play out differently in improvement science • How we can begin to manage the effects of heterogeneity 4 Heterogeneity is not always noise Benefit from Drug X: treated population Results from a standard clinical trial in “ICA” patients RCT Rx benefit: ARR 2 percentage points (pp) 5 Heterogeneity is not always noise Heterogeneity of treatment effect: main sources • Variation in outcome risk when the primary disease is untreated (mainly biological and behavioral variation) • Treatment-related harm • Competing risk • Direct treatment-effect modification 6 Heterogeneity is not always noise How summary results of trials can be misleading Hypothetical example Control - Rx event event rate/100 rate/100 RRR ARR – PP (Prob) NNT Overall result 8 6 0.25 2 (0.02) 50 Average risk subjects 4 3 0.25 1 (0.01 100 High risk subjects 80 60 0.25 20 (0.20) 5 Modified from Kent et al, Trials 2010;11:85 7 Heterogeneity is not always noise Major differences in therapeutic benefit Low- vs. high-risk subgroups in risk stratified analysis • Surgery for carotid stenosis • Anticoagulation in non-valvular atrial fibrillation • CABG for coronary artery disease • Statin therapy as primary prevention in coronary disease • Invasive and non-invasive therapies for acute coronary syndromes • tPA and PCI in ST-elevation myocardial infarction • Drotrecogin in severe sepsis Kent et al, Trials, 2010:11:85 8 Heterogeneity is not always noise Benefit from Drug X: high-risk patient subgroup Risk-stratification of results from clinical trial in “ICA” patients RCT ARR 2 pp Risk stratification Rx benefit: ARR 20 pp 9 Heterogeneity is not always noise Benefit from Drug X: individual high-risk patient Real world results in a “usual” local care system Risk RCT stratification ARR 2 pp ARR 20 pp General Hospital Admin rate 40% Rx benefit: ARR 8 pp 10 Heterogeneity is not always noise Benefit from Drug X: individual high-risk patient Real world results in a local care system that successfully supports changes QI Program ??? RCT ARR 2 pp Risk stratification ARR 20 pp Community Hospital – Admin rate 95% Rx benefit: 19 pp 11 Heterogeneity is not always noise Benefit from Drug X: individual high-risk patient Real world results in a local care system that has trouble supporting changes QI program ??? RCT ARR 2 pp Risk stratification ARR 20 pp Proprietary Hospital – Admin rate 60% Net benefit: 12 pp 12 Heterogeneity is not always noise Heterogeneity of improvement effect: main sources Improvement interventions: • Consist of multiple components: hard to standardize; easily mixed and matched 13 Heterogeneity is not always noise A multi-component improvement intervention: The Michigan ICU central line infection control study • In addition to introducing checklists, prep carts, new skin antiseptic, organizers and leaders: • Recruited advocates within the organization • Kept the team focused on goals • Created alliances with central administration to secure resources • Shifted power relations (particularly with nurses) • Developed social and reputational incentives for cooperating • Opened channels of communication with units that face the same challenges • Used audit and feedback 14 Heterogeneity is not always noise Heterogeneity of improvement effect: main sources Improvement interventions: • Consist of multiple components: hard to standardize; easily mixed and matched • Must first be absorbed and adapted: they change in the process (also easily shared, spread) 15 Heterogeneity is not always noise Heterogeneity of improvement effect: main sources Improvement interventions: • Consist of multiple components: hard to standardize; easily mixed and matched • Must first be absorbed and adapted: change in the process (also easily shared, spread) • Are context-dependent: context can’t be “controlled out” 16 Heterogeneity is not always noise Heterogeneity of improvement effect: main sources Improvement interventions: • Consist of multiple components: hard to standardize; easily mixed and matched • Must first be absorbed and adapted: they change in the process (also easily shared, spread) • Are context-dependent: context can’t be “controlled out” • Are unstable by design: refined over time in response to feedback (“reflexiveness”) 17 Heterogeneity is not always noise Change factor analysis: detail-level An “ex post” theory of a quality improvement program: Michigan study • Isomorphic (peer) pressure applied to join the project • Networked community formed with strong horizontal links • Bloodstream infections reframed as a social problem • Interventions used to shape a “culture of commitment” • Data harnessed as a disciplinary force • “Hard edges” used Dixon-Woods et al, Milbank Quarterly, 2011;89:167-205 18 Heterogeneity is not always noise Change factor analysis: mid-level Improving survival after acute myocardial infarction: (AMI) • Organizational values and goals • Senior management involvement • Broad staff presence and expertise in AMI care • Communication and coordination among staff groups • Support for staff problem solving and learning Curry LA, et al. Ann Intern Med 2011;154:384-90 19 Heterogeneity is not always noise Change factor analysis: high-level In-depth field studies in 9 US/UK hospitals • Six universal challenges: structural, cultural, political, educational, emotional, physical & technological • Single factor (even “dominant” set) rarely explains “heterogeneity of improvement effect” • Answers lie in interactions among a multiplicity of factors • Quality a multi-level phenomenon • “Universal but variable” thesis: six challenges same everywhere, but specifics vary within them – the “cityscape phenomenon” Bate P, et al. Organizing for Quality, 2008 20 Heterogeneity is not always noise SUMMARY Heterogeneity is everywhere in medicine • Interferes with detection of causal relationships noise BUT • Also key source of information regarding individual risk and outcome signal 21 Heterogeneity is not always noise CONCLUSIONS In order to use heterogeneity as a source of knowledge • In clinical science • Need better techniques for understanding effects of biological and behavioral variation on clinical outcomes • In improvement science • Need better techniques for understanding effects of social factor variation on performance change outcomes • Everyday challenge for everyone • Observe, record, reflect, model, share: you might just come up with the techniques we need 22 Heterogeneity is not always noise REFERENCES Davidoff F. Heterogeneity is not always noise. JAMA 2009;302:2580-6. Kent DM, et al. Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal. Trials 2010;11:85 Provost L. Analytical studies: a framework for quality improvement design and analysis. BMJ Qual Saf 2011;20 [Suppl 1]:i-92-i96. Dixon-Woods M, et al. Explaining Michigan: developing an ex post theory of a quality improvement program. Milbank Q 2011;89:167-205. Kaplan HC et al. The Model for Understanding Success In Quality (MUSIQ): building a theory of context in healthcare quality improvement. BMJ Qual Saf 2012;21:13-20. Curry LA., et al. What distinguishes top-performing hospitals in acute myocardial infarction mortality rates. Ann Intern Med 2011;154:384-90. Bate P, et al. Organizing for Quality. 2008; New York: Radcliffe Publishing 23 Heterogeneity is not always noise ACKNOWLEDGMENTS For helpful comments on this presentation Yale-New Haven Hospital medical directors leadership council SQUIRE development group: David Stevens, Paul Batalden, Greg Ogrinc Mary Dixon-Woods Jane Roessner Jules Hirsch 24