Transcript BecomingAnOutcomesInformedClinician

Becoming An
Outcomes Informed Clinician
G.S. (Jeb) Brown, Ph.D.
Center for Clinical Informatics
An outcomes informed clinician…
• Uses the best available data on treatment outcomes
to inform the treatment for each client/patient
• Stays current on the latest research on what makes a
difference in treatment outcomes.
• Recognizes the importance of clinician skill in
providing effective treatments.
• Accepts personal responsibility for evaluating and
improving his or her outcomes.
Sources of outcome data…
• Clinical trials designed to draw causal conclusions
regarding the efficacy of various treatments.
• Meta analyses designed to draw conclusions based
on a large sample of clinical trials.
• Studies evaluating the effectiveness of treatments
delivered in the real world without experimental
controls.
• Data from your own clients.
Clinical trials
• Random assignment and experimental controls
designed to control of sources of variance in
outcomes.

Patient factors, treatment method, dose, duration, etc.
• Double blind placebo controlled considered the
“gold standard”
• Use of analysis of variance to determine if
differences between treatments are “statistically
significant”, i.e. unlikely to occur by chance alone.
Analysis of variance
• Statistical procedures used to analyze data from
clinical trails provides an estimate of the impact of
each factor to the eventual outcome of care.
• Analysis of variance can only calculate the variance
for those factor specified in the hypotheses or
“model”
• Warning: Failure to specify all sources of variance
in the model may lead to erroneous and misleading
findings.
Cautionary tale
• The NIMH funded Treatment of Depression
Collaborative Research Project is one of the largest
studies of the treatment of depression over
conducted. 1
• Evaluated outcomes for Cognitive Behavioral
Therapy (CBT) , Interpersonal Therapy (IT),
Placebo, and Anti-depressant Therapy.
• 40+ articles published in peer reviewed journals
• Most failed to account for the single largest source
of variance… the clinician
Initial findings
• Cognitive/behavioral treatment (CBT) and
interpersonal therapy (IPT) were found to produce
comparable benefits to the depressed patients treated 2
• Imipramine found superior to Placebo 2
• Tendency for IPT to be superior for the treatment of
patients with more severe depression. 3
Critique
• Traditional analysis of variance evaluating
treatments as the primary source of variance is
correct only if the researcher is sure that the
clinician does not matter!
• Psychotherapy research shows that the clinician
matters…. A lot! 4-11
• If the clinician may be a source of variance, then it
is necessary to use a hierarchical linear model which
specifies the clinician as a variable and possible
source of variance.
Reanalyzes – using HLM
• No difference between CBT and IT for severe
depression 12

0% of variance due to treatment method; 8% due to clinicians
• Reanalysis of placebo - imipramine comparison
performed including the 9 psychiatrists as a variable 13

3.4% of variance due to medication; 9.1% due to psychiatrist

Top third of psychiatrists had a better outcome with placebo than the
bottom third with imipramine
Clinical trials – feedback studies
• Michael Lambert, PhD and colleagues at the
Brigham Young Comprehensive Clinic conducted a
series of controlled studies investigating whether
providing clinician’s feedback the client’s trajectory
of change would improve outcomes. 14-19
• Feedback found to significant reduce early dropout
and treatment failures.
• Clinicians’ judgment alone, in the formed in
absence of information from the questionnaires,
proved to be a poor predictor treatment failures.
Evidenced based psychotherapy
• For several decades psychotherapy researchers have
attempted to design randomly controlled trails
(RCT) to investigate the effectiveness of specific
methods of psychotherapy.
• Study design analogous to pharmacy trials, except
that designing credible “placebo treatments” is
much more problematic.
• Various treatment methods are being touted as
“evidenced based” by citing the number of RCTs
providing evidence that the treatment exceeded
placebo (or some other treatment).
Psychotherapy “brands”
• The advocacy for the use of specific therapies is
analogous to the advertising of brands of
antidepressant medication.
• Calls for wide spread use of “evidence based
treatments” in psychotherapy is analogous to the
FDA’s insistence that a drug may not be marketed
for the treatment of depression until at least two
studies have shown superiority to placebo.
• Advocates and practitioners of various “evidence
based treatments” have a vested interest in
discouraging the use of “unproven” treatments.
Brand differentiation
• Advocates of psychotherapy brands insist on the
uniqueness of their therapy and the need to adhere
to specific treatment procedures
• Research methodology requires the use of manuals
and other techniques to standardize treatments
• Treatment effectiveness presumed to be dependent
on the correct application of the “active
ingredients” in the psychotherapy method.
Recommended reading
Rigorous review and analyses
of controlled studies on
psychotherapy outcome.
Conclusion: much more
variance resides with the
clinician than with the
treatments.
The Dodo Bird Effect
20-27
Rosenzweig S. (1936)
Some implicit common factors in
diverse methods of psychotherapy: “At
last the Dodo said, ‘Everybody has
won and all must have prizes.’”
Am J Orthopsychiatry 6:412-5.
Meta-analysis & common factors
• Over two decades of meta-analytic studies have
served to reinforce Rosenzweig’s 1936 observation
that different methods of psychotherapy tend to
produce comparable outcomes… the “Dodo Bird
Effect”
• Lack of evidence for specific treatment effects
bolster the argument that almost all of the effects of
psychotherapy are due to factors common to all
psychotherapies. 20-27
Meta-analyses and placebo
• Meta-analysis of multiple studies of antidepressants
let us estimate the relative importance of common
factors (placebo effects) versus drug effects. 28-30
• Placebo effects are a major portion of the measured
improvement.
• Studies reporting large effect sizes for medication
also had large effect sizes for placebos.
• Similar to findings for psychiatrists from TDCRP
data.
Drug effect accounted for 25% of
measured improvement
Antidepressant effect
(25%)
Placebo effect
Natural course of the
illness
Effectiveness studies
• Effectiveness studies attempt to evaluate outcomes
in real world treatment settings.
• Heterogeneous outpatient treatment populations
require use of statistical methods for case mix
adjustment in order to compare results across sites
or clinicians.
• Outcomes “benchmarking” refers to the practice of
comparing outcomes from one sample to outcomes
of another sample used for comparison purposes.
Benchmarking outcomes
• Takuya Minami, PhD and colleagues have
published a series of articles describing one
benchmarking methodology. 31-33
• Meta-analyses of controlled psychotherapy studies
used to establish effect size benchmarks for the
treatment of depression.
• Outcomes for treatment of depression by PacifiCare
Behavioral Health’s network providers found
clinical equivalent to benchmark from clinical
trials.
PBH Outcomes for Depression
1
0.9
0.8
Effect size
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Benchmark: Clinical trials
PBH Network providers
Therapists and Medications
• Analysis of PacifiCare Behavioral Health (PBH)
data reveals 5% of variance due to therapist for
patients receiving psychotherapy alone.
• 35 % of variance due the therapist than those
receiving psychotherapy in combination with a
medication! 34
Cross validation analysis
• Psychotherapists in PBH network ranked based on
all cases from 1999-2002 if sample size =>30;
N=116.
• If a therapist’s mean residualized final score < 0
then clinician rated “Highly effective”; else
clinician rated “Less effective”.
• Outcomes evaluated in the 2003-2004 cross
validation period for a new sample of cases.
Cross validation results
Clinician cross validation results - 2003 to 2004
Therapists assessed on at least 30 cases between 1999-2002.
Highly effective clinicians had mean Change Index Score > 0
Effect size
1
Highly effective clinicianspsychotherapy only
0.9
0.8
0.7
0.6
Highly effective clinicianspsychotherapy and medication
0.5
0.4
Less effective clinicianspsychotherapy only
0.3
0.2
0.1
0
Less effective clinicianspsychotherapy and medication
0-50
51-120
Intake scores (mean split)
Can we improve outcomes?
• Increasing the percentage of patients treated by
highly effective clinicians (as identified through
practice based evidence) is the most direct pathway
open to a health plan seeking to improving
outcomes across a large system of care.
• Organizations may improve outcomes by fostering
outcomes informed care methods within the
organization.
• Resources for Living and Accountable Behavioral
Healthcare Alliance provide two examples using the
ORS.
qu
ar
te
3r
r2
d
00
qu
2
ar
te
4t
r2
h
00
qu
2
ar
t
er
1s
20
tq
02
ua
rte
2n
r2
d
00
qu
3
ar
t
er
3r
20
d
qu
03
ar
te
4t
r2
h
00
qu
3
ar
t
er
1s
20
tq
03
ua
rte
2n
r2
d
00
qu
4
ar
t
er
3r
20
d
qu
04
ar
te
r2
00
4
2n
d
Effect size
RFL results
0.90
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
Training and feedback
Baseline period
Accountable Behavioral HealthCare Alliance
Severity Adjusted Effect Size
ABHA Outcomes by Year
Clients with scores in the clinical range at intake
Benchmark from clinical trials =.83
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
2004
2005
2006
2007
Therapeutic alliance
• A large body of evidence suggests that the
relationship and working alliance between the
clinician and patient is an important factor in the
outcome. 35-41
• Routine use of a session rating/therapeutic alliance
scale may permit clinicians to identify and repair
problems in the working alliance.
SRS Use and Outcomes
Severity Adjusted Effect Size
Use of SRS at Session 1 and Outcome
0.9
0.8
0.7
SRS at session 1
0.6
0.5
0.4
0.3
0.2
0.1
0
No SRS at session 1
SRS Alliance – positively skewed
RLF Alliance Ratings Frequency
(first and last assessment)
100%
80%
60%
40%
20%
0%
First assessment
Last assessment
Alliance rating
Poor
Fair
Good
SRS=0-34
SRS=35-38
SRS=39-40
SRS Change and Outcomes
Severity Adjusted Effect Size
(RFL sample)
First/last alliance
oo
r
Po
or
/P
Po
or
Fa
ir/
G
oo
d/
Po
or
ai
r
Po
or
/F
Fa
ir
Fa
ir/
r
G
oo
d/
Fa
i
Po
or
/G
oo
d
G
oo
d
Fa
ir/
G
oo
d/
G
oo
d
1.6
1.4
1.2
1
0.8
0.6
0.4
0.2
0
Condemnation with faint praise
Change in Alliance and Outcomes for clients with
Good Alliance Rating at First and Last Sessions
Severity Adjusted Effect Size
(RFL sample)
1.4
1.2
1
0.8
0.6
0.4
0.2
0
Improved Alliance
No Change in
Alliance
Worse Alliance
Do outcomes informed clinicians
get better outcomes?
• United Behavioral Health surveyed clinicians
regarding their use of outcome questionnaires and
outcome reports sent by the managed care company
that monitored patient progress. 42
• Results showed that clinicians who reported using
outcome information had patients who also reported
greater improvement at 6 months from baseline.
• Improvement per session was greatest among patients
whose clinicians reported reading the outcome report
and using outcome measures in their clinical practice.
Regence Outcomes for Depression
Regence solicited self identified “outcomes informed clinicians”
to participate in an outcomes informed care pilot project.
1.2
1
External Benchmarks
(derived from published research)
Effect size
0.8
0.6
0.4
0.2
0
Large managed care
network: Depression
Meta analysis of
psychotherapy studies of
treatmentof depression
Regence outcomes
outcomes
Regence outcomes
outcomes
Regence
Regence
informed clinicians:
informed clinicians: All
informed
clinicians: informed
Depression
diagnosesclinicians:
combined
Depression
All diagnoses
combined
Implications for clinicians
• Good news: The clinician matters!!!!!!
• All treatments (including medications!?) are only as
effective as the clinicians delivering the treatment.
• Psychotherapy is profession requiring a high degree
of skill.
• Expert skill is acquired through practice and
performance feedback.
• Routine use of outcome and alliance questionnaires
provide clinicians the means to evaluate and improve
their skill and effectiveness.
References
1. Elkin, I., Parloff, M.B., Hadley, S.W., Autry, J.H., 1985. NIMH
treatment of depression collaborative research program: background
and research plan. Archives of General Psychiatry 42, 305–316.
2. Elkin, I., Shea, T., Watkins, J.T., Imber, S.D., Sotsky, S.M., Collins,
J.F., et al., 1989. National institute of mental health treatment of
depression collaborative research program: general effectiveness of
treatments. Archives of General Psychiatry 46, 971–982.
3. Elkin, I., Gibbons, R.D., Shea, M.T., Sotsky, S.M., Watkins, J.T.,
Pilkonis, P.A., et al., 1995. Initial severity and differential treatment
outcome in the National Institute of Mental Health treatment of
depression collaborative research program. Journal of Consulting
and Clinical Psychology 63, 841–847. 260
4. Luborsky L, Crits-Christoph P, McLellan T, et al. 1986. Do therapists vary
much in their success? Findings from four outcome studies. Am J
Orthopsychiatry 56:501-12.
References (continued)
5.
Crits-Christoph P, Baranackie K, Kurcias JS, et al. 1991. Meta-analysis
of therapist effects in psychotherapy outcome studies. Psychother Res
1:81-91.
6. Crits-Christoph P, Mintz J. 1991. Implications of therapist effects for the
design and analysis of comparative studies of psychotherapies. J Consul
Clin Psychol 59:20-6.
7. Wampold BE. 1997. Methodological problems in identifying efficacious
psychotherapies. Psychother Res 7:21-43,
8. Elkin I. 1999. A major dilemma in psychotherapy outcome research:
Disentangling therapists from therapies. Clin Psychol Sci Prac 6:10- 32.
9. Wampold BE, Serlin RC. 2000. The consequences of ignoring a nested
factor on measures of effect size in analysis of variance designs.
Psychol Methods 4:425-33.
10. Huppert JD, Bufka LF, Barlow DH, et al. 2001. Therapists, therapist
variables, and cognitive-behavioral therapy outcomes in a multicenter
trial for panic disorder. J Consul Clin Psychol 69:747-55.
References (continued)
11. Okiishi J, Lambert MJ, Nielsen SL, et al. 2003. Waiting for supershrink:
An empirical analysis of therapist effects. Clin Psychol Psychother
10:361-73.
12. Kim DM, Wampold BE, Bolt DM. 2006. Therapist effects and treatment
effects in psychotherapy: Analysis of the National Institute of Mental
Health Treatment of Depression Collaborative Research Program.
Psychother Res. 16(2): 161-172.
13. McKay, K. M., Imel, Z. E., & Wampold, B. E. (2006). Psychiatrist
effects in the psychopharmacological treatment of depression. Journal
of Affective Disorders, 92, 287-290.
14. Lueger RJ. 1998. Using feedback on patient progress to predict the
outcome of psychotherapy. J Clin Psychol 54:383-93.
15. Lambert MJ, Whipple JL, Smart DW, et al. 2001. The effects of
providing therapists with feedback on patient progress during
psychotherapy: Are outcomes enhanced? Psychother Res 11(1):49-68.
References (continued)
16. Hannan C, Lambert MJ, Harmon C et al. 2005. A lab test and algorithms
for identifying clients at risk for treatment failure. J Clin Psychol
61(2):155-63.
17. Lambert MJ, Harmon C, Slade K et al. 2005. Providing feedback to
psychotherapists on their patients progress: Clinical results and practice
suggestions J Clin Psychol 61(2):165-74.
18. Harmon C, Hawkins, Lambert MJ et al. 2005. Improving outcomes for
poorly responding clients: The use of clinical support tools and feedback
to clients. J Clin Psychol 61(2):175-85.
19. Brown GS, Jones DR. 2005. Implementation of a feedback system in a
managed care environment: What are patients teaching us? J Clin
Psychol 61(2):187-98. Claiborn CD, Goodyear EK. 2005. Feedback in
psychotherapy. J Clin Psychol 61(2):209-21.
20. Rosenzweig S. 1936. Some implicit common factors in diverse methods
of psychotherapy: “At last the Dodo said, ‘Everybody has won and all
must have prizes.’” Am J Orthopsychiatry 6:412-5.
References (continued)
21. Shapiro DA & Shapiro D. 1982. Meta-analysis of comparative therapy
outcome studies: A replication and refinement. Psychol Bull 92:581-604.
22. Robinson LA, Berman JS, Neimeyer RA. 1990. Psychotherapy for
treatment of depression: A comprehensive review of controlled outcome
research. Psychol Bull 108:30-49.
23. Wampold BE, Mondin GW, Moody M, et al. 1997. A meta-analysis of
outcome studies comparing bona fide psychotherapies: Empirically, “All
must have prizes.” Psychol Bull 122:203-15.
24. Ahn H, Wampold BE. 2001. Where oh where are the specific ingredients?
A meta-analysis of component studies in counseling and psychotherapy. J
Counsel Psychol 48:251-7.
25. Chambless DL, Ollendick TH. 2001. Empirically supported psychological
interventions: Controversies and evidence. Annual Rev Psychol 52:685716.
26. Luborsky, L., Rosenthal, R., Diguer, L., et al. 2002. The dodo bird verdict
is alive and well--mostly. J. Psychotherapy Integration Vol 12(1) 32-57
References (continued)
27. Wampold BE. 2001. The great psychotherapy debate: Models, Methods,
and Findings. Mahwah NJ: Lawrence Erlbaum Associates. Wampold
BE, Mondin GW, Moody M, et al. 1997. A meta-analysis of outcome
studies comparing bona fide psychotherapies: Empirically, “All must
have prizes.” Psychol Bull 122:203-15.
28. Kirsch, I & Sapirstein, G. 1998. Listening to Prozac but hearing
placebo: A meta analysis of antidepressant medication. Prevention &
Treatment. 1, Article 0002a, No Pagination Specified
29. Kirsch, I. 2000. Are drug and placebo effects in depression additive?
Biological Psychiatry 47, 733-73.
30. Kirsch, I, Moore, TJ, Scoboria, A, Nicholls, SS. 2002. The emperor's
new drugs: An analysis of antidepressant medication data submitted to
the U.S. Food and Drug Administration. Prevention & Treatment. 5(1),
No Pagination Specified
References (continued)
31. Minami, T., Serlin, R. C., Wampold, B. E., Kircher, J. C., & Brown, G. S.
(In press). Using clinical trials to benchmark effects produced in clinical
practice, Quality and Quantity
32. Minami, T., Wampold, B. E., Serlin, R. C., Hamilton, E., Brown, G. S., &
Kircher, J. (2007). Benchmarking the effectiveness of psychotherapy
treatment for adult depression in a managed care environment.
33. Minami, T., Wampold, B. E., Serlin, R. C., Kircher, J. C., & Brown, G. S.
(2007). Benchmarks for psychotherapy efficacy in adult major depression,
Journal of Consulting and Clinical Psychology, 75, 232-243.
34. Wampold BE, Brown GS. 2005. Estimating variability in outcomes due to
the therapist: A naturalistic study of outcomes in managed care. J Consul
Clin Psychol. 73(5): 914-923.
35. Bachelor, A., & Horvath, A. (1999). The therapeutic relationship. In M.A.
Hubble, B.L. Duncan, and S.D. Miller (eds.). The Heart and Soul of
Change: What Works in Therapy. Washington, D.C.: APA Press, 133-178.
References (continued)
36. Blatt, S. J., Zuroff, D.C., Quinlan, D.M., & Pilkonis, P. (1996).
Interpersonal factors in brief treatment of depression: Further analyses
of the NIMH Treatment of Depression Collaborative Research Program.
J Consul Clin Psychol. 64, 162-171.
37. Bordin, E. S. (1979). The generalizability of the psychoanalytic concept
of the working alliance. Psychotherapy: Theory, Research, and Practice,
16, 252-260.
38. Burns, D., & Nolen-Hoeksema, S. (1992). Therapeutic empathy and
recovery from depression in cognitive-behavioral therapy: A structural
equation model. J Consul Clin Psychol. 60, 441-449.
39. Connors, GJ, DiClemente, CC., Carroll, KM, et al. 1997 The
therapeutic alliance and its relationship to alcoholism treatment
participation and outcome. J Consul Clin Psychol, 65(4), 588-598.
40. Horvath, A. O., & Symonds, B. D. (1991). Relation between working
alliance and outcome in psychotherapy: A meta-analysis. J Consul Clin
Psychol. 38, 139-149.
References (continued)
41. Krupnick, J., Sotsky, SM, Simmens, S et al. (1996) The role of the
therapeutic alliance in psychotherapy and pharmacotherapy outcome:
Findings in the National Institute of Mental Health Treatment of
Depression Collaborative Research Project. J Consul Clin Psychol. , 64,
532-539.
42. Azocar, F., Cuffel, B, McCulloch, J., McCabe, J., Tani, S., Brodey, B.
(2007) Monitoring patient improvement and treatment outcomes in
managed behavioral healthcare. Journal for Healthcare Quality
(March/April)
About the presenter
G.S. (Jeb) Brown is a licensed psychologist with a Ph.D. from Duke
University. He served as the Executive Director of the Center for Family
Development from 1982 to 19987. He then joined United Behavioral
Systems (an United Health Care subsidiary) as the Executive Director for
of Utah, a position he held for almost six years. In 1993 he accepted a
position as the Corporate Clinical Director for Human Affairs
International (HAI), at that time one of the largest managed behavioral
healthcare companies in the country.
In 1998 he left HAI to found the Center for Clinical Informatics, a
consulting firm specializing in helping large organizations implement
outcomes management systems. Client organizations include Resources
for Living, Regence, United Behavioral Health, Accountable Behavioral
Health Care Alliance, and assorted treatment centers.
Dr. Brown continues to work as a part time psychotherapist at
behavioral health clinic in Salt Lake City, Utah. He does measure his
outcomes.