Transcript J Consul Clin Psychol - A Collaborative Outcomes Resource Network

Outcomes Informed Care
An introduction to concepts, research, and practical applications
G.S. (Jeb) Brown, Ph.D.
Center for Clinical Informatics
Contributors:
Carson Graves & Christa Castaneda
Regence
Daryl Quick & Dennis Henderson
Western Psychological and Counseling
Top 10 reasons not to
measure outcomes
10. Changes in the patients’ self report of
frequency/severity of symptoms is not a valid
measure of outcome.
9. Therapists will cheat and submit phony data.
8. Patients don’t like to complete questionnaires.
7. Our patients are different.
6. Too busy doing other more important things
(paper work, etc).
Top 10 reasons not to
measure outcomes
5. Insurance company will use information to deny
benefits.
4. Insurance company will use information to
harm therapists
3. It is a violation of patient rights.
2. We practice evidence based medicine so we
know our outcomes are good
And the # 1 reason is
There is no benefit to the
patient or clinician!
What does the research show?
• Decades of research have shown that
psychotherapy is effective for a variety
of conditions.
• Likewise, decades of research have
shown that medication is effective for a
variety of conditions.
• Sometimes, combination treatment has
seemed to be even more effective.
But…
…There is great variability in outcomes from one
patient to next, whether the treatment is
•
Psychotherapy
•
Medication
•
Combination treatment.
What accounts for the
variance in outcomes?
Furthermore…
• Decades of research support the assertion that
different methods of psychotherapy produce
similar results.
• Likewise, different psychotropic medications may
produce similar results.
How to choose one?
What accounts for the
variance in outcomes?
The Dodo Bird Effect 1-9
Rosenzweig S. (1936)
Some implicit common factors in
diverse methods of psychotherapy: “At
last the Dodo said, ‘Everybody has
won and all must have prizes.’”
Am J Orthopsychiatry 6:412-5.
Meta-analysis &
common factors
• Three decades of meta-analytic studies have served
to reinforce Rosenzweig’s 1936 observation
• Lack of evidence for specific treatment effects
bolster the argument that almost all of the effects of
psychotherapy are due to factors common to all
psychotherapies.
• No evidence that effect sizes have increased over the
past three decades of psychotherapy/pharmacy
research!!!!
Meta-analysis and placebo
 Meta-analysis of multiple studies of
antidepressants let us estimate the relative
importance of common factors (placebo
effects) versus drug effects. 10-12
 Placebo effects are a major portion of the
measured improvement.
 Studies reporting large effect sizes for
medication also had large effect sizes for
placebos.
Drug effect accounts for 25%
of measured improvement
Antidepressant effect
(25%)
Placebo effect
Natural course of the
illness
In the last decade…
• Meta-analyses reveal that the clinician is more
important than the technique in the variance in
psychotherapy outcomes. 20-32
• Some recent analyses show the prescribing
clinician is at least as important as the drug in
pharmacotherapy outcomes.
• Other recent analyses suggest the
psychotherapist effects the impact of
medication in combination treatment!
Clinician as active ingredient




Traditional analysis of variance assumes the clinician
does not matter
Hierarchical linear modeling specifies the clinician as a
variable and possible source of variance
Reanalysis of NIMH data: clinician is primary source of
variance
(Bolt, Dong-Min & Wampold, 2006) and (McKay,
Imel & Wampold, 2006)
Analysis of managed care data: therapist is primary
source of variance
(Wampold and Brown, 2005)
Cross validation analysis
 Psychotherapists in national managed care
network ranked based on all cases from 19992002 if sample size =>30
 If a therapist’s mean residualized final score <
0 then clinician rated “Highly effective”; else
clinician rated “Less effective”.
 Outcomes evaluated in the 2003-2004 cross
validation period for a new sample of cases.
Cross validation results
Clinician cross validation results - 2003 to 2004
Therapists assessed on at least 30 cases between 1999-2002.
Highly effective clinicians had mean Change Index Score > 0
Effect size
1
Highly effective clinicianspsychotherapy only
0.9
0.8
0.7
0.6
Highly effective clinicianspsychotherapy and medication
0.5
0.4
Less effective clinicianspsychotherapy only
0.3
0.2
0.1
0
Less effective clinicianspsychotherapy and medication
0-50
51-120
Intake scores (mean split)
What the heck is effect size?
 An effect size of 1 means the client improved
on standard deviation on the outcome
measure.
 An effect size of .8 or high is considered large.
 Meta-analyses of large sample of
psychotherapy studies suggest the effect size
for psychotherapy is approximately .8
 There is no evidence that effect sizes have
increase over past 30 years!
Again,
why measure outcomes?
• A large, growing body of research over the
past decade suggests that
routine measurement of outcomes leads
to improved outcomes, particularly for
those patients most at risk. 26-35
What improves, what doesn’t:
• +
Cases at risk for negative outcome
• -- Clinician ability to identify failing
cases
(M. Lambert, Personal communication)
Feedback must become a
routine part of clinical practice!
And so we conclude…
In behavioral healthcare, focusing
on the treatment (therapy
technique, medication) is not
enough to optimize outcomes –
we must also focus on the
outcomes for each patient
and the skill of the clinician!
What should be validated?
The Psychotherapist!
“We are at the origin of a revolution of using outcomes to
inform practice and guide management of services. Care
must be taken to use this precious data to benefit patients
rather than curtail costs. In the end, we (practitioners,
researchers and third parties) should be united in our
desire to optimize the benefits that psychotherapy can
provide to patients. To do so, we must emphasize those
aspects of psychotherapy that account for the variability in
outcomes. At this point, the evidence indicates that the
psychotherapists is critical.” (page 208)
Bruce Wampold, PhD
In Evidence-Based Practices in Mental Health;
Norcross, Beutler & Levant (Eds), 2006
Outcomes
informed
clinicians…
1. Recognize the importance of clinician skill in
providing effective treatments.
2. Support the desire to improve outcomes by
actively evaluating them, and applying the
feedback to the treatment.
3. Use practice-based evidence to inform the
use of evidence-based practices.
What to measure, with what?




Patient self-report outcome
questionnaires
Data capture as automated as possible
Measure must be sensitive to change
during treatment at all levels of
functioning
Psychometrically sound
So many measures,
so little time!




Dx-specific vs. global
Population-specific vs. single measure
Sensitivity to change: pre-post vs. session-tosession
No industry standards - Everyone has a
favorite
The truth about
outcomes questionnaires!
 All patient self report outcome questionnaires tend to
load on a common factor: “global distress”
 Due to the high degree of correlation between items,
well constructed questionnaires of 10-15 items can
have coefficients of reliability and construct validity
comparable to measures of 30 or more items.
 Even ultra brief questionnaires of 4-9 items may have
adequate reliability and validity for must measurement
needs.
Measurement 2.0
Measurement 1.0 0
Measurement 2.0
 Reliance on copyrighted and
published questionnaires
 Item bank and resulting
questionnaires belong to community
of users.
 Copyright holder may charge
fees for the use of
questionnaires
 Copyright holder may place
conditions or restrictions on
the use of questionnaires
 No fees for questionnaires
constructed from items in the shared
item bank
 Each organization is responsible for
their own measurement methods and
determines the appropriate content
and use of questionnaires
Measurement 2.0
Questionnaire Development
Measurement 1.0
0

A pool of items are tested in various samples

Item analysis used to select items for final
questionnaire

Questionnaire validated, usually in correlation
studies with questionnaires measuring the same
construct

Questionnaire published in final form


Manual published
Many years may pass before a new version is
published
Measurement 2.0

A pool of items are tested in various samples

Item analysis used to select items for multiple
versions of the questionnaires, depending on
the needs of the users

Construct validity determined by factor analysis
and comparison of results to known constructs.
Various organizations may conduct correlations
studies to satisfy internal skeptics.

Questionnaires are constantly evolving as data
accumulates and measure needs change

Online manual constantly update as data
accumulates and needs of users dictate

Multiple versions available, with the community
of users determining which versions offer the
greatest utility
Another Measurement 2.0 project
(added comment)
Outcomes Informed Process





One-page self-report questionnaires
Administer every session
Fax to 800# when most convenient
View data in within 24 hours on secure
personal webpage
Discuss with patients; create a culture
of feedback within each treatment.
Brief Adult Screening &
Outcome Questionnaire
15787
Version 10.0
Org ID:
Site ID:
Date completed
Session #
/
/
Clinician ID:
Client ID:
Please print clearly
This brief questionnaire asks about some of the most commonly reported
Sometimes
thoughts, feelings and behaviors among adults seeking
behavioral health
treatment. Please think about the past two weeks and indicate how often each
of the following occurred. This will help you and your therapist/doctor to
plan your treatment and monitor your improvement.
In the past two weeks, how often
did you
Never
Hardly Someever
times
Often
Very
often
…feel unhappy or sad?......................
…have little or no energy?.................
…have a hard time getting along with
family, friends or coworkers?..............
…feel hopeless about the future?...........
…have a hard time paying attention.........
…feel unproductive at work or other daily
activities?................................
..
…feel tense or nervous?....................
…have problems with sleep (too much or too
little)?...................................
…feel lonely?..............................
…think about harming yourself?.............
…have someone express concerns about your
alcohol or drug use?.......................
…have more than five drinks of alcohol at
one time?..................................
…have a problem at work, school, or home
because of alcohol or drug use?...........
Please estimate the percent of your productivity at work, school or
other important daily activities that is lost due to symptoms of
stress, anxiety or depression.
If this is not your first session,
please take a moment to give feedback
on your most recent session.
I felt that we talked about the
things that were important to me.
I felt that the therapist liked and
understood me.
I felt that the session was helpful.
I felt confident that the therapist
and I worked well together.
%
Agree Somewhat Not Somewhat Do not
agree sure disagree agree
For more information on this and other questionnaires visit
www.psychoutcomes.org
15787
System Features
•
•
•
Flexible use of brief questionnaires
Fax to toll free numbers of data entry
Centralized data center with capabilities to:
1. to capture and archive data
2. utilize advanced algorithms to score, analyze and interpret
data
3. distribute desktop data files with integrated Clinician’s
Decision Support Toolkit
•
Use of TWiki Web 2.0 technology to provide an online
platform to distribute information, elicit feedback, and
foster clinician collaboration.
Flexibility & evolution
• The outcomes system is designed for flexibility
and responsiveness to input from users.
• ACORN item repository permits use of
questionnaires tailored to the measurement
task.
•
System design and features evolve over time based on
provider feedback.
• System permits rapid prototyping, testing and
distribution of Clinician’s Desktop Decision
Support Toolkits designed for specific purposes.
 www.psychoutcomes.org
 Non-profit set up to encourage the use of
client/patient-completed outcome measures in
behavioral health care and related fields.
 TWiki site provides information, fosters
collaboration, and offers support for
organizations launching and nurturing
outcomes-informed care initiatives.
Therapeutic alliance
 A large body of evidence suggests that the
relationship and working alliance between the
clinician and patient is an important factor in
the outcome. 36-42
 Routine use of a session rating/therapeutic
alliance scale may permit clinicians to identify
and repair problems in the working alliance.
Recommendations to clinicians
 Practitioners are encouraged to make the creation and cultivation of
the therapy relationship, characterized by the elements found to be
demonstrably and probably effective, a primary aim of their
treatment.
 Practitioners are encouraged to adapt the therapy relationship to
patient characteristics in ways shown to enhance outcomes.
 Practitioners are encouraged to routinely monitor patients’ responses
to the therapy relationship and ongoing treatment. Such monitoring
leads to increased opportunities to repair alliance ruptures, improve
the relationship, modify technical strategies, and avoid premature
termination.
Norcross & Lambert (page 218); In Evidence-Based Practices in
Mental Health, Norcross, Beutler & Levant (Eds), 2006
Benchmarking outcomes
• Takuya Minami, PhD and colleagues have published a
series of articles describing one benchmarking
methodology. 43-45
• Meta-analyses of controlled psychotherapy studies used
to establish effect size benchmarks for the treatment of
depression.
• Outcomes for treatment of depression by PacifiCare
Behavioral Health’s network providers found clinically
equivalent to benchmark from clinical trials.
Can we improve outcomes?

Increasing the percentage of patients treated by highly
effective clinicians (as identified through practice based
evidence) is the most direct pathway open to a health
plan seeking to improving outcomes across a large
system of care.

Organizations may also improve outcomes by fostering
outcomes informed care methods within the
organization.
Recommended reading
Multiple perspectives on
the question of what really
makes a difference in
treatment outcomes.
Comment: In depth
debate and dialog
among top researchers
in the field.
Recommended reading
Rigorous review and
analyses of controlled
studies on psychotherapy
outcome.
Conclusion: much more
variance resides with
the clinician than with
the treatments.
Recommended reading
Comprehensive review of
the research on the role of
the therapeutic relationship
and factors that make a
difference in outcomes.
Evidenced based
relationships?
References
1.
2.
3.
4.
5.
Rosenzweig S. 1936. Some implicit common factors in diverse methods
of psychotherapy: “At last the Dodo said, ‘Everybody has won and all
must have prizes.’” Am J Orthopsychiatry 6:412-5.
Shapiro DA & Shapiro D. 1982. Meta-analysis of comparative therapy
outcome studies: A replication and refinement. Psychol Bull 92:581-604.
Robinson LA, Berman JS, Neimeyer RA. 1990. Psychotherapy for
treatment of depression: A comprehensive review of controlled outcome
research. Psychol Bull 108:30-49.
Wampold BE, Mondin GW, Moody M, et al. 1997. A meta-analysis of
outcome studies comparing bona fide psychotherapies: Empirically, “All
must have prizes.” Psychol Bull 122:203-15.
Ahn H, Wampold BE. 2001. Where oh where are the specific
ingredients? A meta-analysis of component studies in counseling and
psychotherapy. J Counsel Psychol 48:251-7.
References (continued)
6.
Chambless DL, Ollendick TH. 2001. Empirically supported
psychological interventions: Controversies and evidence. Annual Rev
Psychol 52:685-716.
7. Luborsky, L., Rosenthal, R., Diguer, L., et al. 2002. The dodo bird
verdict is alive and well--mostly. J. Psychotherapy Integration Vol
12(1) 32-57
8. Wampold BE. 2001. The great psychotherapy debate: Models,
Methods, and Findings. Mahwah NJ: Lawrence Erlbaum Associates.
9. Wampold BE, Mondin GW, Moody M, et al. 1997. A meta-analysis of
outcome studies comparing bona fide psychotherapies: Empirically,
“All must have prizes.” Psychol Bull 122:203-15.
10. Kirsch, I & Sapirstein, G. 1998. Listening to Prozac but hearing
placebo: A meta analysis of antidepressant medication. Prevention &
Treatment. 1, Article 0002a, No Pagination Specified
References (continued)
11. Kirsch, I. 2000. Are drug and placebo effects in depression additive?
Biological Psychiatry 47, 733-73.
12. Kirsch, I, Moore, TJ, Scoboria, A, Nicholls, SS. 2002. The emperor's
new drugs: An analysis of antidepressant medication data submitted
to the U.S. Food and Drug Administration. Prevention & Treatment.
5(1), No Pagination Specified
13. Luborsky L, Crits-Christoph P, McLellan T, et al. 1986. Do therapists
vary much in their success? Findings from four outcome studies. Am
J Orthopsychiatry 56:501-12.
14. Crits-Christoph P, Baranackie K, Kurcias JS, et al. 1991. Meta-analysis
of therapist effects in psychotherapy outcome studies. Psychother
Res 1:81-91.
15. Crits-Christoph P, Mintz J. 1991. Implications of therapist effects for
the design and analysis of comparative studies of psychotherapies. J
Consul Clin Psychol 59:20-6.
References (continued)
16. Wampold BE. 1997. Methodological problems in identifying efficacious
psychotherapies. Psychother Res 7:21-43,
17. Elkin I. 1999. A major dilemma in psychotherapy outcome research:
Disentangling therapists from therapies. Clin Psychol Sci Prac 6:1032.
18. Wampold BE, Serlin RC. 2000. The consequences of ignoring a
nested factor on measures of effect size in analysis of variance
designs. Psychol Methods 4:425-33.
19. Huppert JD, Bufka LF, Barlow DH, et al. 2001. Therapists, therapist
variables, and cognitive-behavioral therapy outcomes in a multicenter
trial for panic disorder. J Consul Clin Psychol 69:747-55.
20. Okiishi J, Lambert MJ, Nielsen SL, et al. 2003. Waiting for
supershrink: An empirical analysis of therapist effects. Clin Psychol
Psychother 10:361-73.
References (continued)
21. Brown GS, Jones ER, Lambert MJ, et al. 2005. Identifying highly effective
psychotherapists in a managed care environment. Am J Managed Care
11(8):513-20.
22. Wampold BE, Brown GS. 2005. Estimating variability in outcomes due to the
therapist: A naturalistic study of outcomes in managed care. J Consul Clin
Psychol. 73(5): 914-923.
23. Elkin, I, Shae, T, Watkins, JT., et al. 1989. National Institute of Mental Health
Treatment of Depression Collaborative Research Program: General
effectiveness of treatments. Archive of General Psychiatry. 46: 971-982.
24. Kim DM, Wampold BE, Bolt DM. 2006. Therapist effects and treatment effects
in psychotherapy: Analysis of the National Institute of Mental Health
Treatment of Depression Collaborative Research Program. Psychother Res.
16(2): 161-172.
25. McKay, KM, Imel, ZE & Wampold, BE. In press. Psychiatrist effects in the
pharmacological treatment of depression. J. Affective Disorders.
References (continued)
26. Hannan C, Lambert MJ, Harmon C et al. 2005. A lab test and algorithms
for identifying clients at risk for treatment failure. J Clin Psychol
61(2):155-63.
27. Lambert MJ, Harmon C, Slade K et al. 2005. Providing feedback to
psychotherapists on their patients progress: Clinical results and practice
suggestions J Clin Psychol 61(2):165-74.
28. Harmon C, Hawkins, Lambert MJ et al. 2005. Improving outcomes for
poorly responding clients: The use of clinical support tools and
feedback to clients. J Clin Psychol 61(2):175-85.
29. Brown GS, Jones DR. 2005. Implementation of a feedback system in a
managed care environment: What are patients teaching us? J Clin
Psychol 61(2):187-98.
30. Claiborn CD, Goodyear EK. 2005. Feedback in psychotherapy. J Clin
Psychol 61(2):209-21.
References (continued)
31. Lueger RJ. 1998. Using feedback on patient progress to predict the
outcome of psychotherapy. J Clin Psychol 54:383-93.
32. Lambert MJ, Whipple JL, Smart DW, et al. 2001. The effects of
providing therapists with feedback on patient progress during
psychotherapy: Are outcomes enhanced? Psychother Res 11(1):49-68.
33. Lambert MJ, Whipple JL, Vermeersch DA, et al. 2002. Enhancing
psychotherapy outcomes via providing feedback on client progress: A
replication. Clin Psychol Psychother 9:91-103.
34. Whipple JL, Lambert MJ, Vermeersch DA, et al. 2003. Improving the
effects of psychotherapy: The use of early identification of treatment
failure and problem-solving strategies in routine practice. J Counsel
Psychol 50(1):59-68.
35. Lambert MJ, Whipple JL, Hawkins EJ, et al. 2003. Is it time for clinicians
to routinely track patient outcome? A meta-analysis. Clin Psychol Sci
Prac 10:288-301.
References (continued)
36. Bachelor, A., & Horvath, A. (1999). The therapeutic relationship. In M.A.
Hubble, B.L. Duncan, and S.D. Miller (eds.). The Heart and Soul of
Change: What Works in Therapy. Washington, D.C.: APA Press, 133178.
37. Blatt, S. J., Zuroff, D.C., Quinlan, D.M., & Pilkonis, P. (1996).
Interpersonal factors in brief treatment of depression: Further analyses
of the NIMH Treatment of Depression Collaborative Research Program. J
Consul Clin Psychol. 64, 162-171.
38. Bordin, E. S. (1979). The generalizability of the psychoanalytic concept
of the working alliance. Psychotherapy: Theory, Research, and Practice,
16, 252-260.
39. Burns, D., & Nolen-Hoeksema, S. (1992). Therapeutic empathy and
recovery from depression in cognitive-behavioral therapy: A structural
equation model. J Consul Clin Psychol. 60, 441-449.
40. Connors, GJ, DiClemente, CC., Carroll, KM, et al. 1997 The therapeutic
alliance and its relationship to alcoholism treatment participation and
outcome. J Consul Clin Psychol, 65(4), 588-598.
References (continued)
41. Horvath, A. O., & Symonds, B. D. (1991). Relation between working alliance
and outcome in psychotherapy: A meta-analysis. J Consul Clin Psychol. 38,
139-149.
42. Krupnick, J., Sotsky, SM, Simmens, S et al. (1996) The role of the therapeutic
alliance in psychotherapy and pharmacotherapy outcome: Findings in the
National Institute of Mental Health Treatment of Depression Collaborative
Research Project. J Consul Clin Psychol. , 64, 532-539.
43. Minami, T., Serlin, R. C., Wampold, B. E., Kircher, J. C., & Brown, G. S. (In
press). Using clinical trials to benchmark effects produced in clinical practice,
Quality and Quantity
44. Minami, T., Wampold, B. E., Serlin, R. C., Hamilton, E., Brown, G. S., & Kircher,
J. (2007). Benchmarking the effectiveness of psychotherapy treatment for adult
depression in a managed care environment.
45. Minami, T., Wampold, B. E., Serlin, R. C., Kircher, J. C., & Brown, G. S. (2007).
Benchmarks for psychotherapy efficacy in adult major depression, Journal of
Consulting and Clinical Psychology, 75, 232-243.