Transcript J Consul Clin Psychol - A Collaborative Outcomes Resource Network

Roles of Common Factors
&
“Therapist “Effects”
in
Therapy Outcomes
Session #0931
G.S. (Jeb) Brown, Ph.D.
Center for Clinical Informatics
Common factors
• The effectiveness of all treatments is due, in
some part, to factors common to all treatments.
• Contact with a helping, caring professional
fosters hope and expectancy.
• We have come to accept the potency of “placebo
effects”, and insist that the effectiveness bona
fide treatments exceeds that of placebo
treatments
• So far, so good. Who can argue with this?
Randomized double-blind
placebo controlled drug trials
• Double blind placebo controlled drug studies provide
an exemplar for estimating the role of common
factors.
• Traditionally, the drug is interpreted as the difference
between placebo and the active drug.
• Meta-analysis of multiple studies of antidepressants
lets us estimate the relative importance of common
factors (placebo effects) versus drug effects.
Meta-analyses and placebo
• Meta-analysis involves the use of a statistical
techniques to combine results from multiple studies
in order in an effort to generalize findings.
• Meta-analysis of multiple studies of antidepressants
let us estimate the relative importance of common
factors (placebo effects) versus drug effects. 1-3
Drug effect accounted for 25% of
measured improvement
Antidepressant effect
(25%)
Placebo effect
Natural course of the
illness
Evidenced based psychotherapy
• For several decades psychotherapy researchers have
attempted to design randomly controlled trails
(RCT) to investigate the effectiveness of specific
methods of psychotherapy.
• Study design analogous to pharmacy trials, except
that designing credible “placebo treatments” is
much more problematic.
• Various treatment methods are being touted as
“evidenced based” by citing the number of RCTs
providing evidence that the treatment exceeded
placebo (or some other treatment).
Psychotherapy “brands”
• The advocacy for the use of specific therapies is
analogous to the advertising of brands of
antidepressant medication.
• Calls for wide spread use of “evidence based
treatments” in psychotherapy is analogous to the
FDA’s insistence that a drug may not be marketed
for the treatment of depression until at least two
studies have shown superiority to placebo.
• Advocates and practitioners of various “evidence
based treatments” have a vested interest in
discouraging the use of “unproven” treatments.
Brand differentiation
• Advocates of psychotherapy brands insist on the
uniqueness of their therapy and the need to adher to
specific treatment procedures
• Research methodology requires the use of manuals
and other techniques to standardize treatments
• Treatment effectiveness presumed to be dependent
on the correct application of the “active
ingredients” in the psychotherapy method.
The Dodo Bird Effect
Rosenzweig S. (1936)
Some implicit common factors in
diverse methods of psychotherapy: “At
last the Dodo said, ‘Everybody has
won and all must have prizes.’”
Am J Orthopsychiatry 6:412-5.
The Dodo Bird Lives!
Wampold BE, Mondin GW, Moody M, et al. (1997). A
meta-analysis of outcome studies comparing bona fide
psychotherapies: Empirically, “All must have prizes.”
Psychol Bull 122:203-15.
Luborsky, L., Rosenthal, R., Diguer, L., et al. 2002
The dodo bird verdict is alive and well--mostly. J.
Psychotherapy Integration Vol 12(1) 32-57
Meta-analysis & common factors
• Over two decades of meta-analytic studies have
served to reinforce Rosenzweig’s 1936 observation
that different methods of psychotherapy tend to
produce comparable outcomes… the “Dodo Bird
Effect”
• Lack of evidence for specific treatment effects
bolster the argument that almost all of the effects of
psychotherapy are due to factors common to all
psychotherapies. 5-11
Real world example
• Human Affairs International (HAI) collected
outcome data from a large number of clinicians
between 1996 and 1998.
• Clinicians were asked to specify the primary
method of psychotherapy (or medication
management only)
• Analyses revealed no significant differences in the
outcome or mean number of sessions across all
treatment methods, including medication
management.
Treatment & outcome
HAI data
Mean Value of Residual Change after Controlling for Severity
(positive values = greater than average change; 90% confidence interval)
Residual Effect Size
0.40
0.30
0.20
0.10
0.00
-0.10
-0.20
Insight
Oriented
(n=135)
Solution
Focused
(n=647)
Behavioral
(n=67)
Cognitive
Behavioral
(n=1226)
Fam ily (n=152)
Medication Other (n=129)
Managem ent
(n=514)
Unknow n
(n=85)
Fidelity and practicality
• Propagation of evidenced based treatment methods
requires some method of measuring fidelity to the
treatment.
• BIG PROBLEM!
• If clinicians reports using Cognitive Behavioral
Therapy, how can we have any confidence that
what that clinician did with a specific patient was
comparable to the “Cognitive Behavioral Therapy”
in the RCTs?
Goldilocks Effect
• Clinicians tend to be eclectic and flexible in their
choice of treatment methods.
• Clinicians and patients tend to try different
treatments until they find something that is “just
about right” for them.
• Patients and clinicians tend to adjust the number
and frequency of sessions depending on the
patient’s level of distress and rate of improvement.
• Result: All treatments appear to have similar
outcomes and patients with good outcomes tend to
use fewer session than patients with poor outcomes.
Recommended reading
Rigorous review and analysis
of controlled studies on
psychotherapy outcome.
Conclusion: much more
variance resides with the
clinician than with the
treatments.
Therapists effects
• Wampold and others argue that researchers have
ignored the individual therapist as a source of
variance.11, 16-24
• The person of the therapist is necessary to delivery
the treatment, and personal characteristics of the
therapist modify the effect of the treatment.
• Factors contributing to therapists effects may
include elements clinical skill and knowledge as
well as personality traits.
RCT and ANOVA – brief history
• Some of the earliest applications of randomized
control group design and analysis of variance were
in agriculture and education. 12,13
• RCT methodology later adopted by medicine and
eventually psychotherapy research. 11,14
• Simple ANOVA is appropriate only if the individual
farmer, teacher or clinician has little or no impact
on the effectiveness of the farming, teaching or
treatment method!
HLM & therapist effects
• Hierarchical Linear Modeling (HLM) is an advance
in statistical methodology that permits us to model
variance at the clinicians level and as well as the
treatment level.
• An rapidly growing body of published research
points to the conclusion that therapist effects almost
certainly exceed specific treatment effects by a
large margin.
Variance due to the clinician
• Published research making use of HLM points to
the conclusion that the clinician accounts for much
more of the variance in psychotherapy outcomes
that treatment method per se. 11, 17-21
• Analyses of PacifiCare Behavioral Health’s massive
database database on patient outcomes confirms
significant variance in psychotherapy outcomes at
the clinician level. 24,25
PacifiCare Behavioral Health
ALERT System
• Initiated an outcomes management program in 1998
using 30 item patient self report questionnaires
administered at regular intervals in treatment.
• ALERT System used to capture data and monitor
patient outcomes in real time.
• Currently over 7,000 clinicians are contributing
outcome data on a regular basis.
• Probably largest database on mental health
outcomes in the world.
PBH research collaboration
• PBH actively sought the involvement of leading
psychotherapy outcomes researchers from leading
academic institutions.
• External researchers actively involved in design of
the measurement system and ongoing analysis of
the data.
• PBH encouraged publication of findings in
academic journals.
The (almost) Bell Curve
PBH data
Solo clinicians with sample sizes => 20
% of clinicians
25%
20%
15%
10%
5%
0%
-0.2 -0.1 0
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Effect Size
Where is the variance?
% of variance due to
therapists in the real world
• Analysis of PacifiCare Behavioral Health (PBH)
data reveals 6% of variance due to therapist. 25
• Patients on medication have a higher % of variance
due the therapist than those receiving
psychotherapy alone.
• Huh??
Therapists and meds
Outcomes (residualized scores) of 15 therapists for
patients with concurrent medication or no medication 25
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
meds
nomeds
Test scores and medication
PBH data
Percentage of cases on medication
as a function of intake score
0
-1
2
90
-9
0
81
-8
0
71
-7
0
61
-6
0
51
-5
0
41
-4
0
31
-3
0
21
020
80%
70%
60%
50%
40%
30%
20%
10%
0%
Which treatment is best?
Goldilocks Effect: Clients tend to get the treatment that
is just about right for them.
Treatment, severity and outcome
2.00
Psychotherapy
only
Effect size
1.50
1.00
Psychotherapy
and medication
0.50
0.00
-0.50
0-20 2130
3140
Normal functioning
4150
5160
6170
Intake score
7180
81- 9090 120
Severe symptoms
Cross validation analysis
• Psychotherapists in PBH network ranked based on
all cases from 1999-2002 if sample size =>30;
N=116.
• If a therapist’s mean residualized final score < 0
then clinician rated “Highly effective”; else
clinician rated “Less effective”.
• Outcomes evaluated in the 2003-2004 cross
validation period for a new sample of cases.
Cross validation results
Clinician cross validation results - 2003 to 2004
Effect size
Therapists assessed on at least 30 cases between 1999-2002.
Highly effective clinicians had mean Change Index Score > 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Highly effective clinicianspsychotherapy only
Highly effective clinicianspsychotherapy and medication
Less effective clinicianspsychotherapy only
Less effective clinicianspsychotherapy and medication
0-50
51-120
Intake scores (mean split)
Risk of not using HLM
• Wampold and colleagues at the University of
Wisconsin recently reanalyzed data from the
National Institute of Mental Health’s Treatment of
Depression Collaborative Research Program
(TDCRP) study using HLM. 26-28
• Prior published reports found significant differences
between two methods of psychotherapy as well as
between placebo and antidepressant medication.
• Reanalysis of psychotherapy data using HLM
revealed that 0% of the variance was due to the
psychotherapy methods, while 8% was attributable
to the therapists. 27
Psychiatrist effects
• Wampold and colleagues also used HLM to
reanalyze the results antidepressant and placebo
legs of the TDCRP study. 28
• Included the 9 individual psychiatrists as a variable.
• Outcome measured by change on patient self report
measure (Beck Depression Inventory).
• 9.1% of the variance due to the psychiatrist; only
3.4% due to the medication.
• Top 3 psychiatrists had a better outcome with
placebo than bottom 3 had with the antidepressant.
Placebo & therapist effects
• Hypothesis: Placebo/common factor effects are
mediated by the clinician/patient relationship.
• Common factors tend to account for much more of
the variance than specific treatment effects.
• If the effects of common factors are medicated by
the clinician/patient relationship, then we would
naturally find much of the variance in outcomes
would be due to the clinician.
• The human factor matters!
• DUH!
What’s a clinician to do?
• If a wide variety of treatments appear to be equally
efficacious, can a therapist do to achieve the best
outcomes possible for their patients?
• A growing body of research supports the use of
repeated administrations of patient self report
outcome questionnaires to monitor response to
treatment. 29-36
• Routine measurement and and early identification
of patients with a poor response to treatment has
been shown to reduce treatment failures.
Therapeutic alliance
• A large body of evidence suggests that the
relationship and working alliance between the
clinicians and patient is an important factor in the
outcome. 39-45
• Routine use of a session rating/therapeutic alliance
scale may permit clinicians to identify and repair
problems in the working alliance.
Outcomes informed care
• “Meta-method” designed to improve outcomes across
all patients and diagnoses, regardless of treatment
method.
• Routine use of patient self report questionnaires to
track symptom severity and therapeutic alliance.
• Use of feedback mechanisms to alert clinicians to
patients at risk for poor outcomes.
• Performance feedback to clinicians, including
comparison to outcomes to those of clinicians treating
similar patients.
• Preferential referrals to highly effective clinicians.
2 case studies
• Resources for Living (RFL) provides telephonic
EAP services, data collected over the phone at time
of service; clinicians receive real time feed back on
trajectory of improvement and working alliance
(SIGNAL system)
• Accountable Behavioral Healthcare Alliance
(ABHA) is a managed behavioral healthcare
organization servicing Oregon Health Plan members
in 5 rural county area
Case history # 1: RFL
• Began using the 4 item Outcome Rating Scale and
Session Rating Scale in 2002
• Administered telephonically as part of telephonic
counseling sessions.
• Baseline data collected for 5 months
• Baseline data used to create trajectory of change
graphs
• Real time feedback provided to counselors via
SIGNAL System
qu
ar
te
3r
r2
d
00
qu
2
ar
te
4t
r2
h
00
qu
2
ar
t
er
1s
20
tq
02
ua
rte
2n
r2
d
00
qu
3
ar
t
er
3r
20
d
qu
03
ar
te
4t
r2
h
00
qu
3
ar
t
er
1s
20
tq
03
ua
rte
2n
r2
d
00
qu
4
ar
t
er
3r
20
d
qu
04
ar
te
r2
00
4
2n
d
Effect size
RFL Signal System: results
0.90
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
Training and feedback
Baseline period
Case history # 2: ABHA
• Began utilizing the 4 item Oregon Change Index
(OCI) in 2004.
• OCI administered at every session in outpatient
and day treatment settings.
• OCIs collected at over 80% of all sessions.
• Collected baseline data for 18 months, began
giving feedback in mid 2005.
• Updated Excel based Active Case Report contains
outcome data on all cases seen within the last 6
weeks is emailed to the clinicians weekly.
ABHA results
ABHA Effect Size by Quarter
Clients with scores in clinical range at intake
0.85
Benchmark from clinical trials: .83
0.75
Effect SIze
0.65
0.55
0.45
0.35
0.25
0.15
0.05
-0.05
1st 04
2nd 04
3rd 04
4th 04
1st 05
2nd 05
3rd 05
4th 05
1st 06
Implications for clinicians
• Good news: The clinician matters!!!!!!
• All treatments (including medications!?) are only as
effective as the clinicians delivering the treatment.
• Clinicians have an ethical responsibility to assess
and improve their personal effectiveness as
clinicians… they cannot rely on the treatments alone
to be curative.
• Effective clinicians deliver high value services and
are worth more money!!!
Implications for administrators
& policy makers
• Exclusive focus on the effectiveness of treatments
rather than the value of the clinicians limits the
potential to improve outcomes.
• Use of effective clinicians tends to lower costs.
• Administrators and policy makers have an obligation
to consumers to assure that they have access to
effective clinicians.
• Failure to monitor outcomes at the clinician level
places consumers at risk.
References
1.
2.
3.
4.
5.
Kirsch, I & Sapirstein, G. 1998. Listening to Prozac but hearing
placebo: A meta analysis of antidepressant medication. Prevention &
Treatment. 1, Article 0002a, No Pagination Specified
Kirsch, I. 2000. Are drug and placebo effects in depression additive?
Biological Psychiatry 47, 733-73.
Kirsch, I, Moore, TJ, Scoboria, A, Nicholls, SS. 2002. The emperor's
new drugs: An analysis of antidepressant medication data submitted
to the U.S. Food and Drug Administration. Prevention & Treatment.
5(1), No Pagination Specified
Rosenzweig S. 1936. Some implicit common factors in diverse
methods of psychotherapy: “At last the Dodo said, ‘Everybody has
won and all must have prizes.’” Am J Orthopsychiatry 6:412-5.
Shapiro DA & Shapiro D. 1982. Meta-analysis of comparative
therapy outcome studies: A replication and refinement. Psychol Bull
92:581-604.
References (continued)
6.
Robinson LA, Berman JS, Neimeyer RA. 1990. Psychotherapy for
treatment of depression: A comprehensive review of controlled
outcome research. Psychol Bull 108:30-49.
7. Wampold BE, Mondin GW, Moody M, et al. 1997. A meta-analysis
of outcome studies comparing bona fide psychotherapies:
Empirically, “All must have prizes.” Psychol Bull 122:203-15.
8. Ahn H, Wampold BE. 2001. Where oh where are the specific
ingredients? A meta-analysis of component studies in counseling and
psychotherapy. J Counsel Psychol 48:251-7.
9. Chambless DL, Ollendick TH. 2001. Empirically supported
psychological interventions: Controversies and evidence. Annual Rev
Psychol 52:685-716.
10. Luborsky, L., Rosenthal, R., Diguer, L., et al. 2002. The dodo bird
verdict is alive and well--mostly. J. Psychotherapy Integration Vol
12(1) 32-57
References (continued)
11. Wampold BE. 2001. The great psychotherapy debate: Models,
Methods, and Findings. Mahwah NJ: Lawrence Erlbaum Associates.
Wampold BE, Mondin GW, Moody M, et al. 1997. A meta-analysis
of outcome studies comparing bona fide psychotherapies:
Empirically, “All must have prizes.” Psychol Bull 122:203-15.
12. McCall, WA. 1923 How to experiment in education. New York:
McmIllan.
13. Fisher, RA. 1935 The design of experiments. Edinburgh: Oliver and
Boyd.
14. Gehan, E. & Lemark, NA. 1994. Statistics in medical research:
Developments in clinical trials. New York: Plenum Press.
15. Martindale C. 1978. The therapist-as-fixed-effect fallacy in
psychotherapy research. J Consult Clin Psychol 46:1526-30.
References (continued)
16. Luborsky L, Crits-Christoph P, McLellan T, et al. 1986. Do therapists
vary much in their success? Findings from four outcome studies. Am
J Orthopsychiatry 56:501-12.
17. Crits-Christoph P, Baranackie K, Kurcias JS, et al. 1991. Metaanalysis of therapist effects in psychotherapy outcome studies.
Psychother Res 1:81-91.
18. Crits-Christoph P, Mintz J. 1991. Implications of therapist effects for
the design and analysis of comparative studies of psychotherapies. J
Consul Clin Psychol 59:20-6.
19. Wampold BE. 1997. Methodological problems in identifying
efficacious psychotherapies. Psychother Res 7:21-43,
20. Elkin I. 1999. A major dilemma in psychotherapy outcome research:
Disentangling therapists from therapies. Clin Psychol Sci Prac 6:1032.
References (continued)
21. Wampold BE, Serlin RC. 2000. The consequences of ignoring a
nested factor on measures of effect size in analysis of variance
designs. Psychol Methods 4:425-33.
22. Huppert JD, Bufka LF, Barlow DH, et al. 2001. Therapists, therapist
variables, and cognitive-behavioral therapy outcomes in a
multicenter trial for panic disorder. J Consul Clin Psychol 69:747-55.
23. Okiishi J, Lambert MJ, Nielsen SL, et al. 2003. Waiting for
supershrink: An empirical analysis of therapist effects. Clin Psychol
Psychother 10:361-73.
24. Brown GS, Jones ER, Lambert MJ, et al. 2005. Identifying highly
effective psychotherapists in a managed care environment. Am J
Managed Care 11(8):513-20.
25. Wampold BE, Brown GS. 2005. Estimating variability in outcomes
due to the therapist: A naturalistic study of outcomes in managed
care. J Consul Clin Psychol. 73(5): 914-923.
References (continued)
26. Elkin, I, Shae, T, Watkins, JT., et al. 1989. National Institute of Mental
Health Treatment of Depression Collaborative Research Program:
General effectiveness of treatments. Archive of General Psychiatry. 46:
971-982.
27. Kim DM, Wampold BE, Bolt DM. 2006. Therapist effects and
treatment effects in psychotherapy: Analysis of the National Institute of
Mental Health Treatment of Depression Collaborative Research
Program. Psychother Res. 16(2): 161-172.
28. McKay, KM, Imel, ZE & Wampold, BE. In press. Psychiatrist effects
in the pharmacological treatment of depression. J. Affective Disorders.
29. Hannan C, Lambert MJ, Harmon C et al. 2005. A lab test and
algorithms for identifying clients at risk for treatment failure. J Clin
Psychol 61(2):155-63.
30. Lambert MJ, Harmon C, Slade K et al. 2005. Providing feedback to
psychotherapists on their patients progress: Clinical results and practice
suggestions J Clin Psychol 61(2):165-74.
References (continued)
31. Harmon C, Hawkins, Lambert MJ et al. 2005. Improving outcomes for
poorly responding clients: The use of clinical support tools and
feedback to clients. J Clin Psychol 61(2):175-85.
32. Brown GS, Jones DR. 2005. Implementation of a feedback system in a
managed care environment: What are patients teaching us? J Clin
Psychol 61(2):187-98.
33. Claiborn CD, Goodyear EK. 2005. Feedback in psychotherapy. J Clin
Psychol 61(2):209-21.
34. Lueger RJ. 1998. Using feedback on patient progress to predict the
outcome of psychotherapy. J Clin Psychol 54:383-93.
35. Lambert MJ, Whipple JL, Smart DW, et al. 2001. The effects of
providing therapists with feedback on patient progress during
psychotherapy: Are outcomes enhanced? Psychother Res 11(1):49-68.
References (continued)
36. Lambert MJ, Whipple JL, Vermeersch DA, et al. 2002. Enhancing
psychotherapy outcomes via providing feedback on client progress: A
replication. Clin Psychol Psychother 9:91-103.
37. Whipple JL, Lambert MJ, Vermeersch DA, et al. 2003. Improving the
effects of psychotherapy: The use of early identification of treatment
failure and problem-solving strategies in routine practice. J Counsel
Psychol 50(1):59-68.
38. Lambert MJ, Whipple JL, Hawkins EJ, et al. 2003. Is it time for
clinicians to routinely track patient outcome? A meta-analysis. Clin
Psychol Sci Prac 10:288-301.
39. Bachelor, A., & Horvath, A. (1999). The therapeutic relationship. In
M.A. Hubble, B.L. Duncan, and S.D. Miller (eds.). The Heart and Soul
of Change: What Works in Therapy. Washington, D.C.: APA Press,
133-178.
40. Blatt, S. J., Zuroff, D.C., Quinlan, D.M., & Pilkonis, P. (1996).
Interpersonal factors in brief treatment of depression: Further analyses
of the NIMH Treatment of Depression Collaborative Research Program.
J Consul Clin Psychol. 64, 162-171.
References (continued)
41. Bordin, E. S. (1979). The generalizability of the psychoanalytic
concept of the working alliance. Psychotherapy: Theory, Research, and
Practice, 16, 252-260.
42. Burns, D., & Nolen-Hoeksema, S. (1992). Therapeutic empathy and
recovery from depression in cognitive-behavioral therapy: A structural
equation model. J Consul Clin Psychol. 60, 441-449.
43. Connors, GJ, DiClemente, CC., Carroll, KM, et al. 1997 The
therapeutic alliance and its relationship to alcoholism treatment
participation and outcome. J Consul Clin Psychol, 65(4), 588-598.
44. Horvath, A. O., & Symonds, B. D. (1991). Relation between working
alliance and outcome in psychotherapy: A meta-analysis. J Consul
Clin Psychol. 38, 139-149.
45. Krupnick, J., Sotsky, SM, Simmens, S et al. 1996. The role of the
therapeutic alliance in psychotherapy and pharmacotherapy outcome:
Findings in the National Institute of Mental Health Treatment of
Depression Collaborative Research Project. J Consul Clin Psychol. , 64,
532-539.
About the presenter
G.S. (Jeb) Brown is a licensed psychologist with a Ph.D. from Duke
University. He served as the Executive Director of the Center for Family
Development from 1982 to 19987. He then joined United Behavioral
Systems (an United Health Care subsidiary) as the Executive Director for
of Utah, a position he held for almost six years. In 1993 he accepted a
position as the Corporate Clinical Director for Human Affairs
International (HAI), at that time one of the largest managed behavioral
healthcare companies in the country.
In 1998 he left HAI to found the Center for Clinical Informatics, a
consulting firm specializing in helping large organizations implement
outcomes management systems. Client organizations include PacifiCare
Behavioral Health/ United Behavioral Health, Department of Mental
Health for the District of Columbia, Accountable Behavioral Health Care
Alliance, Resources for Living and assorted treatment programs and
centers throughout the world.
Dr. Brown continues to work as a part time psychotherapist at
behavioral health clinic in Salt Lake City, Utah. He does measure his
outcomes.
http://www.clinical-informatics.com
[email protected]
1821 Meadowmoor Rd.
Salt Lake City, UT 84117
Voice 801-541-9720