Transcript Honors for Outcomes - A Collaborative Outcomes Resource Network

The importance of therapist
effects in the treatment of
children and adolescents
G.S. (Jeb) Brown, Ph.D.
Center for Clinical Informatics
Overview
• Findings based on analyses of data collected
through the PacifiCare Behavioral
Health/United Behavioral Health ALERT
clinical information system
• Novel methodology demonstrates that much of
the variance in outcomes resides with the
individual psychotherapist.
Therapist effects
• Therapist differ in their “effectiveness”
resulting in wide differences in outcomes.
 Results cannot be explained by theoretical orientation,
treatment methods, years of training or experience.
• The effectiveness of all treatments, including
medications, are mediated by therapist
effects.
• Failure to measure and account for therapist
effects is simply BAD SCIENCE!
Outcome Measures
•
30-item patient self-report questionnaires

Life Status Questionnaire (adults)

Youth Life Status Questionnaire (children)
• Administered at 1st, 3rd, 5th sessions and every
5th session there after
• Over 20,000 questionnaires per month come
into the system
Case mix
• Case mix variables are those variables
present at the beginning of the treatment
episode that are predictive of the outcome
• Intake score accounts for 18% of variance in
change scores in PBH data
• Addition of age, sex and diagnosis to
predictive model accounts for < 2%
additional variance
Where’s the variance?
% of variance in change scores
Case mix adjustment
Unexplained 80%
(external stressors,
health condition,
social supports, etc.)
Acuity (intake score)
18%
Diagnosis, age & sex
2%
Where else is the variance?
Honors for Outcomes
• 1046 Provider Locations on Honor Roll
• 56 Group Practices
• 740 Individuals (not including those in
groups)
• These providers accounted for about 30% of
PBH outpatient episodes in 2004
Honor Rating Method
• Honors for Outcomes Selection Criteria:
– Minimum of 10 cases with two Y/LSQ data points in past
3 years
– Average patient change must be reliably above average:
65% confidence that the provider’s Change Index >0
– Change Index is a case-mix adjusted measure, compares
outcomes to PBH’s large normative database
• Honors for Outcomes is updated quarterly
Website
Honors for Outcomes - Search
Honors for Outcomes - Results
Study Question 1
• Honors for Outcomes depends on predictive
validity of Honors rating; prior performance
predicts future performance
• Question: Does a therapist’s outcomes with
adults predict outcomes with children and
adolescents?
• Implication if yes: Therapists’ effectiveness is
likely to be global in nature rather than
specific to age and or diagnostic group.
Study Question 2
• Question: Does a therapist’s outcomes with
adults predict outcomes with children and
adolescents on medications?
• Implication if yes: The therapist effectiveness
of the therapists is apparently mediating the
effect of the medication(s).
Study Method
• Use Honors for Outcomes methodology to rank
clinicians based on their outcomes with adult
patients only.
• Therapist included in the study if they treated at
least one child/adolescent with psychotherapy
only and one with psychotherapy plus
medication. (929 Honors, 1352 Non-Honors)
• Compare outcomes for children and adolescents
for Honors clinicians to other clinicians.
Result: Outcomes for adults predicts
outcomes for children
Honors-psychotherapy
only
1.2
1
Honors-psychotherapy
and medication
Effect size
0.8
Non-Honorspsychotherapy only
0.6
0.4
Non-Honorspsychotherapy and
medication
0.2
0
-0.2
0-41
42-120
mild symptoms
Intake scores
moderate to severe symptoms
Residual effect size
Results after adjusting for intake
score, age, sex, diagnosis and prior
treatment history.
0.25
0.2
0.15
0.1
0.05
0
-0.05
-0.1
-0.15
-0.2
-0.25
Honorspsychotherapy
only
Honorspsychotherapy
and medication
0-41
mild symptoms
42-120
moderate to severe symptoms
Non-Honorspsychotherapy
only
Non-Honorspsychotherapy
and medication
All diagnoses and medications
Intake score below mean
Intake score at mean or above
Delta
residual
N
Delta
residual
N
Honors-psychotherapy
only
1.9
2.3
430
12.5
2.3
286
Honors-psychotherapy
and medication
-1.3
-1.3
79
15.3
2.7
134
Non-Honorspsychotherapy only
-0.9
-0.5
565
8.3
-2.2
449
Non-Honorspsychotherapy and
medication
-1.2
-2.7
102
10.3
-1.5
186
Children diagnosed with depression and
treated with psychotherapy alone or in
combination with an antidepressant
0.4
Residual effect size
0.3
0.2
0.1
0
-0.1
-0.2
-0.3
0-41
42-120
Honorspsychotherapy
only
Honorspsychotherapy
and medication
Non-Honorspsychotherapy
only
Non-Honorspsychotherapy
and medication
Depression & antidepressants
Intake score below mean
Intake score at mean or above
Delta
residual
N
Delta
residual
N
Honors-psychotherapy
only
2.6
3.5
77
15.4
4.6
84
Honors-psychotherapy
and medication
0.11
0.3
28
15.5
2.9
41
Non-Honorspsychotherapy only
-1.7
-2.9
87
9.2
-2
123
Non-Honorspsychotherapy and
medication
-1.7
-3.2
27
11.1
-0.9
53
Implications for clinicians
• The effectiveness of the individual clinician is
the major source of variance in outcomes.
• All treatments (including medications) are
only as effective and the clinicians delivering
the treatment.
• Clinicians have an ethical responsibility to
assess and improve their personal
effectiveness as clinicians… they cannot rely
on the treatments alone to be curative.
Implications for administrators
& policy makers
• Exclusive focus on the effectiveness of
treatments rather than clinicians limits the
potential to improve outcomes.
• Administrators and policy makers have an
obligation to consumers to assure that they
have access to effective clinicians.
• Failure to monitor outcomes at the clinician
level places consumers at risk.
http://www.clinical-informatics.com
[email protected]
1821 Meadowmoor Rd.
Salt Lake City, UT 84117
Voice 801-541-9720
Suggested readings
Ahn H, Wampold BE. Where oh where are the specific ingredients? A meta-analysis of component studies in
counseling and psychotherapy. Journal of Counseling Psychology; 2001: 48, 251-257.
Blatt SJ, Sanislow CA, Zuroff DC, Pilkonis PA. Characteristics of effective therapists: Further analyses of data
from the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Journal of
Consulting and Clinical Psychology; 1996: 64, 1276-1284.
Brown GS, Burlingame GM, Lambert MJ, et al. Pushing the quality envelope: A new outcomes management
system. Psychiatric Services; 2001: 52 (7), 925-934.
Brown GS, Herman R, Jones ER, Wu J. Improving substance abuse assessments in a managed care
environment. Joint Commission Journal on Quality and Safety; 2004: 30(8), 448-454.
Brown GS, Jones ER, Betts W, Wu J. Improving suicide risk assessment in a managed-mare environment.
Crisis; 2003: 24(2), 49-55.
Brown GS, Jones ER, Lambert MJ, Minami T. Identifying highly effective psychotherapists in a managed care
environment. American Journal of Managed Care, 2005:11(8):513-20 .
Brown GS, Jones ER. Implementation of a feedback system in a managed care environment: What are patients
teaching us? Clinical Psychology/In Session: 2005: 61(2), 187-198.
Burlingame GM, Jasper BW, Peterson G, et al. Administration and Scoring Manual for the YLSQ. Wilmington,
DL, American Professional Credentialing Services; 2001.
Crits-Christoph P, Mintz J. Implications of therapist effects for the design and analysis of comparative studies of
psychotherapies. Journal of Consulting and Clinical Psychology, 1991; 59, 20-26.
Crits-Christoph P., Baranackie K., Kurcias JS et al. Meta-analysis of therapist effects in psychotherapy outcome
studies. Psychotherapy Research; 1991: 1, 81-91.
Elkin I. A major dilemma in psychotherapy outcome research: Disentangling therapists from therapies. Clinical
Psychology: Science and Practice; 1999: 6, 10-32.
Honors for Outcomes
• 1046 Provider Locations on Honor Roll
• 56 Group Practices
• 740 Individuals (not including those in
groups)
• These providers accounted for about 30% of
PBH outpatient episodes in 2004
Honor Rating Method
• Honors for Outcomes Selection Criteria:
– Minimum of 10 cases with two Y/LSQ data points in past
3 years
– Average patient change must be reliably above average:
65% confidence that the provider’s Change Index >0
– Change Index is a case-mix adjusted measure, compares
outcomes to PBH’s large normative database
• Honors for Outcomes is updated quarterly
Website
Honors for Outcomes - Search
Honors for Outcomes - Results
Study Question 1
• Honors for Outcomes depends on predictive
validity of Honors rating; prior performance
predicts future performance
• Question: Does a therapist’s outcomes with
adults predict outcomes with children and
adolescents?
• Implication if yes: Therapists’ effectiveness is
likely to be global in nature rather than
specific to age and or diagnostic group.
Study Question 2
• Question: Does a therapist’s outcomes with
adults predict outcomes with children and
adolescents on medications?
• Implication if yes: The therapist effectiveness
of the therapists is apparently mediating the
effect of the medication(s).
Study Method
• Use Honors for Outcomes methodology to rank
clinicians based on their outcomes with adult
patients only.
• Therapist included in the study if they treated at
least one child/adolescent with psychotherapy
only and one with psychotherapy plus
medication. (929 Honors, 1352 Non-Honors)
• Compare outcomes for children and adolescents
for Honors clinicians to other clinicians.
Result: Outcomes for adults predicts
outcomes for children
Honors-psychotherapy
only
1.2
1
Honors-psychotherapy
and medication
Effect size
0.8
Non-Honorspsychotherapy only
0.6
0.4
Non-Honorspsychotherapy and
medication
0.2
0
-0.2
0-41
42-120
mild symptoms
Intake scores
moderate to severe symptoms
Residual effect size
Results after adjusting for intake
score, age, sex, diagnosis and prior
treatment history.
0.25
0.2
0.15
0.1
0.05
0
-0.05
-0.1
-0.15
-0.2
-0.25
Honorspsychotherapy
only
Honorspsychotherapy
and medication
0-41
mild symptoms
42-120
moderate to severe symptoms
Non-Honorspsychotherapy
only
Non-Honorspsychotherapy
and medication
All diagnoses and medications
Intake score below mean
Intake score at mean or above
Delta
residual
N
Delta
residual
N
Honors-psychotherapy
only
1.9
2.3
430
12.5
2.3
286
Honors-psychotherapy
and medication
-1.3
-1.3
79
15.3
2.7
134
Non-Honorspsychotherapy only
-0.9
-0.5
565
8.3
-2.2
449
Non-Honorspsychotherapy and
medication
-1.2
-2.7
102
10.3
-1.5
186
Children diagnosed with depression and
treated with psychotherapy alone or in
combination with an antidepressant
0.4
Residual effect size
0.3
0.2
0.1
0
-0.1
-0.2
-0.3
0-41
42-120
Honorspsychotherapy
only
Honorspsychotherapy
and medication
Non-Honorspsychotherapy
only
Non-Honorspsychotherapy
and medication
Depression & antidepressants
Intake score below mean
Intake score at mean or above
Delta
residual
N
Delta
residual
N
Honors-psychotherapy
only
2.6
3.5
77
15.4
4.6
84
Honors-psychotherapy
and medication
0.11
0.3
28
15.5
2.9
41
Non-Honorspsychotherapy only
-1.7
-2.9
87
9.2
-2
123
Non-Honorspsychotherapy and
medication
-1.7
-3.2
27
11.1
-0.9
53
Implications for clinicians
• The effectiveness of the individual clinician is
the major source of variance in outcomes.
• All treatments (including medications) are
only as effective and the clinicians delivering
the treatment.
• Clinicians have an ethical responsibility to
assess and improve their personal
effectiveness as clinicians… they cannot rely
on the treatments alone to be curative.
Implications for administrators
& policy makers
• Exclusive focus on the effectiveness of
treatments rather than clinicians limits the
potential to improve outcomes.
• Administrators and policy makers have an
obligation to consumers to assure that they
have access to effective clinicians.
• Failure to monitor outcomes at the clinician
level places consumers at risk.
http://www.clinical-informatics.com
[email protected]
1821 Meadowmoor Rd.
Salt Lake City, UT 84117
Voice 801-541-9720
Suggested readings
Ahn H, Wampold BE. Where oh where are the specific ingredients? A meta-analysis of component studies in
counseling and psychotherapy. Journal of Counseling Psychology; 2001: 48, 251-257.
Blatt SJ, Sanislow CA, Zuroff DC, Pilkonis PA. Characteristics of effective therapists: Further analyses of data
from the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Journal of
Consulting and Clinical Psychology; 1996: 64, 1276-1284.
Brown GS, Burlingame GM, Lambert MJ, et al. Pushing the quality envelope: A new outcomes management
system. Psychiatric Services; 2001: 52 (7), 925-934.
Brown GS, Herman R, Jones ER, Wu J. Improving substance abuse assessments in a managed care
environment. Joint Commission Journal on Quality and Safety; 2004: 30(8), 448-454.
Brown GS, Jones ER, Betts W, Wu J. Improving suicide risk assessment in a managed-mare environment.
Crisis; 2003: 24(2), 49-55.
Brown GS, Jones ER, Lambert MJ, Minami T. Identifying highly effective psychotherapists in a managed care
environment. American Journal of Managed Care, 2005:11(8):513-20 .
Brown GS, Jones ER. Implementation of a feedback system in a managed care environment: What are patients
teaching us? Clinical Psychology/In Session: 2005: 61(2), 187-198.
Burlingame GM, Jasper BW, Peterson G, et al. Administration and Scoring Manual for the YLSQ. Wilmington,
DL, American Professional Credentialing Services; 2001.
Crits-Christoph P, Mintz J. Implications of therapist effects for the design and analysis of comparative studies of
psychotherapies. Journal of Consulting and Clinical Psychology, 1991; 59, 20-26.
Crits-Christoph P., Baranackie K., Kurcias JS et al. Meta-analysis of therapist effects in psychotherapy outcome
studies. Psychotherapy Research; 1991: 1, 81-91.
Elkin I. A major dilemma in psychotherapy outcome research: Disentangling therapists from therapies. Clinical
Psychology: Science and Practice; 1999: 6, 10-32.
Suggested readings (continued)
Hannan C, Lambert MJ, Harmon C, Nielsen SL, Smart DW & Shimokawa K, Sutton SW. A lab test and
algorithms for identifying clients at risk for treatment failure. Journal of Clinical Psychology/In Session: 2005:
61(2), 155-164.
Harmon C, Hawkins, EJ, Lambert, MJ, Slade K & Whipple JL. Improving outcomes for poorly responding
clients: the use of clinical support tools and feedback to clients. 61(2), 175-186.
Huppert JD, Bufka LF, Barlow DH, Gorman JM, Shear MK, Woods SW. Therapists, therapist variables, and
cognitive-behavioral therapy outcomes in a multicenter trial for panic disorder. Journal of Consulting and Clinical
Psychology; 2001: 69, 747-755.
Kim DM, Wampold BE, Bolt DM. Therapist effects and treatment effects in psychotherapy: Analysis of the
National Institute of Mental Health Treatment of Depression Collaborative Research Program. Psychotherapy
Research: 2006: 12(2), 161-172.
Lambert MJ., Whipple J., Smart, DW et al (Vermeersch, D. A., Nielsen, S.L., & Hawkins, E. J.) The effects of
providing therapists with feedback on patient progress during psychotherapy: Are outcomes enhanced?
Psychotherapy Research; 2001: 11, 49-68.
Lambert MJ, Harmon C, Slade K, Whipple JL & Hawkins EL. Providing feedback to psychotherapists on
their patients’ progress: clinical results and practice suggestions. Journal of Clinical Psychology/In Session:
2005: 61(2), 165-174.
Lambert MJ, Hatfield DR, Vermeersch DA., et al. Administration and scoring manual for the LSQ (Life Status
Questionnaire). East Setauket, NY: American Professional Credentialing Services; 2001.
Lambert MJ, Whipple JL, Hawkins EJ et al. Is it time for clinicians to routinely track patient outcome? A
meta-analysis. Clinical Psychology: Science & Practice; 2003: 10, 288-301.
Lambert MJ. Emerging methods for providing clinicians with timely feedback on treatment effectiveness.
Journal of Clinical Psychology/In Session: 2005: 61(2), 141-144.
Suggested readings (continued)
Luborsky L, Crits-Christoph P, McLellan T et al. Do therapists vary much in their success? Findings from
four outcome studies. American Journal of Orthopsychiatry; 1986: 56, 501-512.
Luborsky L, Rosenthal R, Diguer L et al. The dodo bird verdict is alive and well--mostly. Clinical
Psychology: Science & Practice; 2002: 9(1) 2-12.
Matsumoto K, Jones E, Brown, GS. Using clinical informatics to improve outcomes: A new approach to
managing behavioral healthcare. Journal of Information Technology in Health Care; 2003: 1(2), 135-150
Okiishi J, Lambert MJ, Nielsen SL, Ogles BM. Waiting for supershrink: An empirical analysis of therapist
effects. Clinical Psychology and Psychotherapy; 2003: 10, 361-373.
Porter ME & Teisberg, EO. Redefining competition in health care. Harvard Business Review,2004:65-76.
Shapiro DA, Shapiro, D. Meta-analysis of comparative therapy outcome studies: A replication and
refinement. Journal of consulting and Clinical Psychology; 1982: 92, 581–604.
Vermeersch DA, Lambert MJ, Burlingame GM. Outcome Questionnaire: Item sensitivity to change. Journal
of Personality Assessment; 2002: 74, 242-261.
Wampold BE, Brown GS. Estimating therapist variability: A naturalistic study of outcomes in private practice.
Journal of Consulting and Clinical Psychology; 2005: 75(5) pp 914-923.
Wampold BE, Mondin GW, Moody M et al. A meta-analysis of outcome studies comparing bona fide
psychotherapies: Empirically, “all must have prizes.” Psychological Bulletin; 1997: 122, 203-2154.
About the presenter
G.S. (Jeb) Brown is a licensed psychologist with a Ph.D. from Duke
University. He served as the Executive Director of the Center for Family
Development from 1982 to 19987. He then joined United Behavioral
Systems (an United Health Care subsidiary) as the Executive Director for
of Utah, a position he held for almost six years. In 1993 he accepted a
position as the Corporate Clinical Director for Human Affairs
International (HAI), at that time one of the largest managed behavioral
healthcare companies in the country.
In 1998 he left HAI to found the Center for Clinical Informatics, a
consulting firm specializing in helping large organizations implement
outcomes management systems. Client organizations include PacifiCare
Behavioral Health/ United Behavioral Health, Department of Mental
Health for the District of Columbia, Accountable Behavioral Health Care
Alliance, Resources for Living and assorted treatment programs and
centers throughout the world.
Dr. Brown continues to work as a part time psychotherapist at
behavioral health clinic in Salt Lake City, Utah. He does measure his
outcomes.