Transcript Oregon Change Index Outcomes Management System

The Oregon Change Index
Outcomes Management System:
Development, Implementation, and Results
Seth Bernstein, Ph.D.
ABHA Executive Director
Jeb Brown, Ph.D
Center for Clinical Informatics
Outcomes Informed Care
• Use of valid and reliable clients self report
outcomes questionnaires.
• Frequent measurement for all clients.
• Frequent feedback to clinicians on
outcomes for all clients as treatment
unfolds.
• Use of therapeutic alliance measure
encouraged.
Outcome Measurement
• An outcome questionnaire provides a
structured and consistent method that
invites the patient to communicate how they
are feeling
• Repeated administrations during treatment
lets the patient communicate how they are
changing over time
Measurement and Judgment
• Clinicians are trained to use clinical judgment to
assess symptoms and level of patient distress.
• No outcome measure can replace clinical
judgment, but it can inform judgment.
• An outcome measure is a separate channel of
communication, and the information obtained may
be different than from clinician assessment based
on face to face interviewing
Measuring Distress
• All outcome measures appear to primarily
measure “global distress”.
• Subscales for symptoms, role functioning and
interpersonal relationships are all highly correlated
with core “global distress factor”
• Reliability: consistency with which test measures
factor; .90 or higher desirable
• Validity: all outcome measures show high degree
of correlation with one another
Picking An Outcome Measurement Tool
• Most assessment measures were . . .
– Developed for research – not the real world of practice
•
•
•
•
Diagnostically based rather than treatment-oriented
Instrumentation is lengthy and burdensome
Not designed for concurrent measurement
Often not linked to improved outcomes or enhanced service
quality
– Designed in piecemeal fashion – not integrated or
comprehensive
– Provided for separate stakeholders – not integrated for
system, clinical and consumer levels
Picking An Outcome Measurement Tool
What we needed was a tool that provided:
• Sensitive, accurate, real time estimates of
meaningful clinical change over time
• Sensitive and informative characterization of
process indicators (therapeutic alliance, treatment
modality, readiness to change, etc.)
• Sensitive, informative, real time feedback to
consumers (youth and families) and clinicians
Oregon Change Index (OCI) 1.0
Low
High
How are you feeling about yourself?
O O O O O O O O O O
How are your relationships with family and friends?
O O O O O O O O O O
How are you functioning at work or school or other activities?
O O O O O O O O O O
How are you feeling overall?
O O O O O O O O O O
How is your progress with treatment plan goal #1?
O O O O O O O O O O
How is your progress with treatment plan goal #2?
O O O O O O O O O O
How are you feeling about your relationship with your therapist/case
manager?
O O O O O O O O O O
[1]The first four questions of the Oregon Change Index, ABHA ©2002 are derived from the ORS, developed
by Scott D. Miller and Barry L. Duncan ©2000 and are used with their permission .
Oregon Change Index 2.0
How are your relationships with family?
How are your relationships with friends?
How are you functioning at work or school or other activities?
How are you feeling overall?
How is your progress with treatment plan goal #1?
How is your progress with treatment plan goal #2?
Is the therapy being helpful to you?
[1]The first four questions of the Oregon Change Index, ABHA ©2002 are derived from the ORS, developed
by Scott D. Miller and Barry L. Duncan ©2000 and are used with their permission.
The Youth Oregon Change Index
(Y-OCI) 2.0
How are your child’s relationships with family?
How are your child’s relationships with friends?
How is your child functioning at school?
How is your child functioning at play?
How is your child’s progress with treatment plan goal #1?
How is your child’s progress with treatment plan goal #2?
Is the therapy being helpful to your child?
[1]The first four questions of the Oregon Change Index, ABHA ©2002 are derived from the ORS, developed
by Scott D. Miller and Barry L. Duncan ©2000 and are used with their permission .
Outcomes Management System
• Provides means to capture outcome data
and provide feedback to clinicians.
• Forms mailed weekly for data entry.
• Monitoring and feedback on OCI
completion rates.
• Weekly Active Case Reports emailed to
clinicians.
Active Case Report
Clinician: 35
County: Deschutes
Mean intake score:
18.4
Mean recent score:
23.7
Mean change:
5.3
Mean Benchmark:
2.9
Case Count
91
Sort by:
To view the change graph for a specific client, use the mouse
to click on the row number beside the client you wish to view
and then click on "View Client Graph":
Client ID
32214
319246
319995
320958
173251
322554
315492
322854
197502
30147
312845
319154
320176
128982
321771
309933
301474
151012
317263
174131
Age
Group
adult
adult
adult
adult
adult
adult
adult
adult
adult
adult
adult
adult
adult
adult
adult
adult
adult
adult
adult
adult
Clinician at Most recent
Intake date
intake
clinician
35
35
8/15/2005
35
35
6/21/2004
35
35
7/12/2005
24
35
3/2/2005
35
35
9/12/2005
35
35
11/9/2005
35
35
1/27/2005
35
35
12/13/2005
35
35
1/5/2005
35
35
7/15/2005
33
35
4/20/2004
31
35
5/11/2004
35
35
12/22/2004
33
35
8/9/2005
35
35
5/18/2005
35
35
1/12/2004
35
35
7/7/2004
35
35
1/18/2006
35
35
4/20/2004
35
35
2/23/2005
Intake
OCI
35.0
28.0
13.0
13.0
18.0
7.0
24.0
9.0
15.0
14.0
14.0
23.0
8.0
17.0
11.0
16.0
19.0
9.0
14.0
21.3
Clinician ID
Intake OCI
Intake date
Benchmark Score
Most recent OCI
Most recent date
OCI
Change
Score
-33.0
-24.0
-7.0
-5.0
-7.0
-1.0
-8.0
-1.0
-2.0
-1.0
0.0
-4.0
3.0
-1.0
2.0
0.0
-1.0
4.0
2.0
-1.3
Status
Significantly worse
Significantly worse
Somewhat worse
Somewhat worse
Somewhat worse
Somewhat worse
Somewhat worse
Somewhat worse
Somewhat worse
Somewhat worse
No change
Somewhat worse
Somewhat improved
Somewhat worse
Somewhat improved
No change
Somewhat worse
Somewhat improved
Somewhat improved
Somewhat worse
View Client Graph
Most recent
date
9/7/2005
2/24/2005
1/19/2006
6/27/2005
1/9/2006
12/7/2005
10/25/2005
1/24/2006
4/13/2005
9/20/2005
1/24/2006
1/24/2006
2/8/2005
1/31/2006
1/26/2006
1/12/2006
1/17/2006
2/1/2006
5/16/2005
3/9/2005
Most
recent
OCI
2.0
4.0
6.0
8.0
11.0
6.0
16.0
8.0
13.0
13.0
14.0
19.0
11.0
16.0
13.0
16.0
18.0
13.0
16.0
20.0
OCI
Count
2
10
7
19
3
2
5
3
6
2
26
25
5
10
10
35
18
2
5
2
Benchmark
score
-27.8
-22.0
-12.0
-10.0
-9.6
-8.7
-7.9
-7.8
-6.0
-5.5
-4.5
-4.3
-4.3
-4.1
-3.9
-3.6
-3.2
-2.8
-2.5
-2.4
Trajectory of Change Graph
Client ID:
First Date:
First Score
Most Recent Date:
Most Recent Score:
Total OCIs:
320958
3/2/2005
13
6/27/2005
8
19
First clinician:
Most recent clinician:
Pre-post change score:
Status:
Benchmark Score:
24
35
-5.0
Somewhat worse
-10.0
Return to Active Case Report
The first 20 sessions are graphed.
If there are more than 20 sessions, the final point is the most recent session
40
Client Scores
35
30
Clinical Cutoff
25
20
6/27/2005
6/20/2005
6/8/2005
6/6/2005
6/1/2005
5/25/2005
5/17/2005
5/16/2005
5/10/2005
5/9/2005
5/2/2005
4/25/2005
4/20/2005
4/18/2005
4/14/2005
0
4/11/2005
10th percentile
5
4/7/2005
25th percentile
10
4/4/2005
Expected
Change
15
3/2/2005
75th percentile
Annual Outcomes Report
Clinician:
35
Report period: 01/01/2005 to 12/31/2005
This report includes all patients that began treatment
within the date ranges specified.
The change score on the OCI measures treatment
outcome. The change scores are calculated by subtracting
the last score in the treatment episode from the intake score.
Of course, change scores are only calculated for those with
more than one data point. Positive change scores indicate
improvement. The report provides results for clients that
started treatment during the current reporting period (Current
Report) as well as for all clients in the ABHA database
(ABHA norms)
The Benchmark Score tells you how your patients
compare to the ABHA norms and is calculated by
subtracting the change score for the ABHA sample from
the change score for your patients. The ABHA norms are
calculated after adjusting for differences in case mix, so
that the norms reflect change for cases from the same
age group (youth or adults) and intake scores to the cases
in the current report. A positive Benchmark Score
indicates that your patients averaged more improvement
than the comparable patients in the ABHA system.
Outcomes for All Cases
Total number of cases: 70
Number of cases with > one data point: 42
% of cases with > one data point: 60%
Patients with intake scores in the top quartile report a level
of symptoms more characteristic of the general population
than of individuals seeking mental health services. These
patients typically report a high level of satisfaction with brief
psychotherapy. However, on average they do not report
improvement on the outcome measures. The scores actually
tend to increase with time.
Severity at intake
Adults
Top quartile - minimal distress
Mild to moderate distress
Bottom quartile: severe distress
Combined adult
Total
Cases
29
28
13
70
% of
report
sample
41%
40%
19%
100%
Change scores
Current
ABHA
Report
Norms
3.37
0.88
Benchmark
Score
2.50
Patients with intake scores in the middle two quartiles tend to
improve with treatment. Many of these patients do well with
psychotherapy alone. While the length of treatment varies, must
respond within three to nine weeks.
Patients in the bottom quartile do best with a combination of
psychotherapy and medications and may not realize fill benefit
until 12 or more weeks of treatment.
# cases
% cases
with > 1
with > 1
data point data point
18
62%
16
57%
8
62%
42
60%
Change Scores
Your
ABHA
Patients
Norms
-1.8
-3.6
4.4
2.6
13.0
7.4
3.4
0.9
Benchmark
Score
1.8
1.8
5.6
2.5
Outcomes trending upwards
Effect Size
ABHA Outcomes by Year
Clients with scores in the clinical range at intake
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
2004
2005
2006
Outcomes are good, but….
• Aggregated data shows that the average patient
improves during treatment, but…
– There is wide variation in outcomes from one patient to
another.
• Knowledge of the “average outcome” doesn’t help us
improve outcomes.
• The interesting information is in the differences in
outcomes…
– What practices or treatment methods are associated with better outcomes?
– Which patients do we do well with? Where do we need to improve?
– Are services allocated effectively so that those with the most severe
symptoms are receiving treatment of sufficient frequency and duration?
Challenges Ahead
• Clinicians and managers remain unsure about the
meaning/validity of the data
– Solution: Help clinicians and managers “own” the data,
identify threats to the reliability and validity of the data,
and recommend solutions.
• Clinician/manager anxiety due to fear of
measurement
– Solution: Continuously reinforce message that
outcomes are important but the data is not used to
“punish” anyone. Foster on organizations culture of
continuous quality improvement.
Challenges Ahead
• Uncertainty about use of OCI with various sub
populations of patients.
– Solution: Use “study code” field on OCI from to
identify sub populations for further study.
• Clinicians uncertain about the value of feedback
– Solution: Improve data entry processes so that Active
Case Report consistently contains data from sessions
within the past week.
• Mandates for “Evidence based practices”
– Solution: Provide evidence that outcomes informed care
can lead to improved outcomes through use of
“evidence based meta-methods’.