Transcript Measuring and Improving Fidelity to Evidence-Based Practices

Measuring and Improving
Fidelity to Evidence-Based
Practices
Our Obligation to Effective Service
Delivery and Supervision
Kimberly Gentry Sperber, Ph.D.
Workshop Objectives
• Review literature on fidelity
• Identify opportunities for assessing
fidelity
• Identify resources for monitoring
and improving fidelity
• Address barriers to monitoring
fidelity
Objective 1
What Does Science Say About
Fidelity?
Latessa, Cullen, and Gendreau (2002)
• Article notes 4 common failures of
correctional programs:
– Failure to use research in designing
programs
– Failure to follow appropriate assessment
and classification practices
– Failure to use effective treatment
models
– Failure to evaluate what we do
The Role of Fidelity
• Lowenkamp and Latessa (2005)
• Examined data from 38 residential
correctional programs for adults
• Looked at relationship between program
fidelity and program effectiveness.
• Program fidelity was assessed using the
CPAI.
• Found significant correlation between
fidelity and effectiveness
– CPAI scores correlated to reincarceration
Lowenkamp and Latessa Findings
Continued
• Differences in recidivism rates
based on CPAI scores:
– Scores of 0-49% demonstrated 1.7%
reduction compared to comparison
group.
– Scores of 50-59% demonstrated 8.1%
reduction.
– Scores of 60-69% demonstrated 22%
reduction.
CPAI Data Continued
• Holsinger (1999)
– Examined data from Adolescent Community
Correctional Facilities in Ohio
– Looked at relationship between program
fidelity and program effectiveness.
– Program fidelity was assessed using the
CPAI.
– Outcome measures examined included any
court contact, felony or misdemeanor,
felony, personal offense, and commitment
to a secure facility
CPAI Data Continued
• Total composite score significantly
correlated with all outcome measures.
• Each individual domain of the CPAI also
significantly correlated with all of the
outcomes
–
–
–
–
–
Program Implementation
Client Assessment
Program Characteristics
Staff Quality
Evaluation
More Fidelity Research
• Landenberger and Lipsey (2005)
– Brand of CBT didn’t matter but quality of implementation
did.
– Implementation defined as low dropout rate, close
monitoring of quality and fidelity, and adequate training for
providers.
• Sexton (2001)
– Direct linear relationship between staff competence
and recidivism reductions.
• Schoenwald et al. (2003)
– Therapist adherence to the model predicted post-treatment
reductions in problem behaviors of the clients.
• Henggeler et al. (2002)
– Supervisors’ expertise in the model predicted
therapist adherence to the model.
More Fidelity Research Cont’d.
• McHugo et al. (1999) found that Assertive Community
Treatment (ACT) teams with higher fidelity showed greater
reductions in substance use, higher rates of remission, and
fewer hospitalizations for dual diagnosis clients.
• Kirigin et al. (1982) found that higher fidelity among staff was
associated with greater reductions in delinquency.
• Bruns et al. (2005) compared high fidelity Wraparound sites
to low fidelity sites and found high fidelity sites to result in
improved social/academic functioning of children and lower
restrictiveness of placements.
• Drake et al. (1996) found that the site with higher fidelity to
the Supported Employment Model had better employment
outcomes for clients.
More Fidelity Research Cont’d.
• Schoenwald and Chapman (2007)
– A 1-unit increase in therapist adherence
score predicted 38% lower rate of criminal
charges 2 years post-treatment
– A 1-unit increase in supervisor adherence
score predicted 53% lower rate of criminal
charges 2 years post-treatment.
Washington State Example
(Barnoski, 2004)
• For each program (FFT and ART), an
equivalent comparison/control group was
created
• Felony recidivism rates were calculated for
each of three groups, for each of the
programs
• Youth who received services from therapists
deemed ‘competent’
• Youth who received services from therapists
deemed ‘not competent’
• Youth who did not receive any services (control
group)
Functional Family Therapy Results:
% New Felony
70
60
50
40
32
10
27
25
30
20
FFT Not Competent
Control group
19
13
9 6
FFT Competent
17
11
0
6 Months
12 Months
18 Months
Results calculated using multivariate models in order to control for potential differences between groups
Washington State Study Continued
• When FFT was delivered competently, the
program reduced felony recidivism by 38%
• When considering how much the program
costs, substantial savings in ‘avoided crime’
were observed – particularly for the
competent therapists
• When ART was competently delivered, felony
recidivism was reduced by 24%
• Also resulted in substantial savings
Project Greenlight
• Short-term prison-based reentry
program in New York
–
–
–
–
–
–
–
–
–
CBT Skills Training
Employment Services
Housing Services
Drug Education and Awareness
Family Counseling
Practical Skills Training
Community-Based Networks
Familiarity With Parole
Individualized Release Plans
Project Greenlight Benefits
• Participants received more service
referrals
• Participants reported more contacts
with community services after release
• Participants demonstrated significantly
more familiarity with parole conditions
• Participants were more positive about
parole
But Did It Work?
34.1
35
30
26.8
24.2
25
20
15
17.2
14.4 13
Greenlight
No Services
10
Existing Trans
Services
5
0
Arrests at 6 Months
Arrests at 12
Months
What Went Wrong?
• Violation of the risk principle
– Ceased use of risk assessment instrument when staff
deemed process too cumbersome
• Violation of the need principle
– All offenders received same services whether needed
or not
• Violation of the fidelity principle
– Staff modified delivery of the CBT curriculum
(shortened the duration, increased frequency,
increased class size)
• Differential staff competence
– Certain case managers produced worse outcomes
2010 UC Halfway House/CBCF Study in Ohio:
Adherence to CBT in Groups and Changes in Recidivism
7
6
5
4
3
2
1
0
-1
-2
-1
0
1
Fidelity and the Validity of the
LSI-R
• Flores, Lowenkamp, Holsinger, & Latessa, (2006)
– Higher correlations among scores produced by
trained staff and future incarceration
– Lower correlations among scores produced by
untrained staff and future incarceration
– Higher correlations among agencies using the LSI-R
for at least 3 years and future incarceration
– What are the implications of these findings?
Fidelity, Emotional Exhaustion,
and Staff Retention
• Aarons et al. (2009):
– Staff who received ongoing consultation
for fidelity in the EBP group had
significantly lower turnover than staff
receiving ongoing consultation in the
TAU group.
– Staff in EBP also demonstrated less
emotional exhaustion
What Do We Know About
Fidelity?
• Fidelity is related to successful
outcomes (i.e., reductions in recidivism,
relapse, and MH instability).
• Poor fidelity can lead to null effects or
even iatrogenic effects.
• Fidelity cannot be assumed
• Fidelity can be measured and
monitored.
Relationship Between Evaluation and
Treatment Effect (based on UC Halfway House and CBCF
study)
% Change in Recidivism
8
6
6
4
2
1
0
Internal QA
No Internal QA
NPC Research on Drug Courts
Drug Court Uses Evaluation Feedback to Make Modifications
Percent Improvement in Outcome Costs*
50%
44%
40%
30%
20%
11%
10%
0%
Yes
N=4
No
N=6
* "Percent improvement in outcome costs" refers to the percent savings for
drug court compared to business-as-usual
Objective 2
Opportunities for Assessing
Fidelity
Opportunities to Monitor Fidelity
• Training
• Assessments
• Treatment groups
• Individual sessions
• Case Management
• Milieu
• Documentation Review
• Program Assessments
Ensuring Training Transfer
• Use of knowledge-based pre/post-tests
• Use of knowledge-based proficiency tests
• Use of skill-based rating upon completion
of training
• Mechanism for use of data
– Staff must meet certain criteria or score to be
deemed competent.
– Failure to meet criteria results in consequent
training, supervision, etc.
Assessments
• Desktop Reviews
– Accurate scores
– Reviews of overrides
– Integration with service plan/dosage
• Observations
– Use of standardized audit sheet
– Assess interviewing skills
– Assess accuracy of item ratings
Treatment Groups
• Observation-based ratings of adherence
to treatment model.
– CBT:
•
•
•
•
•
•
•
•
Frequency of role-plays
Structure of role-plays
Appropriateness of role-plays
Use of behavioral reinforcers
Effective use of authority and disapproval
Teaches the thought-behavior chain
Teaches structured skill building
Follows curriculum
Individual Sessions
• Observation-based ratings of adherence
to treatment model.
– CBT:
•
•
•
•
•
•
•
Teaches thought-behavior chain
Teaches problem-solving
Teaches structured skill building
Conducts role-plays
Appropriate use of thinking reports/homework
Graduated practice
Appropriate use of reinforcers
Case Management
• Observation-based ratings of
adherence to treatment model.
– CBT:
• Teaches thought-behavior chain
• Teaches problem-solving
• Teaches structured skill building
• Appropriate use of reinforcers
• Helps client to integrate skills learned
into real world environment (e.g.,
employment)
Milieu
• Observation-based ratings of competence
in core correctional practices
– Focus is more on effective use of authority and
disapproval and appropriate use of reinforcers and
sanctions.
• Standardized list of behavioral indicators
• Structure for observing and rating staff
interacting with clients in milieu
• Can also review incident data for trends
Documentation Review:
Why Do It?
• Clinical Implications
– Documentation is not separate from service
delivery.
– Did the client receive the services he/she
needed?
• Operational Implications
– Good documentation should drive decisionmaking.
– Means of communication
• Risk Management Implications
– If it isn’t documented, it didn’t happen.
– Permanent record of what occurred in the facility
• Source of Staff Training
• Reflection of the provider and
organization’s competency:
– EBP
– Outcome of care
Program Assessments
• Correctional Program Checklist
(CPC)
• Correctional Program Assessment
Inventory (CPAI)
• ICCA Treatment Survey
Sample Measures
• Percentage of groups containing role-plays
• Percentage of successful completers receiving
appropriate dosage based on risk/needs
assessment
• Percentage of staff achieving 4:1 ratio
• Percentage of groups observed where staff
modeled the skill prior to having clients
engage in role-play
• Percentage of role-plays containing practice of
the correctives
• Percentage of role-plays that required
observers to identify skill steps and report
back to the group
Objective 3
Resources Required to
Monitor and Improve Fidelity
Observation-Based Ratings
• Creation of audit sheets
• Schedule for conducting the reviews
• Staff qualified to conduct and rate the
observations
• Time for staff to conduct observations
• Mechanism to record and use the data
– Supervision and individual staff
development
– QI and training initiatives
Documentation Review
• Staff to conduct the review
• Schedule for review rotation
• Audit sheet
• Time to conduct the review
• Mechanism for recording and using
the data
– Action planning
Objective 4
Barriers to Monitoring Fidelity
Common Barriers
• Strength of conceptual understanding of
the EBP to be measured
• Resources
• Setting priorities
• Understanding/skill sets required for
measurement
• Conflicting philosophies (helper vs.
evaluator)
• Time!
Potential Strategies
• Start small
– For example, desk top review of
assessments versus observation-based
ratings
• Use technology to increase efficiencies
– For example, videotape interactions for
observation-based ratings
• Take the time to build expertise
– Train on model
– Train on evaluation methodology
– Insure understanding of purpose (e.g., QI
versus punishment)
Conclusions
• Many agencies are allocating resources to
selection/implementation of EBP with no
evidence that staff are adhering to the model.
• There is evidence that fidelity directly affects
client outcomes.
• There is evidence that internal evaluation
processes directly affect client outcomes.
• Therefore, agencies have an obligation to
routinely assess and assure fidelity to EBP’s.
• Requires a formal infrastructure to routinely
monitor fidelity performance.
Questions and Answers
Contact Information:
[email protected]