Transcript Evaluation of the Fighting Back Initiative by Kay E. Sherwood

Evaluation of the Fighting
Back Initiative
by Kay E. Sherwood
Presented by
Maddie Velez
What is Fighting Back?
A community-based drug abuse
prevention program
Why study this Initiative?
Shows the importance of taking context into
evaluations
Raises questions about how community
interventions are conceptualized and
evaluated
Provides a warning about the manageability of
large-scale, comprehensive evaluations.
Background: Duration and
Scale Contribute to Complexity
◊ A 12 year initiative
◊ Original stakeholders differed greatly from the
stakeholders involved 12 years later
◊ Reduction of intervention sites went from 15
to 5
◊ First evaluation team replaced after 2 years.
◊ High staff / leadership turnover
◊ Original key leader retires
◊ Few examples of credible, successful
evaluations that truly measured the
interventions impact
The Foundation Takes on
Substance abuse
◊ Robert Wood Johnson heads up the
Foundation’s first efforts in the area of
substance abuse.
◊ First grant was made to Vanderbilt University
for $26.4 million in 1988
◊ Foundation explores addressing the national
problems of substance abuse and
dependence
Continued…
◊ July 1988…the goal became “by pulling
together into a single unified effort,
communities can begin to solve the pressing
problem of drug and alcohol abuse.”
◊ The expectation…”to reduce the demand for
illegal drugs and alcohol in the funded
communities.”
◊ Project STAR and ALERT
◊ Poly abuse - combination of mental health
problems and substance abuse occurring
New Leadership: Kathryn
Edmundson
◊ New evaluation agenda: Could you
organize to create political will for
change at the local level and get it to
add up to a national-level movement?
◊ An element of racism and elitism in the
law enforcement
◊ Expected outcomes
Evaluation I: Lost time,
Money, and Credibility
◊ 1990-1994
◊ The first evaluation team replaced, 4
years, $4.6 million, and a baseline
◊ Division between stakeholders missed
changes
◊ Augment between the 2nd evaluation
team and foundation staff regarding
lack of baseline data.
A 1996 Watershed
◊ Become unified with an emphasis on
prevention, early intervention, treatment, and
aftercare.
◊ NPO (National Program Office) moved to
Boston University School of Public Health.
◊ NPO joined another foundation funded
program called “Joined Together”, with new
director David Rosenbloom.
◊ Board of Trustees makes a recommendation
to give the program Fighting Back more time.
◊ Preliminary analysis indicates data that during
A National Program Office
Change
◊ Fighting Back reduces # of sites eligible for
new funding.
◊ Measure most substance abuse within the
communities to be able to do something
measurable at community level.
◊ Increasing treatment and treatment capacity
an important goal.
1994-2000 Evaluation II
◊ Consensus 2nd evaluation team does an
credible job with difficult circumstances.
◊ 1st Evaluators spend $4.6 million dollars
with little to show for it.
◊ Fighting Back Program and evaluation
staff is moving forward w/out
replacement dollars.
Relying on Survey Data
Phone surveys throughout the community.
Management Information Systems (MIS).
Ethnographic Studies.
Community Indicators
Four Research questions were identified by
the 2nd evaluation team.
◊ Strong correlations between strategies and
outcomes.
◊ Community Indicators
◊ School survey data difficult to use.
◊
◊
◊
◊
◊
The Price of Relying on Survey
Data
◊ 199 6 residue of distrust
◊ Saxe’s research team became known as the
“national evaluation
◊ Community has been seen as the “human
subject”
◊ National evaluation offer no alternative
to outcomes perspective
◊ High emotions surrounding analysis
emerged accusations
◊ Bickman claims bias evaluations; Eval. Team
are required to point out potential problems
in the interventions
The Evaluation’s Ability to Explain
◊ Evaluation illustrate all central problems for
evaluation
◊ Saxe wanted to undertake a more extensive
implementation analysis, foundation unwilling
to pay for it
◊ Fighting Back site activities revised after an
initial publication in 1997
◊ Knickman claims the foundation had the
wrong goals; He felt that there was a need
for shorter-term goals
Measuring and Interpreting
Outcomes
◊ Key disagreements remain a piece of
the national evaluation that focuses on
the use of household survey data
◊ 3 waves of surveys- 1995, 1997, 1999
◊ Jellinek described early thinking on the
evaluation
◊ Presentation of Results - A second area
of disagreement
The Continuing Debate and the
Foundation’s Takeaway
◊ Knickman and Morris presented a
summary of the Fighting Back
experience to the foundation’s board in
4/’04
◊ Knickman focused on the fundamentals
of complexity and the lessons about
realistic scale for expected outcomes
◊ Teams were formed
◊ Substance abuse- D.A.R.E. and
treatment reform