Transcript Formative Evaluation

FORMATIVE
EVALUATION
Intermediate Injury Prevention Course
August 23-26, 2011, Billings, MT
Session Goal
To provide participants with a basic
knowledge of formative evaluation and
its use when developing an injury
prevention program.
Session Objectives
 Describe the purpose of
formative evaluation
 Describe several formative
evaluation methods
 Provide resources for each
formative evaluation method
The 4 Stages of Evaluation
Review
 Formative
 Will this work?
 Process
 Are we doing what we set out to do?
 Impact
 Can we see any changes?
 Outcome
 Have we reduce injuries?
The 4 Stages of Evaluation
Review
Baseline Data Collection
Project Implementation
Looks at Changes in
Target Audience
Formative
Evaluation
Process
Evaluation
Impact
Evaluation
Short-term:
Knowledge,
Attitudes &
Practice
Timely Feedback
Pre-Testing
Monitoring
Outcome
Evaluation
Long-term:
Health
Outcomes
Formative Evaluation
Defined
The process of testing and /or obtaining
expert review of program plans,
messages, materials, strategies, or
modifications, for weaknesses and
strengths before they are put into effect.
Formative Evaluation
Purpose
 Get target audience perception of the
possible interventions
 Gain community support
 Ensure program materials & activities are
acceptable or culturally appropriate
 Identify problems with program concepts
or design
 Used in conjunction with process
evaluation
Formative Evaluation
When to Conduct
 New program
 New materials
 Existing program is:
 modified,
 having problems, or
 adapted for a new setting, population,
problem, or behavior
 Ongoing at all stages of program
Formative Evaluation
Who to Involve - Stakeholders
The people with a vested interest in
a program and the future use of
its evaluation
Formative Evaluation
Categories of Stakeholders
 Implementers
 Partners
 Participants
 Decision makers
Formative Evaluation
Stakeholders
Example: Car seat distribution program
Who should we include?
 Implementers
 Partners
 Participants
 Decision makers
Formative Evaluation
Data Collection Methods
1.
2.
3.
4.
5.
Focus Groups
Key Informant Interviews
Town Hall Meetings
Expert Reviews
Surveys
Formative Evaluation
Focus Groups - Purpose
To gather information on a particular
topic by conducting an informal
discussion between 6 to 10 individuals
similar to target population
Formative Evaluation
Focus Groups
 8 Important Steps
 Secret Ingredients:
-Focus group similar in makeup to
target population
-Requires an experienced facilitator
Formative Evaluation
Key Informant Interviews - Purpose
To obtain detailed information by
asking broad open-ended questions
and by probing interviewee
Formative Evaluation
Key Informant Interviews
 7 Important Steps
 Secret Ingredients:
-Take care to select informants with various
points of view
-Remember to first establish rapport
KEY INFORMANT EXERCISE
Scenario:
Local IPS is seeking to determine attitudes
and beliefs about ATV related injuries and
possible interventions. They will interview two
key community members.
Formative Evaluation
Town Hall Meetings - Purpose
To gather a diverse group of community
members together to think and talk
about an issue
Formative Evaluation
Town Hall Meetings
 8 important steps
 Special Ingredients
 Plan early
 Make sure there is plenty of time for interaction
between audience and presenters
Formative Evaluation
Expert Reviews - Purpose
To identify mistakes, concerns, and
general areas for improvement.
Formative Evaluation
Expert Reviews
 7 important steps
 Special Ingredients
 The committee should be continuously
involved in the process.
 Include detailed instruction in the job
description.
Formative Evaluation
Surveys - Purpose
To obtain SPECIFIC information
from a targeted group of people
using a set of defined questions.
Formative Evaluation
Surveys
 7 important steps
 Special Ingredients
 Avoid overloading the survey
 Design questionnaire such that its clear
and simple.
Formative Evaluation
Examples in IHS
 Sleep Safe and Ride Safe Curriculums –
Focus Groups, Key Informant
Interviews, Expert Review
 Animal Control Project - Surveys
 Child Passenger Safety Training - Expert
Review
 IHS Area Injury Prevention Program
Evaluations - Expert Review
Formative Evaluation
Conclusion
 Formative evaluation
 Tests program plans, messages,
materials, strategies, or modifications
for weaknesses and strengths before
they are put into effect.
 Allows the evaluator to probe the
feelings, beliefs, and impressions of
participants.
 Generates community support for the
proposed project or program.