Transcript No Slide Title

WASHINGTON STATE UNIVERSITY EXTENSION
Evaluation Based
Program
Development
This material was developed in partnership with
Rutgers's University and
Lydia B. Blalock,
Youth Development Specialist,
New Brunswick, New Jersey
Why Evaluate?
• I dunno
• I think I knew why–
once
• They make me do it
– Department chair
– Funding
organization
– My mom
• I don’t
• Demonstrate success
• Understand failure
• Provide feedback for
program
improvement
– Process
– Outcome
• Nip problems in the
bud
Process Objectives
If I:
• Develop and present seminar on
EBPD
• Develop and distribute
appropriate handouts
• Engage participants in group
activities
• Provide participants an
opportunity to practice
• Encourage questions
THEN
Outcome Objectives
Participants will:
•
Program Development
– Understand how to clearly define issue
program will address.
– Use program theory to plan activities.
– Write clear outcome and process
objectives.
– Use EBPD Model to plan programs.
•
Program Evaluation
– Develop skills to evaluate programs.
– Select appropriate evaluation methods and
designs.
– Select criteria to evaluate objectives.
– Use EBPD Model to plan evaluation.
BUT FIRST: Is it a Program or a program
•Stakeholder leadership
•Faculty leadership
•Multi-disciplinary
PROGRAM
•Sustainable
•Academic department
•Student component
•Research, teaching, outreach
program/s
activity/ies
program/s
PROGRAM
event/s
activity/ies
event/s
The Steps of Evaluation-Based Program Development
6. Dissemination &
Scholarship 4
5. Implementation
6. Dissemination &
Implementation
4. Plan
Program
Evaluation
3.5.Plan
ProgramÑ
Scholarship
4
2. Develop
Outcome
Objectives
Ï
1. Define the Problem
4. Plan Program Evaluation
3. Plan ProgramÑ
2. Develop Outcome Objectives Ï
1. Define the Problem
Step 1: Define the Problem
•
•
•
•
Describe Current Situation
– How does the world look today?
Why a Problem
– Clearly articulate why a problem
– Ask why until get to root of problem
– Symptom vs. Cause
Identify Stakeholders
– Population: Who is affected? How many?
– Stakeholder Leadership: interest in solving
problem?
– Potential Collaborators: Already working
on problem?
How: Evidence? Learn more? References &
Facts
– Learn all you can about the issue/problem
–
•
Set Priorities
– Factors that influence priorities
•
•
•
•
•
Difference between actual and target states
Degree of difficulty in addressing needs
Cost of implementing solutions
Political and other factors
– Derive priorities from several data sources
– Priorities without resources
Select Solution Strategies to Meet Needs
– Consider alternative solutions for each
need
– Evaluate alternatives
– Select solutions to implement
– Scholarship Opportunities?
Table 1. Define the Problem
Identify who, what, where, when, why.
1. Describe Current Situation
How does the world look today?
2. Why a
Problem?
3. Identify Stakeholders
Clearly articulate
why current
situation is a
problem. Ask why
until you get to root
of problem.
Population.
Who is affected
by problem?
How many
people are
affected?
Stakeholder
Leadership.
Who’s
interested in
solving
problem?
Potential
Collaborators.
Who’s
interested in
solving problem
and/or already
working on
problem?
What is the evidence? How can I learn more? Provide references and facts.
TALK IT OVER
Fill Out Table 1 with your group.
Step 2: Develop Outcome Objectives
• Describe Ideal Situation
– How would world look w/o problem
(perfect world)?
• Define Target Audience/s
– Who/what can we change (youth,
families, policies, etc.)?
– What do we know about culture/s of
audience?
• Identify Outcome Objectives
– What changes can I influence in my
target audience/s?
• Identify Resources
– What do we need?
– What do we have?
Step 2. Develop Outcome Objectives
Ï
2. Develop Outcome ObjectivesÏ
What can we change in audience?
Define Target
Audience
Who can we change
to solve problem?
Short Term LEARNING&
Awareness, Skills, Attitudes, Opinions,
Knowledge Aspirations, Motivations
Medium Term ACTIONS·
Behavior, Practice, Decision-making,
Policies, Social Action
Long Term CONDITIONS þ
Social, Economic,
Civil, Environmental
IMPACT?
Identify Resources
What resources are
required, available,
needed?
Writing Measurable Objectives
• Specific
– Will increase knowledge vs. Will know
3 sources of
Vitamin C
• Time delineated
– At the end of the program, in 6 weeks,
etc.
• Measurable
– 75%, all, etc.
• Example
– Participants will increase knowledge
of nutrition.
– 75% of program participants will know
3 sources of Vitamin C 6 weeks after
the program.
Table 2: Clarifying Your Projected Impact
Table 2. Identify Outcome Objectives Ï
1. Describe Ideal Situation
How would the world look if the problem did not exist (the perfect world)?
2. Define Target
Audience/s
Who or what can we
realistically change
(youth, families,
farmers, policies,
etc.)?
What do we know
about the culture/s of
the audience? Do we
need to learn more?
3. Identify Outcome Objectives Ï
What changes can I influence in my target audience/s?
Learning
Objectives &
Action
Objectives ·
Short term changes
in awareness, skills,
knowledge,
attitudes,
aspirations,
motivation. List for
each target.
Medium term
changes in
behavior, policies,
practice, decisionmaking, social
actions. List for
each target.
Condition
Objectives 
Long term
changes in social,
civic, economic,
environmental
conditions.
Impact?
Target 1:
Target 2:
4. Identify Resources
Resources Required
Revenue Enhancement
Development
Opportunities
Talk it Over
Fill out Table 2 with your group.
#4: Brainstorm Resources
Step 3. Plan Program
Plan ProgramÑ
(Activities)
Develop Process ObjectivesÑ
What will you do to facilitate changes in
target audience/s?
Use “Program Theory” to develop process
objectives! IF:THEN
Program Theory: The Key Ñ to Unlocking Ï Change
IF we
Ñ

IF:THEN Statements
IF we ABC: THEN XYZ
do this . . .
THEN they
will change . . .
where
ABC = the events, programs, activities you will conduct, deliver,
develop, train, provide, access, facilitate, etc.

and where
Delivery
Modes
XYZ = participant outcome objectives previously developed
Ñ
PRODUCE
Results
Projects and Activities
ABC will become your process objectives.
Outcome Objectives
Process Objectives
Ñ
The actions you will take
to produce the desired
outcomes (write
curriculum, conduct
training)

PROCESS:OUTCOME
Statements
Changes in target
population as result of
your actions
(teachers adopt
curriculum, kids are
respectful)
Output the Process of an Outcome …wha…?
Outcome Objectives: the changes you will influence
in your target
audience
(Learning, Action, Condition)
Process Objectives: the events, programs, activities
you will conduct, deliver, develop, train,
provide, access, facilitate to reach your outcome
objectives
Program Outputs: your deliverables
Outcome & Process Objective Activity
Talk it Over
Fill out table 3 with your group
SO WHAT?
Friend to Groucho Marx: “Life is difficult!”
Marx to Friend: “Compared to what?”
Why We Don’t Evaluate
•
•
•
•
•
•
Too hard
Not qualified
Costs too much
Takes too much time
Don’t think about it “in time”
No one ever reads evaluation
reports
• Everyone “knows” it’s a good
program
Step 4: Program Evaluation
4. Plan Program
Evaluation
Specify Outcome Indicators Ï
How will you know if the audience
achieved the specified outcomes?
Select Evaluation Design
Specify Process Indicators Ñ
How will you know if you
successfully implemented program?
Select Data Collection Method
Evaluation Design 101
•
One-Shot
•
– Data collected following program,
• Retrospective Pretest
– Data collected following program;
participants recall behavior prior
to and following program.
• One Group Pretest-Posttest
- Data collected prior to
•
and following program
• Time Series
– Data collected prior to, during, after
program
Pretest-Posttest
Control-Group
– Data collected from 2 different
groups prior to and following
program.
– Treatment group: program
– Control group: no program
Post-Only Control-Group
– Data from 2 different groups
collected following program only
Evaluation Design 101
Evaluation Design
Pretest
One-Shot Design
Posttes # Times
Follow-up
t Assessed
X
1
Control
Group
Retrospective Pretest
X
X
1
One Group Pretest-Posttest
X
X
2
Time Series
X
X
3 (min)
Pretest-Posttest ControlGroup
X
X
2x per
group
X
X
1x per
group
X
Post-Only Control-Group
X
Data Collection Methods
•
Face to Face Surveys
– Respondent completes survey in presence of evaluator
– Evaluator may record responses and is able to answer
questions
•
Self Administered Questionnaires
– Respondent given survey to complete on his/her own.
– Respondent’s responsibility to return survey.
•
Personal Interviews
– Respondent asked pre-determined questions.
– Evaluator records responses, can ask respondent to clarify
answers.
•
Focus Groups
– Several respondents meet with evaluator.
– Evaluator facilitates discussion around pre-determined
questions.
– Discussion recorded.
Learned Anything? Let’s Check!
Learning, Action & Condition Objectives
Learning Objectives &
Short term changes in awareness, knowledge, skills, attitudes, aspirations, motivation
1. Develop Success Indicators &
NOTE: May have multiple indicators for
each objective.
We will know
(insert Learning
Objectives)
If (insert criteria
for success)
2. Select Evaluation
Design*
Type
Pro/Cons
3. Select Data Collection
Method*
Method
Pros/Cons
Think it Over
Fill out Table 4 with your group.
Learning Indicators
Action Indicators
Condition Indicators
Cultural Aspects of EBPD
•
•
•
What is culture?
– “Behaviors and beliefs characteristic of a
specific social, ethnic, or age group.”
– We are a product of our culture
– Culture influences (unconsciously) what we
think of people from other cultures.
– Recognize cultural biases towards participants
and strive to eliminate them.
Be aware of how differences between cultures
may impact EBPD
Neutralize cultural biases and prevent EBPD
design flaws:
– Know your audience (Step 2 – Develop
Outcome Objects)
– Develop cross-cultural efficacy!
Culturally Sensitive Design
Cultural Competence vs. Cross-cultural Efficacy
Cultural
competence
I can deal with
them.
My culture is superior
to their culture. (It’s
all about me!)
Cross-cultural
efficacy
I can effectively
interact with
people from other
cultures.
I can benefit from our
interactions. Neither
culture is superior to
the other. (It’s all
about us!)
Reporting & Scholarship
Table 6. Dissemination and Scholarship 4
How will you share your work?
Evaluation-Based Program
Development
1. Identify Scholarship
Activities
Table 1. Define Problem
Table 2. Identify Outcome
Objectives and Resources
Table 3. Plan Program
Table 4. Plan Program
Evaluation
Table 5. Implement Program,
Conduct Evaluation and
Document Impact
Table 6. Dissemination and Scholarship
Report Results
Demonstrate Impact on
Profession
Attain Recognition
2. Identify Products &
Deliverables
1. Define the Problem
2b.
ResourcesÏ
4. Plan Program Evaluation
3. Plan
ProgramÑ
2a. Target
Audience Ï
5. Implementation
2. Develop Outcome Objectives Ï
6. Dissemination & Scholarship 4
Process Objectives
If I, as facilitator:
• Develop and present seminar
on EBPD
• Develop and distribute
appropriate handouts
• Engage participants in group
activities
• Provide participants an
opportunity to practice
• Encourage questions
THEN
Outcome Objectives
Participants will:
•
Program Development
– Understand how to clearly define issue
program will address.
– Use program theory to plan activities.
– Write clear outcome and process
objectives.
– Use EBPD Model to plan programs.
•
Program Evaluation
– Develop skills to evaluate programs.
– Select appropriate evaluation methods and
designs.
– Select criteria to evaluate objectives.
– Use EBPD Model to plan evaluation.
Condition Changes
We’ll report stunning impacts
from well planned programs
And the folks in Pullman will
love us the most
And people will dump bucket
loads of money into our
programs
And the world will be a better
place!