Transcript No Slide Title

AGRICULTURE
YOUTH &
FAMILIES
HEALTH
ECONOMY
ENVIRONMENT
ENERGY
Evaluation Based Program
Development
Based on work by Lydia B. Blalock, Ph.D., Youth
Development Specialist
New Brunswick, New Jersey
COMMUNITIES
AGRICULTURE
YOUTH &
FAMILIES
HEALTH
ECONOMY
ENVIRONMENT
ENERGY
From Assessment to Impact:
Day 1
Karen Barale, Pierce County
Renee Overath, Kitsap County
Christy Price, Grant County
COMMUNITIES
Introduction
• Why are you here?
• When I hear Program Evaluation I
think . . .
• Education Philosophy
– Experiential: I can’t teach anything,
but I can provide opportunities to
learn!
– There’s no such thing as a stupid
question!
How We Got . . . Here
• Various backgrounds
• Program Evaluation Philosophy
– We can’t develop or perform an
evaluation if the program wasn’t built
upon a strong foundation!
• “Train the trainer” Evaluation-Based
Program Development
Process Objectives
If I:
• Develop and present seminar on EBPD
• Develop and distribute
appropriate handouts
• Engage participants in group activities
• Provide participants an opportunity to
practice
• Encourage questions
THEN
Outcome Objectives
Participants will:
• Program Development
–Understand how to clearly define
issue program will address.
–Use program theory to plan activities.
–Write clear outcome and process
objectives.
–Use EBPD Model to plan programs.
Outcome Objectives
Participants will:
• Program Evaluation
– Develop skills to evaluate programs.
– Select appropriate evaluation methods and
designs.
– Select criteria to evaluate objectives.
– Use EBPD Model to plan evaluation.
Outcome Objectives
Participants will:
• Scholarship of Engagement
– Understand scholarship opportunities in EBPD
process.
– Identify several avenues to share work.
– Use EBPD to plan scholarly activities.
Commit to putting new knowledge
and Aspects
skills into
practice!
• Cultural
of EBPD
– Understand influence of culture on EBPD (personal &
audience).
– Understand importance of learning culture of target
audience/s.
– Select evaluation methods appropriate for diverse
audiences.
BUT FIRST: Is it a Program or a program
•Stakeholder leadership
•Faculty leadership
event/s
PROGRAM
PROGRAM
•Multi-disciplinary
•Sustainable
program/s
•Academic department
program/s
•Student component
program/s
activity/ies
•Research, teaching, outreach
PROGRAM
activity/ies
activity/ies
event/s
event/s
Group Assignments
•
•
•
•
Form groups of 4
Sit with new group
Select bag
Develop program based upon contents
of bag
• Develop 10-15 minute (tops) program
presentation
Program Development Project: Requirements
• Define problem
– What method/s did you use
– Identify stakeholders/collaborators
• Develop outcome objectives
– Target audience
– Resources needs
– Short, medium, long term objectives
Program Development Project: Requirements
• Plan program activities (process objectives)
• Develop evaluation plan
– Success indicators
– Research design and data collection
methods
• Identify scholarship opportunities
Evaluation-Based Program Development: By the Numbers
6. Dissemination &
Scholarship 4
5. Implementation
&
4.Dissemination
Plan
Program
Evaluation
4. Plan6.
Program
Evaluation
5.
Implementation
3. Plan
ProgramÑ
2. Develop
Outcome
Objectives Ï
Scholarship
1. Define
the4
Problem
3. Plan ProgramÑ
2. Develop Outcome Objectives Ï
1. Define the Problem
Step 1. Define the Problem
• Describe Current Situation
– How does the world look today?
• Why a Problem
– Clearly articulate why a problem
– Ask why until get to root of problem
– Symptom vs. Cause
Step 1. Define the Problem
• Identify Stakeholders
– Population: Who is affected? How many?
– Stakeholder Leadership: interest in
solving problem?
– Potential Collaborators: Already working
on problem?
Step 1. Define the Problem
• How: Evidence? Learn more? References &
Facts
– Learn all you can about the issue/problem –
• “prove” the problem exists
• Identify potential solutions
• “Needs” Assessment
– Need: difference between actual and target
“state”
– A.K.A.: Asset, Gap, etc.
Needs Assessments
Small Group Activity:
Needs Assessment
• Purpose of “needs” assessment is to clearly
define or clarify issue.
• What kinds of information might you need?
• Who or what can provide the information?
• What methods could you use to gather
information?
• Where can you gather or find the
information?
Needs Assessments
• How will we gather the information?
– Surveys
– Interviews
– Focus groups
– Card sort
– Archival/secondary data
• Criteria for selecting methods
– Consider characteristics of target and
respondent groups
– Choose combo of methods to gain different
types of info
• When
– Develop target dates and timelines
Using Results
• Set Priorities
– Factors that influence priorities
•
•
•
•
Difference between actual and target states
Degree of difficulty in addressing needs
Cost of implementing solutions
Political and other factors
– Derive priorities from several data source
– Priorities without resources
Using Results
• Select Solution Strategies to Meet Needs
– Consider alternative solutions for each
need
– Evaluate alternatives
– Select solutions to implement
– Scholarship Opportunities?
Table 1. Define the Problem
Identify who, what, where, when, why.
1. Describe Current Situation
How does the world look today?
2. Why a
Problem?
3. Identify Stakeholders
Clearly articulate why
current situation is a
problem. Ask why until
you get to root of
problem.
Population. Who
is affected by
problem? How
many people are
affected?
Stakeholder
Leadership.
Who’s interested
in solving
problem?
Potential
Collaborators.
Who’s interested
in solving
problem and/or
already working
on problem?
What is the evidence? How can I learn more? Provide references and facts.
15 Minute Break
Step 2. Develop Outcome Objectives
Ï
• Who do you want to change?
• What do you want to accomplish?
• Do you have resources to accomplish
changes?
Step 2. Develop Outcome Objectives
Ï
• Describe Ideal Situation
– How would world look w/o problem
(perfect world)?
• Define Target Audience/s
– Who/what can we change (youth,
families, policies, etc.)?
– What do we know about culture/s of
audience?
Step 2. Develop Outcome Objectives
Ï
• Identify Outcome Objectives
– What changes can I influence in my
target audience/s?
• Identify Resources
– What do we need?
– What do we have?
Step 2. Develop Outcome Objectives
Ï
2. Develop Outcome ObjectivesÏ
What can we change in audience?
Define Target
Audience
Who can we change
to solve problem?
Short Term LEARNING&
Awareness, Skills, Attitudes, Opinions,
Knowledge Aspirations, Motivations
Medium Term ACTIONS·
Behavior, Practice, Decision-making,
Policies, Social Action
Long Term CONDITIONS þ
Social, Economic,
Civil, Environmental
IMPACT?
Identify Resources
What resources are
required, available,
needed?
Writing Objectives
• Specific
– Will increase knowledge vs. Will know 3
sources of Vitamin C
• Time delineated
– At the end of the program, in 6 weeks, etc.
• Measurable
– 75%, all, etc.
Writing Objectives
• Example
– Participants will increase knowledge of
nutrition.
– 75% of program participants will know 3
sources of Vitamin C 6 weeks after the
program.
Table 2. Identify Outcome Objectives Ï
1. Describe Ideal Situation
How would the world look if the problem did not exist (the perfect world)?
2. Define Target
Audience/s
Who or what can we
realistically change
(youth, families,
farmers, policies,
etc.)?
What do we know
about the culture/s of
the audience? Do we
need to learn more?
3. Identify Outcome Objectives Ï
What changes can I influence in my target audience/s?
Learning
Objectives &
Short term changes
in awareness, skills,
knowledge,
attitudes,
aspirations,
motivation. List for
each target.
Action
Objectives ·
Condition
Objectives 
Medium term
Long term
changes in
changes in social,
behavior, policies,
civic, economic,
practice, decisionenvironmental
making, social
conditions.
actions. List for
Impact?
each target.
Target 1:
Target 2:
4. Identify Resources
Resources Required
Revenue Enhancement
Development
Opportunities
Step 3. Plan Program
Plan ProgramÑ
(Activities)
Develop Process ObjectivesÑ
What will you do to facilitate changes in
target audience/s?
Use “Program Theory” to develop process
objectives!
Program Theory: The Key Ñ to Unlocking Ï Change
Ñ
IF we ABC: THEN XYZ
IF we
IF:THEN Statements

THEN they
do this . . .
will change . . .
where
ABC = the events, programs, activities you will conduct, deliver,
develop, train, provide, access, facilitate, etc.
and where
XYZ = participant outcome objectives previously developed
Ñ
Delivery Modes

Results
ABC will become PRODUCE
your process objectives.
Projects and Activities
Outcome Objectives
Process Objectives
Ñ
The actions you will take
to produce the desired
outcomes (write
curriculum, conduct
training)

PROCESS:OUTCOME
Statements
Changes in target
population as result of
your actions
(teachers adopt
curriculum, kids are
respectful)
Small Group Activity:
Process Outcome Challenge
Template: Process Objectives
2. Process Objectives Ñ
1. Outcome Objectives Ï
Develop Program Theory
Insert Outcome Objectives from Table 2.
(IF:THEN)
Condition Objectives
Learning Objectives
 If we = Conduct, deliver,

Action Objectives ·
&
develop, train, provide, etc.
Medium term changes Long term changes in
Short term changes in
 Then = Outcome
social, civic,
in behavior, policies,
awareness,
skills,
objectives from Table 2
economic,
practice, decisionknowledge, attitudes,
environmental
making, social actions.
aspirations, motivation.
conditions.
List for each target.
List for each target.
Impact?
If we (insert activities ABC on Then (insert Learning
lines below)
Objectives)
Then (insert Action
Objectives)
Then (insert Condition
Objectives)
3. Plan Program Outputs
Deliverables
Program, Curriculum, Emergency Kits
Scholarship 4
Go to Table 6!
WORKING LUNCH
1. Define the problem
2. Outcome objectives
3. Process objectives (program activities)
Step 4. Plan Program Evaluation
4. Plan Program
Evaluation
Specify Outcome Indicators Ï
How will you know if the audience
achieved the specified outcomes?
Specify Process Indicators Ñ
How will you know if you
successfully implemented program?
Why Evaluate?
• I dunno
• I think I knew why–once
• They make me do it
– Department chair
– Funding organization
– My mom
• I don’t
Why Evaluate?
• Demonstrate success
• Understand failure
• Provide feedback for program
improvement
– Process
– Outcome
• Nip problems in the bud
Why We Don’t Evaluate
•
•
•
•
•
•
•
Too hard
Not qualified
Costs too much
Takes too much time
Don’t think about it “in time”
No one ever reads evaluation reports
Everyone “knows” it’s a good program
Small Group Activity: Setting Criteria
• Establish criteria
– What are criteria (benchmarks, indicators)
for judging your item?
• Construct standards
– How well should your item perform on
benchmarks (standards)?
• Develop criteria rating system
Small Group Activity:
• AFTER developing criteria: measure
performance and compare with standards.
• How well does each type of item measure up
against criteria?
Template: Develop Success Indicators
1. Develop Success Indicators &
NOTE: May have multiple indicators for each objective.
We will know
(insert Learning Objectives)
If (insert criteria for success)
The chocolate chip cookie meets our
standard of excellence for aroma
It receives a 3.5 average score
for “aroma” from our test panel.
Process Indicators
• How will you know you’ve successfully
accomplished your process objectives?
Step 5. Implement Program
and Conduct Evaluation
• You planned your work.
– Now is the time to work the plan.
• If all goes well – great!
• If problems arise – check plan, revise
as needed
15 Minute Break
Cultural Aspects of EBPD
• What is culture?
– “Behaviors and beliefs characteristic of a
specific social, ethnic, or age group.”
– Product of our culture
– Culture influences (unconsciously) what we
think of people from other cultures.
– Recognize cultural biases towards
participants and strive to eliminate them.
Cultural Aspects of EBPD
• Neutralize cultural biases and prevent
embarrassing (or worse) EBPD design flaws is:
– Know your audience (Step 2 – Develop
Outcome Objects)
• Be aware of how differences between cultures
may impact EBPD
– Develop cross-cultural efficacy!
Cross Cultural Efficacy
Cultural Competence vs. Cross-cultural Efficacy
Cultural
competence
Cross-cultural
efficacy
My culture is superior
I can deal with them. to their culture. (It’s all
about me!)
I can effectively
interact with people
from other cultures.
I can benefit from our
interactions. Neither
culture is superior to
the other. (It’s all about
us!)
Small Group Activity:
Cultural Influences
Step 6. Dissemination of Results -Extension Scholarship
5. Implementation
4. Plan Program Evaluation
3. Plan ProgramÑ
2. Develop Outcome Objectives Ï
1. Define the Problem
6. Dissemination &
Scholarship 4
Step 6. Dissemination of Results - Extension Scholarship
• Tenure track faculty engage in appropriate
scholarly activities
• Scholarship
– Development and implementation of creative
concepts . . .
– . . . broadly communicated and validated by
appropriate peers
Step 6. Dissemination of Results - Extension Scholarship
• Forms
– Publications
• peer-reviewed journals, books, proceedings,
abstracts
• approved extension publications
– Peer-validated presentations
– Creation of other forms of durable scholarly
products
• includes digital and electronic media
• that have been scrutinized by peers and widely
communicated.
• Scholarship not simply act of doing creative
work . . .
• . . . must be peer validated and communicated
Examples of Outreach Scholarship
• PowerPoint Presentation
• Original Handouts and Templates
• Professional presentation (abstract,
talks, papers)
• Participant Workbook
• Curriculum
• Train the Trainer Curriculum
Impact Reporting: A Core Competency
• Evaluate & clearly document
programmatic outcomes & impacts
• Short, intermediate, long-term
• http://ext.wsu.edu/pd/documents/Guideli
nesforDocumentingOutcomesFinal.pdf
Table 6. Dissemination and Scholarship 4
How will you share your work?
Evaluation-Based Program
Development
1. Identify Scholarship
Activities
Table 1. Define Problem
Table 2. Identify Outcome
Objectives and Resources
Table 3. Plan Program
Table 4. Plan Program
Evaluation
Table 5. Implement Program,
Conduct Evaluation and
Document Impact
Table 6. Dissemination and Scholarship
Report Results
Demonstrate Impact on
Profession
Attain Recognition
2. Identify Products &
Deliverables
Small Group Activity: Extension Scholarship
Day 1: It’s Over!!!
Day 2 Overview
• 8:30 Sign-in, Continental Breakfast
• 9:00 Review, Questions
• 9:15 Part I Program Evaluation
• 10:15 Break
• 10:30 Part II Program Evaluation
• 11:15 Program Development Template
• 12 Working Lunch
• 1:00 Questions
• 1:30 Group Projects
• 3:35 Day 2 Post-Assessment, Day 3
Overview
Thank You!
Have a Good Evening
AGRICULTURE
YOUTH &
FAMILIES
HEALTH
ECONOMY
ENVIRONMENT
ENERGY
Evaluation-Based
Program Development
Day 2
COMMUNITIES
Welcome
• Evaluation Models, Approaches and
Design
• Program Template
• Group Projects
Step 4. Plan Program Evaluation
4. Plan Program
Evaluation
Specify Outcome Indicators Ï
How will you know if the audience
achieved the specified outcomes?
Select Evaluation Design
Specify Process Indicators Ñ
How will you know if you
successfully implemented program?
Select Data Collection Method
Evaluation Design 101
• One-Shot
– Data collected following program,
• Retrospective Pretest
– Data collected following program; participants recall
behavior prior to and following program.
• One Group Pretest-Posttest
– Data collected prior to and following program
Evaluation Design 101
• Time Series
– Data collected prior to, during, after program
• Pretest-Posttest Control-Group
– Data collected from 2 different groups prior to and
following program.
– Treatment group: program
– Control group: no program
• Post-Only Control-Group
– Data from 2 different groups collected following
program only
Evaluation Design 101
Evaluation Design
Pretest Posttest
One-Shot Design
# Times
Assessed
X
1
Follow-up
Control
Group
Retrospective Pretest
X
X
1
One Group Pretest-Posttest
X
X
2
Time Series
X
X
3 (min)
Pretest-Posttest Control-Group
X
X
2x per
group
X
X
1x per
group
X
Post-Only Control-Group
X
Data Collection Methods
• Face to Face Surveys
– Respondent completes survey in presence of
evaluator
– Evaluator may record responses and is able to
answer questions
• Self Administered Questionnaires
– Respondent given survey to complete on
his/her own.
– Respondent’s responsibility to return survey.
Data Collection Methods
• Personal Interviews
– Respondent asked pre-determined questions.
– Evaluator records responses, can ask
respondent to clarify answers.
• Focus Groups
– Several respondents meet with evaluator.
– Evaluator facilitates discussion around predetermined questions.
– Discussion recorded.
Small Group Activity: Evaluation Designs and Methods
Data Collection Template
Learning Objectives &
Short term changes in awareness, knowledge, skills, attitudes, aspirations, motivation
1. Develop Success Indicators &
NOTE: May have multiple indicators for
each objective.
We will know
(insert Learning
Objectives)
If (insert criteria
for success)
2. Select Evaluation
Design*
Type
Pro/Cons
3. Select Data Collection
Method*
Method
Pros/Cons
Evaluation-Based Program Development: By the Numbers
6. Dissemination &
Scholarship 4
5. Implementation
6.4.Dissemination
&
Plan
Program
Evaluation
5.
Implementation
3.
Plan
ProgramÑ
2. Develop
Outcome
Objectives Ï
Scholarship
1. Define
the4
Problem
4. Plan Program Evaluation
3. Plan ProgramÑ
2. Develop Outcome Objectives Ï
1. Define the Problem
WORKING LUNCH
Please return by 1:30
EBPD: Full Model
1. Define the Problem
What is issue or problem? Why* is this a problem?
Who is affected by or interested in solving it?
2. Develop Outcome ObjectivesÏ
What can we change in audience?
2. Develop Outcome ObjectivesÏ
3. Plan ProgramÑ
Plan ProgramÑ
4.3. Plan
Program
(Activities)
Short Term LEARNING&
Evaluation
6.
Dissemination
&
1.Awareness,
Define
the
Skills,Problem
Attitudes, Opinions,
(Activities)
What can we change in audience?
Define Target
Identify Resources
Define Target
What resources areIdentify Resources
Audience
Short Term LEARNING&
Audience
required,
available,What resources are
Who can we change
Skills, Attitudes,
Opinions,
Scholarship
4Awareness,
What is issue or
problem?
Why*
this
a problem?
5.
Implementation
required, available,
Who can
we
changeis Motivations
Knowledge
Aspirations,
Knowledge Aspirations, Motivations
needed?
to solve problem?
needed?
to solve problem?
(Program
& Evaluation)
Who is affected
by or interested
in solving it?
IF:THEN
IF we (conduct, provide, train, etc.):
THEN we can expect these Medium
outcomes
Medium Term ACTIONS·
Term ACTIONS·
Behavior, Practice, Decision-making,
Behavior, Practice, Decision-making,
Policies, Social Action
Policies, Social Action
Develop Process ObjectivesÑ
Specify Outcome Indicators Ï
Specify
Process
Indicators Ñ
Long Term
CONDITIONS þ
Social,
How will you know if the audience
How
willEconomic,
you know if you
Long Term CONDITIONS þ
Civil, Environmental
achieved the specified outcomes? Social, Economic,successfully implemented program?
What will you do to facilitate changes in
target audience/s?
Develop Process ObjectivesÑ
Civil,
Environmental
What will you do
to facilitate
changes in
target audience/s?
4. Plan Program
Evaluation
Specify Process Indicators Ñ
How will you know if you
successfully implemented program?
Specify Outcome Indicators Ï
How will you know if the audience
achieved the specified outcomes?
5. Implementation
(Program & Evaluation)
6. Dissemination &
Scholarship 4
The Template
Day 3 Overview
• 8:00 – 9:30 Sign In, Working Breakfast
(projects)
• 9:30 – 11:00 Presentations
• 11:00 – 11:15 Debriefing
• 11:15 – 12 :00 Lunch (non-working!)
• 1:00 – 3:00 Planning Session: Discuss
Training Strategies
• 3:00 Travel Home Safely!
Small Group Activity:
Develop Your Program
AGRICULTURE
YOUTH &
FAMILIES
HEALTH
ECONOMY
ENVIRONMENT
ENERGY
Evaluation-Based
Program Development
Day 3
COMMUNITIES
Group Presentations
Resources
• Extension Websites
– www.cyfernet
– www.sammie.osu.edu
– http://www.uwex.edu/ces/pdande/evaluation
/index.html
• Journals
– Journal of Extension
– Youth Development Journal
– American Evaluation Association
Resources
• Databases
– Discipline Journals
– PsycLit
– OVID
• Evaluation Specialists
• Faculty with assessment & evaluation experience
Evaluation-Based Program Development: Process Objectives
If I, as facilitator:
• Develop and present seminar on EBPD
• Develop and distribute appropriate
handouts
• Engage participants in group activities
• Provide participants an opportunity to
practice
• Encourage questions THEN
Outcome Objectives
Participants will:
• Program Development
–Understand how to clearly define
issue program will address.
–Use program theory to plan activities.
–Write clear outcome and process
objectives.
–Use EBPD Model to plan programs.
Outcome Objectives
Participants will:
• Program Evaluation
– Develop skills to evaluate programs.
– Select appropriate evaluation methods and
designs.
– Select criteria to evaluate objectives.
– Use EBPD Model to plan evaluation.
Outcome Objectives
Participants will:
• Scholarship of Engagement
– Understand scholarship opportunities in EBPD
process.
– Identify several avenues to share work.
– Use EBPD to plan scholarly activities.
Commit to putting new knowledge
and Aspects
skills into
practice!
• Cultural
of EBPD
– Understand influence of culture on EBPD (personal &
audience).
– Understand importance of learning culture of target
audience/s.
– Select evaluation methods appropriate for diverse
audiences.
1. Define the Problem
What is issue or problem? Why* is this a problem?
Who is affected by or interested in solving it?
6. Dissemination &
Scholarship 4
3. Plan ProgramÑ
2. Develop Outcome ObjectivesÏ
5. Implementation
What can we change in audience?
(Activities)
Define Target
Audience
Who can we change
to solve problem?
Short Term LEARNING&
Awareness, Skills, Attitudes, Opinions,
4. Plan Program Evaluation
IF:THEN
IF we (conduct, provide, train, etc.):
THEN we can expect these outcomes
Knowledge Aspirations, Motivations
Identify Resources
What resources are
required, available,
needed?
Medium Term ACTIONS·
Behavior, Practice, Decision-making,
Policies, Social Action
3. Plan ProgramÑ
Develop Process ObjectivesÑ
What will you do to facilitate changes in
target audience/s?
Long Term CONDITIONS þ
Social, Economic,
2. Develop Outcome Objectives
Civil,Ï
Environmental
4. Plan Program
Evaluation
Specify Process Indicators Ñ
How will you know if you
successfully implemented program?
1. Define the Problem
5. Implementation
Specify Outcome Indicators Ï
How will you know if the audience
achieved the specified outcomes?
(Program & Evaluation)
6. Dissemination &
Scholarship 4
The Last Word: Begin with the End in Mind
If you don’t know where you’re going, how are you
gonna’ know when you get there?
Yogi
Berra
Alice asked: Would you tell me please which way I
ought to go from here?
That depends a good deal on where you
want to get to, said the Cat. I don’t much care
where . . . said Alice. Then it doesn’t matter which
way you go, said the Cat.
From Alice’s Adventures in Wonderland
THE END
• Thank You!
• Final Questions
• Post-Assessment
AGRICULTURE
YOUTH &
FAMILIES
HEALTH
ECONOMY
ENVIRONMENT
ENERGY
COMMUNITIES
The “Logic” Behind Evaluation Based
Program Development
A Rose by Any Other Name . . .
Can Still Have Thorns!
1. Define the Problem
2b.
ResourcesÏ
4. Plan Program Evaluation
3. Plan
ProgramÑ
2a. Target
Audience Ï
5. Implementation
2. Develop Outcome Objectives Ï
6. Dissemination & Scholarship 4