Transcript Using the Logic Model to Develop Evaluation Research

Using the Logic Model to
Develop Evaluation
Research Design and
Measures
Program Planning and Evaluation
Solid program planning is vital to a well
designed evaluation.
 Community partners (stakeholders) need
to have a meaningful way to provide input
to program planning.
 Extension staff must be able to
communicate outcomes to funders and
stakeholders.

Logic Models
Well developed, easy to use tool for
planning educational programs, using
stake holder input.
 Provide a common language
 Simple, graphic way to communicate
program theory
 Provides planners with a road map –
asking them to determine where they want
to end up (goals) and then chart their
course of action

Outcomes based
Outcomes must be clearly written and the
relationship between learning, action and
long-term outcomes must be logical.
 Forces planners to think about and
articulate their goals and objectives.
 Is an easy model for stakeholders to
understand and provide feedback on.

Generic logic model
Strategy
Inputs
»
Impact
Outputs
Activities
»
Outcomes
»
Outcomes
»
Outcomes
Participation
»
CONTEXT
A diagram of the theory of how a program is supposed to work
A graphic depiction of relationships between activities and results
Source: University of Wisconsin – Extension
http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html
University of New Hampshire
Cooperative Extension






Began staff development efforts in 2001
Provided in-services, mouse pads, web support and
even one-on-one support to staff
Began presenting the model to advisory committees
Some resistance from staff, but many liked the model
More staff are using it now to obtain grant funding
Suggested model for communities proposing CYFAR
– New Communities projects
Benefits of logic modeling to
partnerships





Provides framework to discuss and articulate joint
work
Keeps end outcome upfront and center
Helps facilitate conversation with focus on agreed
upon goal that might not happened otherwise
Utilizes local needs assessments and works with
community partners to inform Logic Model
development
Separates indicators of achievement - # papers
published, # and type of experiments completed –
from theory of change. Indicators of achievement are
part of evaluation plan for the partnership.
Case Examples
Seacoast Youth
Leadership Program
Hillsboro After School
Program
Seacoast Youth Leadership
Program
This project will evaluate the efficacy of the youth diversion
program’s ability to achieve improvements in the following
objectives:






Leadership
Future Aspirations/Goal Setting
Social Responsibility
Decision Making/Social Skills
Social Connectedness
Media Literacy
SYLP – Logic Model
Inputs
Outputs
Participants
Activities
Learning
Outcomes
Action
Outcomes
Condition
Outcomes
H.S.
graduation
rates and living
wage
crime
social capital
among youth
and adults
safe, positive
environments
for youth
SYLP – Logic Model, cont
Inputs
Outputs
Participants
Activities
Learning
Outcomes
Action
Outcomes
Improved
school
attendance &
performance
Healthy, drug
free lifestyle
Demonstrate
care and
concern for
others
Positive
decision
making
Condition
Outcomes
H.S.
graduation
rates and living
wage
crime
social capital
among youth
and adults
safe, positive
environments
for youth
SYLP – Logic Model, cont
Inputs
Outputs
Participants
Activities
Learning
Outcomes
Action
Outcomes
life skill
knowledge &
competencies
Improved
school
attendance &
performance
technology
intelligence &
media literacy
Healthy, drug
free lifestyle
Improved prosocial norms,
citizenship &
commitment to
peers, family &
community
Goal setting,
‘Can do’ attitude
Demonstrate
care and
concern for
others
Positive
decision
making
Condition
Outcomes
H.S.
graduation
rates and living
wage
crime
social capital
among youth
and adults
safe, positive
environments
for youth
SYLP – Logic Model, cont
Inputs
Outputs
Participants
Activities
Learning
Outcomes
Action
Outcomes
Research-based,
best practice
curriculum
life skill
knowledge &
competencies
Improved
school
attendance &
performance
Life skills through
experiential
learning –
leadership and
service learning
technology
intelligence &
media literacy
Healthy, drug
free lifestyle
Support &
learning for
parents
Mobilize
community
partnership
Improved prosocial norms,
citizenship &
commitment to
peers, family &
community
Goal setting,
‘Can do’ attitude
Demonstrate
care and
concern for
others
Positive
decision
making
Condition
Outcomes
H.S.
graduation
rates and living
wage
crime
social capital
among youth
and adults
safe, positive
environments
for youth
SYLP – Logic Model, cont
Inputs
Outputs
Participants
Activities
Youth/
parents
High-risk,
low-income
families &
communities
Lower
Seacoast
Youth
Collaborative
Learning
Outcomes
Action
Outcomes
Research-based,
best practice
curriculum
life skill
knowledge &
competencies
Improved
school
attendance &
performance
Life skills through
experiential
learning –
leadership and
service learning
technology
intelligence &
media literacy
Healthy, drug
free lifestyle
Support &
learning for
parents
Mobilize
community
partnership
Improved prosocial norms,
citizenship &
commitment to
peers, family &
community
Goal setting,
‘Can do’ attitude
Demonstrate
care and
concern for
others
Positive
decision
making
Condition
Outcomes
H.S.
graduation
rates and living
wage
crime
social capital
among youth
and adults
safe, positive
environments
for youth
SYLP – Logic Model, cont
Inputs
Outputs
Participants
Activities
Human
Resources
Youth/
parents
Material
Resources
High-risk,
low-income
families &
communities
Community
Partnership
Lower
Seacoast
Youth
Collaborative
Learning
Outcomes
Action
Outcomes
Research-based,
best practice
curriculum
life skill
knowledge &
competencies
Improved
school
attendance &
performance
Life skills through
experiential
learning –
leadership and
service learning
technology
intelligence &
media literacy
Healthy, drug
free lifestyle
Support &
learning for
parents
Mobilize
community
partnership
Improved prosocial norms,
citizenship &
commitment to
peers, family &
community
Goal setting,
‘Can do’ attitude
Demonstrate
care and
concern for
others
Positive
decision
making
Condition
Outcomes
H.S.
graduation
rates and living
wage
crime
social capital
among youth
and adults
safe, positive
environments
for youth
Hillsboro After school Program
This project will evaluate the efficacy of the after
school program’s ability to achieve improvements in
the following objectives:








Academic Performance
Future Aspirations
Social Connectedness
Social Responsibility
Sense of Self
Parenting Efficacy
Social Skills/Decision Making
Community Involvement with Youth/School
Hillsboro After School – Future Aspirations
Inputs
Outputs
Participants
Activities
Learning
Outcomes
Action
Outcomes
Impact
Youth pursue
postsecondary
education and
apprenticeship
-type
experiences to
attain what
they need to
enter and
succeed in a
chosen field.
Future Aspirations, cont
Inputs
Outputs
Participants
Activities
Learning
Outcomes
Action
Outcomes
Impact
Youth
articulate a
positive view
of their future,
including
planned
career/work
goals/life
goals.
Youth pursue
postsecondary
education and
apprenticeship
-type
experiences to
attain what
they need to
enter and
succeed in a
chosen field.
Youth make
connections/
network within
the
community.
Future Aspirations, cont
Inputs
Outputs
Participants
Activities
Learning
Outcomes
Youth learn
about various
career options
related to their
interests,
talents, skills.
Youth develop a
new skill and
explore related
careers.
Action
Outcomes
Impact
Youth articulate
a positive view
of their future,
including
planned
career/work
goals/life goals.
Youth pursue
post-secondary
education and
apprenticeshiptype
experiences to
attain what they
need to enter
and succeed in
a chosen field.
Youth make
connections/
network within
the community.
Future Aspirations, cont
Inputs
Outputs
Participants
Activities
7-8 week series of
special interest 4H projects using
4-H CCS, JASON,
Robotics, NASA
and computer
literacy curriculum
Field trips within
community to
learn about local
careers and jobs.
Learning
Outcomes
Youth learn
about various
career options
related to their
interests,
talents, skills.
Youth develop a
new skill and
explore related
careers.
Action
Outcomes
Impact
Youth articulate
a positive view
of their future,
including
planned
career/work
goals/life goals.
Youth pursue
post-secondary
education and
apprenticeshiptype
experiences to
attain what they
need to enter
and succeed in
a chosen field.
Youth make
connections/
network within
the community.
Future Aspirations, cont
Inputs
Outputs
Participants
Activities
Youth
Program Staff
4-H
Volunteers
Business &
industry
members from
the community
& region
7-8 week series of
special interest 4H projects using
4-H CCS, JASON,
Robotics, NASA
and computer
literacy curriculum
Field trips within
community to
learn about local
careers and jobs.
Learning
Outcomes
Youth learn
about various
career options
related to their
interests,
talents, skills.
Youth develop a
new skill and
explore related
careers.
Action
Outcomes
Impact
Youth articulate
a positive view
of their future,
including
planned
career/work
goals/life goals.
Youth pursue
post-secondary
education and
apprenticeshiptype
experiences to
attain what they
need to enter
and succeed in
a chosen field.
Youth make
connections/
network within
the community.
Future Aspirations, cont
Inputs
UNHCE staff &
volunteers
Access to
school library/
Computers
Program space
4-H CCS,
JASON, NASA
& computer
literacy
curriculum
Material,
supplies
Snacks
Transportation
College
student interns
Outputs
Participants
Activities
Youth
Program Staff
4-H
Volunteers
Business &
industry
members
from the
community &
region
7-8 week series of
special interest
4-H projects using
4-H CCS, JASON,
Robotics, NASA
and computer
literacy curriculum
Field trips within
the community to
learn about local
careers and jobs.
Learning
Outcomes
Youth learn
about various
career options
related to their
interests,
talents, skills.
Youth develop a
new skill and
explore related
careers.
Action
Outcomes
Impact
Youth articulate
a positive view
of their future,
including
planned
career/work
goals/life goals.
Youth pursue
post-secondary
education and
apprenticeshiptype
experiences to
attain what they
need to enter
and succeed in
a chosen field.
Youth make
connections/
network within
the community.
Hillsboro After School – Community Involvement
Inputs
Outputs
Participants
Activities
Learning
Outcomes
Action
Outcomes
Impact
Citizens
recognize and
act to ensure
community
wide support
and
involvement in
school, after
school and
youth/family
programs
After school
program is
expanded and
sustained
Hillsboro After School – Community Involvement, cont
Inputs
Outputs
Participants
Activities
Learning
Outcomes
Action
Outcomes
Youth and
adults work
together to set
and reach
goals.
youth serving
as equal
partners on
community
boards &
committees
Farmers’
Market
participation
with growers
accepting Food
Stamp program
Impact
Citizens
recognize and
act to ensure
community
wide support
and
involvement in
school, after
school and
youth/family
programs
After school
program is
expanded and
sustained
Hillsboro After School – Community Involvement, cont
Inputs
Outputs
Participants
Activities
Learning
Outcomes
Action
Outcomes
knowledge of
jobs, careers &
resources in
community.
Youth and
adults work
together to set
and reach
goals.
Youth & adults
learn to work in
partnership.
Adults learn the
importance &
value of youth
as a resource to
community.
Youth learn
components of
an effective &
healthy
community.
youth serving
as equal
partners on
community
boards &
committees
Farmers’
Market
participation
with growers
accepting Food
Stamp program
Impact
Citizens
recognize and
act to ensure
community
wide support
and
involvement in
school, after
school and
youth/family
programs
After school
program is
expanded and
sustained
Hillsboro After School – Community Involvement, cont
Inputs
Outputs
Participants
Activities
Field trips / visits
with community
leaders.
Training in
community youth
development
Steering
committee
reviews/ sets
program goals
and shares vision
Adults & youth
partner on after
school program &
Teen Center
development
Learning
Outcomes
Action
Outcomes
knowledge of
jobs, careers &
resources in
community.
Youth and
adults work
together to set
and reach
goals.
Youth & adults
learn to work in
partnership.
Adults learn the
importance &
value of youth
as a resource to
community.
Youth learn
components of
an effective &
healthy
community.
youth serving
as equal
partners on
community
boards &
committees
Farmers’
Market
participation
with growers
accepting Food
Stamp program
Impact
Citizens
recognize and
act to ensure
community
wide support
and
involvement in
school, after
school and
youth/family
programs
After school
program is
expanded and
sustained
Hillsboro After School – Community Involvement, cont
Inputs
Outputs
Participants
Activities
Community
adults
 Youth
 Program
Staff & 4-H
volunteers
 Social
services &
health care
workers
T own
government
 Business
leaders,
regional
planning
commission
 Main Street
project
leaders

Field trips / visits
with community
leaders.
Training in
community youth
development
Steering
committee
reviews/ sets
program goals
and shares vision
Adults & youth
partner on after
school program &
Teen Center
development
Learning
Outcomes
Action
Outcomes
knowledge of
jobs, careers &
resources in
community.
Youth and
adults work
together to set
and reach
goals.
Youth & adults
learn to work in
partnership.
Adults learn the
importance &
value of youth
as a resource to
community.
Youth learn
components of
an effective &
healthy
community.
youth serving
as equal
partners on
community
boards &
committees
Farmers’
Market
participation
with growers
accepting Food
Stamp program
Impact
Citizens
recognize and
act to ensure
community
wide support
and
involvement in
school, after
school and
youth/family
programs
After school
program is
expanded and
sustained
Hillsboro After School – Community Involvement, cont
Inputs
Program
space
Snacks
Tech – AV,
computers
Community
development
plan
CYD
resources
Community
gardens
Staff from
various
disciplines
Outputs
Participants
Activities
Community
adults
Youth
Program Staff
& 4-H
volunteers
Social
services &
health care
workers
Town
government
Business
leaders,
regional
planning
commission
Main Street
project
leaders
Field trips / visits
with community
leaders
Training in
community youth
development
Steering
committee
reviews/ sets
program goals
and shares vision
Adults & youth
partner on after
school program &
Teen Center
development
Learning
Outcomes
Action
Outcomes
knowledge of
jobs, careers &
resources in
community
Youth and
adults work
together to set
and reach
goals
Youth & adults
learn to work in
partnership
Adults learn the
importance &
value of youth
as a resource to
community
Youth learn
components of
an effective &
healthy
community
youth serving
as equal
partners on
community
boards &
committees
Farmers’
Market
participation
with growers
accepting Food
Stamp program
Impact
Citizens
recognize and
act to ensure
community
wide support
and
involvement in
school, after
school and
youth/family
programs
After school
program is
expanded and
sustained
Seacoast Youth Leadership
Program
This project will evaluate the efficacy of the youth diversion
program’s ability to achieve improvements in the following
objectives:






Leadership
Future Aspirations/Goal Setting
Social Responsibility
Decision Making/Social Skills
Social Connectedness
Media Literacy
Hillsboro After school Program
This project will evaluate the efficacy of the after
school program’s ability to achieve improvements in
the following objectives:








Academic Performance
Future Aspirations
Social Connectedness
Social Responsibility
Sense of Self
Parenting Efficacy
Social Skills/Decision Making
Community Involvement with Youth/School
Collapsing extensive outcome
goals and objectives into
manageable and meaningful
evaluation research conceptual
categories for design and
measurement development
Identifying existing measures for
conceptual categories
• LEADERSHIP
•INDEPENDENCE (WRIGHTSMAN, 1964,1974)
•MASTERY (PEARLIN, 1981)
•SELF-ESTEEM (ROSENBERG, 1965)
•FUTURE ASPIRATIONS (PEER LEADER SURVEY & DYS)
•SOCIAL CONNECTEDNESS
•SOCIAL SUPPORT (VAUX)
•ATTACHMENT TO PARENTS & SCHOOL
(RESNICK, 1997; ADD HEALTH SURVEY)
Collaboration with Extension staff
to determine appropriate evaluation
design and measurement
•EVALUATION AS A PROCESS
•COMMUNICATION IS KEY
Utilization of triangulation of
methods to provide depth and
breadth to the evaluation research
process
Contact Information:
Scott R. Meyer, MSW, Ph.D - (603) 535-2377 ~ [email protected]
Stephanie Halter,M.A., Research Associate, (603) 536-5725 ~ [email protected]
Rick Alleva, Extension Educator, (603) 679-5616 ~ [email protected]
Dan Riedy, Extension Educator, (603) 641-6060 ~ [email protected]
Penny Turner, Extension Educator, (603) 641-6060, [email protected]
Lisa Townson, Extension Specialist, (603) 862-1031 ~ [email protected]