Transcript Source - Future Fund Evaluation Workshop

What Works, and Why:
Issues and Methods in Evaluation in
the Non-Profit Sector
Edward T. Jackson
Carleton University, Ottawa
Presented to the
Future Fund Evaluation Workshop,
Ontario Trillium Foundation, Toronto
April 28, 2010
Overview
•
•
•
•
•
•
•
Context
Issues in Evaluation
Results-Based Management
Participatory Strategies
Social Return on Investment
Next Steps
Resources
2
Context I: Ontario Economy – Challenges
• Restructuring driven by globalization (BRICs),
•
•
•
•
high dollar
Shrinking of the manufacturing sector
A return to a branch-plant, staples-driven
economy
Reduced number of Ontario multi-nationals
Decrease in corporate and individual tax
revenues to governments
3
Context II: Ontario Economy –
Opportunities
• Enabling New Ontarians, Aboriginal Ontarians and young
people to fill upcoming labour shortages in all sectors
• Accelerating the development of good jobs in the green
economy (i.e. energy, water, etc)
• Building new, large-scale Ontario-based anchor companies
in key sectors with a focus on value-added production,
global markets and local suppliers
• Levering social innovation-mobilizing technology, ingenuity
and entrepreneurship to solve pressing social problems
• Exporting Ontario’s expertise in higher education
management, education and research
• Optimizing the role of and linkages with non-profit
organizations and social enterprises in these initiatives
4
Context III: OTF’s Response
• Funding Priority: “Enhanced employment and economic
potential”
• Future Fund: “To support initiatives that create
significant and sustainable opportunities for
communities, networks and individuals to participate
fully in Ontario’s labour market… to break down barriers
to employment and self-employment”
• Implication: To rapidly convert innovation into largescale implementation.
5
Context IV: OTF Evaluation
• Main Purpose: “To measure the success of a project or
•
•
•
•
program by the end of the grant”
Learning: What works, what doesn’t and why?
Meeting Needs: Are the needs of the target population
being met? If not, how can they be better met?
Replication: What are the key components of the
intervention worth replicating?
Accountability: To what extent is the project or program
efficient and effective?
6
Context: OTF Approach to Measuring
Results
• Results should be specific, concrete and
measurable, of short-term or long-term
duration, and should be based on qualitative
or quantitative data, or both.
• Results should be measured at the following
levels:
–
–
–
Individuals
Organizations
Communities
7
Key Issues in Evaluation
• Who will use the evaluation, and how will they use it?
• How feasible is it to actually achieve the proposed results (are
•
•
•
•
they realistic or overstated/hyped)?
What are the different expectations among stakeholders as to
what constitutes a credible evidence base for assessing
effectiveness?
What are the costs – in terms of money and time – of
conducting the evaluation?
To what extent are there asymmetries in stakeholder
participation in the design and implementation of the
evaluation?
What are the real prospects and mechanisms for downstream
replication and scaling-up?
8
Results-Based Management
•
•
•
•
•
Results-Based Management
Results Chain
Impact Value Chain
Results Grid
Case Study – Alterna Micro-Loan Fund
Evaluation
9
Results-Based Management
• “A life-cycle approach to management that
integrates strategy, people, resources,
processes and measurement to improve
decision-making, transparency, and
accountability. The approach focuses on
achieving outcomes, implementing
performance measurement, learning and
changing, and reporting performance” (CIDA)
10
Results Chain
• “A depiction of the causal or logical
relationships between inputs, activities,
outputs and outcomes of a given policy,
program, or initiative” (CIDA, OECD)
11
Results Chain
Inputs
Activities
Outputs
Immediate
Outcomes
(Short-Term)
Intermediate
Outcomes
(MediumTerm)
Ultimate
Outcomes
(Long-Term)
12
Impact Value Chain
Impact = Measure of Change
Inputs
Activities
What you
put in
What you do
•
•
•
•
•
•
•
•
•
•
Funds
Time
Expertise
Networks
Influence
Grantmaking
Programs
Information
Sharing
Investing
Outputs
Outcomes
Results that
are measured
Collection of all
results intended
and/or unintended
Activity results
(e.g. number
of people
served)
—MINUS—
Goal
Alignment
How well
outcomes align
with intended
goals
What would have
happened anyway,
or next best
alternative
(results “but for”
the intervention)
= IMPACT
(Source: Goedeke and Pomares, 2009)
13
Results Grid
Level
Immediate
Outcomes
Intermediate
Outcomes
Ultimate
Outcomes
(Impacts)
Macro-Level
(Policy)
Meso-Level
(Institution)
Micro-Level
(Local
Organizations,
Households,
Individuals)
(Source: Jackson, 2010)
14
Case Study: Logical Framework for the Alterna Savings
Community Micro-Loan Program
Mission Statement: To provide a compelling financial alternative to Canadians
through a unique banking philosophy based on cooperative values and focused on professional financial
counsel
Program
Components
Inputs
Activities
Outputs
Intermediate
Outcomes
Small
Business
Development
loans
Alterna’s
Capital
(Credit Union
members’
shares)
1 Full-Time
Community
Micro-Loan
Manager +
1 Full-Time
CSR Manager
Provision of
small loans
(maximum
$15,000 at
prime rate +
6%) upon
approval of
loan requests
submitted by
interested
applicants
Loans
disbursed to
qualified
applicants in a
timely fashion
(e.g. within two
weeks from
the applicants’
initial request)
Start-up or
expansion of
small businesses
(including
purchase of
productive assets,
workspace rental,
e-commerce
development)
Free financial
literacy
programs and
professional
development
opportunities
Volunteer
work by both
Alterna Staff
and
recognized
local business
professionals/
Instructors
-
-
-
Networking
Café
conducted as
planned
-
Members’
enhanced
understanding
and adoption of
effective business
practices (e.g.
business plan
development,
accounting,
marketing)
D
evelopment of
partnerships
among borrowers
in different fields
(Source: Tarsilla, 2010)
Business
financial
literacy
programs
offered to both
members and
prospective
applicants
Member
Display Booth
and
Information
Boards at
Alterna
Presentations
made by
Alterna CSR
Officers at
local business
development
centres and
training
colleges
Final
Outcomes
Impacts
Micro-Level (Borrowers)
Meso-Level (Corporate)
Macro-Level (Policy)
15
Case Study: Results Grid for the Alterna Micro-Loan Program:
Outcome
Evaluation Layers
Final Outcomes
Impacts
Micro-Level
Increase in Borrowers’ productivity and personal assets
Increased borrowers’ payment of federal
and provincial income taxes
(Borrowers)
Encouragement of borrowers’ self-sufficiency
Reduced number of borrowers relying on
government assistance
Increase in borrowers’ annual income
Enhanced well-being of borrowers and
their households (including housing,
nutrition, health)
Increased free media-coverage of corporate CSR as well as
of their services/products at large; boost corporate
membership
Increased corporate reputation within the
community
Development of new products and services specifically
catering to the most marginalized and vulnerable population
groups (e.g. new Canadians, individuals with no or low
credit rating)
Enhanced corporate brand differentiation
Consolidation of Alterna’s membership base (e.g. through
referral) and enhance borrowers’ opening of additional
accounts other than business related (e.g. personal lines of
credit, mortgages).
Increased customer loyalty
Contributions to the health and success of the local
economy; save government a large amount of resources
otherwise allocated to assistance programs; contribute to
federal and provincial revenues due to the borrowers
paying higher income taxes over time.
Promotion of self-employment and
entrepreneurship among
vulnerable population groups in Toronto
Fostering the creation of new jobs (e.g. micro-loan
borrowers hiring employees to run their business).
Contribution to the reduction in the local
unemployment rate
Meso-Level
(Corporate)
Macro-Level
(Policy)
(Source: Tarsilla, 2010)
16
Participatory Strategies
•
•
•
•
•
•
•
•
Participatory monitoring and evaluation
Why is PME important?
Core Principles
Front-end costs
Multi-Stakeholder engagement strategies
Stakeholder network
The PME cycle
Framework for Organizational Self-Assessment
17
Participatory Monitoring and Evaluation:
Definition – PME
• Participatory monitoring and evaluation is an
approach to performance review in which
stakeholders in an intervention (local citizens,
policy makers, funding agencies, and nongovernmental organizations) work together to
decide how to assess progress, conduct data
collection and analysis, and take action on
their findings
(Source: Jackson, 2005)
18
Why is PME Important?
• PME is geared towards not only measuring the
effectiveness of a project, but also towards
building ownership and empowering
beneficiaries; building accountability and
transparency; and taking corrective actions to
improve performance and outcomes
(Source: World Bank, 2002)
19
Core Principles
• Primary stakeholders are active participants –
not just sources of information
• Building capacity of local people to analyze,
reflect and take action
• Catalyzes commitment to taking corrective
measures
(Source: World Bank, 2002)
20
Front-End Costs
• Time-consuming and expensive at the front
end
• Authentic stakeholder engagement takes time
• Additional professional involvement as
facilitators/intermediaries also adds costs
(Source: Jackson, 2000)
21
Multi-Stakeholder Engagement Strategies
• Evaluation Coordination: Evaluation
committee, working group; evaluation team
• Methods:
–
–
–
Qualitative: Workshops, community meetings, participant
observation, online collaboration, interviews,
questionnaires, video/film/theatre, other visualization tools
Quantitative: Large-scale surveys, cost-benefit analysis,
statistical analysis
Other: Mapping, GPS, corporate/legal research, scientific
monitoring (e.g. water quality)
• Role of Evaluators: Catalyst and facilitators;
ensure voice, choice; do no harm
22
Stakeholder Network
Taxpayer*/
Donor*
Government
Agency/
Foundation
Non-Profits
Consultants
Implementing
Agency
Non-Profits
Consultants
Group/
Enterprise
Individual*
Spouse
Children
*The most results-oriented stakeholders
(Source: Jackson, 1998)
Other Family
Members
23
The PME Cycle
10
Taking Action on the Findings
9
Sharing the Findings
8
Preparing the Report
7
Analyzing the Findings
6
Gathering Relevant Data
1
Deciding Who Participates
2
Forming a Multi-Stakeholder
Vehicle to Lead the Process
3
Establishing Goals for
the PME Exercise
4
Identifying Key
Performance Issues
5
Developing Results Indicators
24
Framework for Organizational
Self-Assessment
(Sources: International Development Research Centre/Universalia)
25
Social Return on Investment
• Social Return on Investment: Definition
• Case Study: Turnaround Couriers
• Expanded Value-Added Statement
• Case Study: PARO Women’s Centre
26
Social Return on Investment: Definition
• Discounted, monetized valuation of the social
value that has been created, relative to the
value of the investment
• Key leaders in SROI include the Roberts
Enterprise Development Fund, Jed Emerson,
New Economics Foundation, Social Capital
Partners, and the Social Economy Centre at
OISE/UToronto
• Receiving greater attention, but SROI can be
resource-intensive and difficult to replicate
27
SROI Snapshot: Turnaround Couriers - Theory of
Change
Goals
• Hire couriers and office
administrative staff from
disadvantaged youth
population
• Provide transitional work
experience to enable youth to
develop employability skills, a
resume and a support
network
• Enable youth to access the
mainstream job market
• Enable youth to stabilize life
situation, begin a career path
and leave the shelter system
(Source: Social Capital Partners, 2009)
Methods
• Recruit youth from youth
shelters and youth serving
agencies across Toronto
• Provide a real job,
not a job training
experience
• Establish a
supportive
management
environment
• Assist youth with planning
and making next steps
regarding housing and
employment
Success Metrics
• Youth are able to
get out of shelter
system and into
independent
housing
• Youth meet or
exceed job
expectations
• TurnAround helps
youth secure next
job and establish a
career path
• Youth are able to
get off and stay off
government
financial assistance
28
SROI Snapshot: Turn Around Couriers (Extract)
Overview of Target Population (sample)
• 38% recruited directly from shelters
• 23% female
• Average age: 21
• 100% unemployed at time of hire
• 54% receiving social assistance at hire
• 54% been involved with justice system
• 54% did not complete high school
Sustainable Livelihood Outcomes (sample)
• 89 youth in total have been hired over 5
years
Avg Change in Societal Contribution
(Target Employees):
$9,391
Average Number of Target Employees:
10
Current Year Cost Savings to Society:
$93,910
Cumulative Cost Savings (prior to Y5):
$191,170
Total Cost Savings to Date:
$285,080
Cumulative Societal Payback Period:
1.8 years
Cumulative SROI:
285%
Note: initial SCP investment = $100,000
• 100% target population recruited from
shelters able to get out of shelter system
and secure independent housing within 6
months of employment at TAC
Employment Outcomes (sample)
• 85% who relied on income support through
social assistance at time of hire able to get
off and stay off
• 15% moved onto mainstream employment in
window cleaning industry (2)
(Source: Social Capital Partners, 2009)
• Increased target/non-target staff ratio to 83%
• 69% continue to work at TAC (9)
• 8% went on to post secondary education (1)
29
Expanded Value-Added Statement
• An integrated social accounting method that employs
carefully constructed assumptions to quantify, in dollar
terms, the social value that a non-profit or co-operative
produces (e.g. volunteer hours contributions, unpaid
contributions to other stakeholders, etc.)
• Usually applied to a single fiscal year, but can be used
to assess multiple indicators over 10-20 years
• The EVAS integrates the calculations of different types
of social value into the overall financial statements of
the organization being assessed.
30
Case Study: PARO Women’s Centre
• Provides training, mentoring and small loans to women
in northern Ontario, and manages a store that sells
artisnal products
• EVAS, calculated for 2005-2006, found that PARO
created at least $132K from volunteer contributions, the
value of mentorship and consulting by the Executive
Director
• For every dollar of purchases on goods and services,
PARO created more than $2 worth of value added,
including about $.70 worth of social value
(Source: Babcock, 2007)
31
Next Steps
1. Reflect on key issues relating to the
2.
3.
4.
5.
evaluation of your Future Fund grant
Learn more about evaluation strategies and
methods
Work with OTF and other stakeholders to
design an evaluation that is results-oriented,
participatory and measures social return
Build a community of practice on FF
evaluation
Document your experience for others, and
learn from them as you proceed forward.
32
Resources
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Blended Value (blendedvalue.org)
Canadian Evaluation Society (evaluationcanada.ca)
Canadian International Development Agency (acdi-cida.gc.ca)
Carleton Centre for Community Innovation (carleton.ca/3ci)
International Development Research Centre (idrc.ca)
International Program for Development Evaluation Training
(ipdet.org)
Institute of Development Studies, University of Sussex (ids.ac.uk)
New Economics Foundation (neweconomics.org)
Organisation for Economic Co-operation and Development (oecd.org)
Roberts Enterprise Development Foundation (redf.org)
Social Capital Partners (socialcapitalpartners.ca)
Social Economy Centre, OISE/UT (sec.oise.utoronto.ca)
Social Finance (socialfinance.ca)
Treasury Board of Canada Secretariat (tbs-sct.gc.ca)
Universalia (universalia.com)
World Bank (worldbank.org)
33