Transcript Monitoring and Evaluation for CBA Projects

UNDP M&E framework for adaptation,
VRA and IAS
C. O. Nyandiga and A.F Wittmann
Presented at the UNDP CBA- UNV Workshop, August 2-10, Dakar-Senegal


Monitoring = A continuous data collection and analysis process
implemented to assess a project (or program or policy) and compare it
with the expected performance
Evaluation = A systematic and objective measurement of the results
achieved by a project, a program or a policy, in order to assess its
relevance, its coherence, the efficiency of its implementation, its
effectiveness and its impact, as well as its sustainability
(source : World Bank)

Monitoring and Evaluation = a systematic and on-going process,
that is part of project inception, planning and implementation

M&E is necessary to ensure that the OBJECTIVES of the project are met
M&E
 Provides project management and stakeholders with on-going
indications of progress or lack of progress in achievements of objectives
 Helps identifying problems and successes of projects, during
implementation
 Provides a basis for corrective actions
 Helps reinforcement of initial positive results
 Helps in determining if a project is still relevant
ADAPTATION :
What do we want to measure ?
We want to make sure our objectives are achieved : improving the
adaptive capacity / reducing the vulnerability of the communities and
the ecosystems on which they rely.
What can we measure ?
-
-
Coverage : extent to which the project reaches vulnerable stakeholders
(individuals, households, businesses, government agencies,
policymakers…)
Impact : extent to which the project reduces the vulnerability, through
policymaking, capacity building…)
Sustainability : ability of stakeholders to continue the adaptation process
beyond project lifetime
Replicability : extent to which projects generate and disseminate results
and lessons of value for replication
Coverage
 i. Number of households, businesses engaged in vulnerability reduction or
adaptive capacity development activities, as a proportion of households in
the community or region targeted by the project.
 ii. Number of policies introduced or adjusted to incorporate climate change
risks.
 iii. Number of investment decisions revised or made to incorporate climate
change risks.
 iv. Number of stakeholders (individuals, households, communities, etc.)
served by new or expanded climate information management systems (e.g.
early warning systems, forecasting,etc.).
Impact
 i. Percent change in stakeholders’ behaviours utilizing adjusted practices or
resources for managing climate change risks, assessed via QBS.
 ii. Percent improvement in stakeholders’ capacities to manage climate
change (e.g. communicate climate change risks, disseminate information, or
make decisions based on high quality information), as relevant, assessed via
QBS.
 iii. Percent reduction in perceived vulnerability:
 a. Percent improvement in stakeholder perceptions of vulnerability to a
recurrence of primary climate change-related threat(s), assessed via QBS.
Sustainability
 i. Number of beneficiaries of project receiving training in
implementation of specific adaptation measures or decisionsupport tools.
 ii. Local (or spatially appropriate) availability of skills and
resources necessary to continue adaptation after conclusion
of project, assessed via QBS.
 iii. Support for project activities among participating
communities as assessed by QBS.
 iv. Number of outside programmes, policies or projects
incorporating project results into their processes.
Replicability
 i. Number of “lessons learned” from the project


CBA falls within the UNDP Framework for M&E of
Adaptation to Climate Change
The specificity of CBAdaptation = it is a challenge to
monitor and evaluate it, because :




“Adaptation is not generally an outcome, but rather
consists of a diverse suite of ongoing processes that
enable the achievement of development objectives under
climate change”
“Adaptation cuts across many development objectives”
The results of adaptation can be measured in the longterm, whereas projects have a short duration
It is difficult to decouple climate risks from other drivers
(UNDP Framework for Monitoring and Evaluation of ACC)

M&E aims to ensure and show that the CBA
objectives are met :
◦ Building adaptive capacity
◦ Reducing community vulnerability
◦ Increasing the resilience of global environmental
benefits to climate change (including variability) in
the GEF focal areas

M&E applies to the 3 CBA outcomes
◦ Outcome 1 – Local level (projects)
◦ Outcome 2 – National level
◦ Outcome 3 – Global level

4 sets of indicators are monitored
◦ The UNDP Adaptation Indicators
 Adaptive capacities
 Sustainable community management of natural
resources
◦ The SGP’s Impact Assessment System (IAS)
 GEB
 Livelihood / Empowerment
◦ The Vulnerability Reduction Assessment
◦ Volunteerism and Inclusive Participation indicator

4 sets of indicators are monitored
◦ At the project level
◦ Aggregated at the national level
◦ Then again aggregated at the global
level


Adaptive capacity fostered among natural resource dependent communities through awareness
building, enhanced access to climate change and scenario information, and improved access to
alternative resources
◦ 2.1 Number of stakeholders at community level (e.g. businesses, community
representatives, CBOs, NGOs) engaged by project and provided with training in climate
change risk management and scenario planning.
◦ 2.2 Population covered by awareness building programmes to increase understanding of
risks associated with climate change among general public and key stakeholder groups.
◦ 2.3 Increase in awareness of climate change related risks to natural resources (QBS)
◦ 2.4 Percentage change in natural resource dependent population with access to alternative
or supplementary livelihood options (QBS)
Sustainable community management of natural resources in the face of climate change
promoted
◦ 4.1 Percentage of population in relevant areas engaged in sustainable community
management activities
◦ 4.2 Number of measures deployed as part of sustainable resource management activities.
◦ 4.3 Percentage of area of concern in which sustainable resource management activities are
implemented.
◦ 4.4 Number of stakeholders (e.g. families/households) benefiting from sustainable resource
management activities (e.g. in terms of increased income or food security).
◦ 4.5 Success of sustainable resource management interventions in securing livelihoods and
protecting resources (QBS).


Why ?
Measure the Global Environmental Benefits and
the Livelihood / Empowerment benefits
generated by the project
What ?
◦ GEB /
o Biodiversity-(# of species, innovations/new
technologies, local/national policies)
o Land Degradation- (ha. of land restored, land
sustainably managed, tons of soil erosion prevented, #
of innovations/new technologies, local/national
policies)
o Livelihood / Empowerment /
o Poverty Reduction –(# of households or individuals
who benefited from the project, income generation
achieved through the project…)
o Capacity Building –(# of NGOs, community groups
whose capacities were increased, #of women
participating, support ensured from local /
governmental institutions…)

Who, How ? And When ?
At project conception, chose one GEB indicator
and one LIVELIHOOD / EMPOWERMENT indicator
During project development, measure the
baseline values of the indicators and prepare
your monitoring plan (in project proposal) –
During project implementation, measure the
evolutions of the indicators (project reports) –
RESPONSIBILITY OF THE GRANTEE
At the end of the project, measure the final
indicator (final participatory evaluation / final
report) – RESPONSIBILITY OF THE GRANTEE
Monitoring Plan –
-What you will measure
(indicator)
-What is the target value
of your indicator
-How you will measure it
-When you will measure it
-Who will measure it
IAS
Indicator
to be
measured
Nb of ha
of
degraded
land
restored /
combat
land
degradatio
n
Nb
of
NGOs and
Communit
y Groups
participati
ng
/
capacity
building
How it will be
measured
When it will
be measured
Addition of the
surface of all
the pilot sites
that have been
planted,
and
that
have
benefitted from
resilient farming
practices
For all activities,
a
list
of
participants is
made,
desegregating
the
different
community
groups (farmers,
NGO members,
women, elders)
Regular
monitoring
after
completion
of
each
activity + At
the end of
the project
Regular
monitoring
after
completion
of
each
activity + At
the end of
the project
Target
value to be
achieved by
project end
4 ha of
farmland +
10 ha of
forest land
-One NGO
-3
community
groups
(farmers,
women,
youth).
Don’t forget to report the
indicators in the SGP
database
Arid & semi-Arid zone ecosystems
 Local population estimates of arid and semi-arid zone species
 Changes in extent of undegraded arid and semi-arid habitat.
Coastal, Marine and Freshwaters:
 Local population estimates of target aquatic species
 Changes in extent of healthy coral reef
 Changes in extent and quality of mangroves (patch size, maximum tree size).
Forest:
 Local population estimates of forest species
 Changes in natural forest extent
 Number of concessionaries in the formulation of timber extraction
Mountain:
 Local population estimates of montane species
 Changes in extent of undegraded montane habitat
 Rate of deforestation on slopes >20% in project areas
Agrobiodiversity:
 Changes in number of local land races & domestic livestock breeds.
 Changes in extent of agricultural area using low-input high diversity
production methods
 Incorporation of local land races and indigenous breeds in national breeding
programmes
 Certification standards for agricultural products
WHAT
 Systematically capture and document off-site,
indirect and longer-term impacts- “ex-post
evaluation”. Generates GEB Measurements
HOW
 Using proxy and development indicators while
project is ongoing
 Using structured QBS during and after project
closure
 External independent consultants
WHEN
 Minimum 5 years after project completion-Done by
UNDP EO

What ?
The objective of a CBA project is to REDUCE
the vulnerability of the community. VRA is a
tool that contributes to measure achievement
of this objective.
VRA is a form of Participatory Impact
Assessment, which focuses on the
community’s own perception of its
vulnerability and adaptation capacities.

Why ?
◦
◦
◦
◦
◦
Measure the community’s perceptions of current and future
climate change risks and adaptive capacities (now and in the
long-run)
Measure the obstacles to adaptation, and the assets
Evaluate the sustainability of the project
Capture qualitative information / knowledge from the
community, essential for project development/management
(to make sure the project responds to community priorities)
and for knowledge-management
Capture quantitative (thus comparable) information
(=scores) that will allow to verify the achievement of CBA
objectives (at project / national / global levels)

How ? When ?
VRA is composed of 4 indicators, that are transformed into 4 questions (tailored to the
local context).
These questions form the core of a “VRA participatory workshop”, that is organized at
least 3 times in the course of a project, in the community (at the beginning, at midcourse of project implementation, at the end).
UNDP Adaptation Policy
Framework Step
VRA Indicator
VRA Question
In these examples, we consider the case of a
community facing increasing drought risks
Assessing current
vulnerability
1. Vulnerability of livelihood/welfare to
existing climate change and/or climate
variability.
Example: What happens when there is
drought? How does this affect you and your
community?
Assessing future climate
risks
2. Vulnerability of livelihood/welfare to
developing climate change risks.
Example: What would happen if drought was
twice as frequent? How would this affect you
and your community?
Formulating an
adaptation strategy
3. Magnitude of barriers (institutional,
policy, technological, financial, etc)
barriers to adaptation.
Example: What stands in the way of adapting
to increasing drought? What means do you or
your community have to manage events
occurring more frequently?
Continuing the
adaptation process
4. Ability and willingness of the
community to sustain the project
intervention
Example: Rate your confidence that the
(project activity) will continue after the project
period.

The H-Form = the tool that is used for VRA facilitation and
data collection
Where ?
Choose an appropriate location in the village, according to what you already know
about the community.
The location has to be accepted by / accessible to everyone
When ?
Accommodate the community schedule, including women’s schedule
Who ?
Facilitating team :
1 or 2 lead facilitators (with leverage and experience ; who know the local language
and culture)
1 person to take notes on the H-form
1 person to take extensive notes on the side
1 person to take pictures and coordinate the logistics
MOBILIZE the grantee as well as community-members.
BE GENDER-SENSITIVE
QUICK TIPS
-Make sure all the members of the community are invited
and informed about the meeting
-Create a friendly atmosphere, where everyone is at ease
to speak
-Promote open discussion (the participants are the
“teachers”, because we want to learn from them and
capture their perceptions)
-Use a language that is appropriate
-Prepare your H-forms in advance (as well as all the
material you might need)
-Inform the authorities in advance
1-Introduction
Present the facilitating team
Explain the context of the workshop (CBA project) and the
objectives of the workshop
Explain the “rules” : promote open participation (there is no
wrong / right answer; everyone is allowed to speak…)
2-Launch a general discussion about the local environment and
climate
Use simple questions and tackle climate aspects one by one
what is the “usual” climate here ?
when does it rain ? when is it hot ? For how long…
Are there any storms, droughts ? When ?
you can draft a season calendar for example
3-How does the community experience climate change?
Have you noticed any changes in the past years ?
Has it rained more ? less ? When did it rain last year ? for
how long ?
Is it hotter than in the past ? is the dry season longer ?
Are there more storms etc.
QUICK TIPS
-Adapt to relevant climatic matters
-Be very specific : referring to “climate
change” is too vague in most contexts
-Ask simple questions one by one, so
people have time to discuss them
4-What are the impacts of these changes on your livelihood
= VRA QUESTION 1 / Fill out H-form
5-What will be the impacts if these changes (more storms / more droughts) double in the
future ?
= VRA QUESTION 2 / Fill out H-form
The impacts are noted in 2 columns (positive impacts / negative impacts)
After the discussion, assessment : how serious are these impacts, in the
view of the community
What are the possible solutions to limit these impacts ?
6-What are the obstacles / assets for adaptation ?
= VRA QUESTION 3 / Fill out H-form
What prevents you from implementing the solutions that you mention ?
After the discussion, assessment : are the obstacles stronger than the
assets
What are the possible solutions to overcome the obstacles and build on
the assets ?
7-Do you think that with the project, you will be able to adapt further in the future ?
= VRA QUESTION 4 / Fill out H-form
(Assess the sustainability of the project, and how people will contribute)
After the discussion, assessment : are the participants optimistic about the
outcomes of the projects ? are they willing to actively participate ?
8-Conclusion
Thank people for their participation
Present the next steps of the project
Men

What are the results of a VRA session
◦
◦
QUANTITATIVE DATA = SCORING
Generate a score for each question (noted on 5; 1 shows
that the
vulnerability is high / 5 shows that the
vulnerability is low
Calculate the average from each question
Multiply the total average by 2 to obtain a score noted on
10
QUALITATIVE DATA = all the information captured during the
session, and classified in the different parts of the H-form




Negative impacts
Positive impacts
Ideas, Solutions
How to use the results of a VRA session
◦
The initial VRA results


◦
Are used for project development an incorporated in the project
proposal (provides data for baseline assessment / local
knowledge / and ideas for project outcomes/outputs)
Provide the baseline vulnerability measurement, reference for
project monitoring
The 2nd and 3rd VRA results are to be compared with baseline,
in order to monitor the project.
Women
Q1
1
1
Q2
1
1
Q3
1
1
Q4
2
2
5
5
1,25
1,25
Total
Average (noted /5)
Average (noted /10)
2,5
VRA SCORE (/10)
2,5
2,5
QUICK TIPS
Always make sure you thoroughly
document the VRA sessions
Take detailed notes on your own H-form,
Note your personal side-comments –
how many people ? Where ? When ?
Who facilitated ? What went right / wrong
? What can be improved ? …
Take pictures
Don’t forget to report the indicators in
the SGP database

Lessons learned from implementation
 VRA is a powerful way to capture a lot of information from the community
 It gives an opportunity for mobilization and discussion, and contributes to fostering
community ownership of the project
 Language and communication are critical
 Gender appropriateness is essential (in the choice of the location, scheduling,
language, facilitators) : in Morocco, for example, we do specific women’s workshops
 VRA is a flexible tool : its organization can be tailored to the specific context (for ex.
In Morocco, we don’t always use the big H-form, and we added some
questions/discussions to prepare for the VRA questions) – PROVIDED ALL THE INFO IS
COLLECTED FROM THE DISCUSSION
 VRA can be incorporated in a larger workshop (information about CC, training, rural
assessment…) so that the communities are prepared and so that the discussions are
to the point.
 The main challenge is the QUANTITATIVE DATA (for discussion)
How can they be better understood (scale of value is not necessarily translated in numbers) ?
How can they be better captured (voting/scoring might block participants; people don’t want to
stand out)?
How can they be more reliable for monitoring (composition of the group changes from one VRA to
the next ; participants might over/underestimate the scores in order to ensure the funding… ?
There is a potential for improvement, in order to be able to prove the impacts of
our projects.
BASELINE indicators (project proposal)
- What are the mechanisms for volunteerism that already
exist in the community before the CBA project (for
example, traditional mechanisms for mutual assistance,
associations, etc.)?
- Number of volunteers in the community already
engaged in climate change adaptation activities before
the CBA project
- What are the opportunities or obstacles that could
facilitate or impede people from engaging in voluntary
activities?
The CBA project aims at reinforcing Volunteerism : these
indicators should be monitored over the course of the
project
Be sure to capture quantitative / qualitative data on
community volunteers and participation during the
project : in the activity reports / follow-up on the
Volunteerism chart
Inclusive participation : monitor the vulnerable group’s
participation (esp. women, youth, people with a
handicap, elders, property-less farmers…). In the
activity reports / follow-up on the Volunteerism chart
Project Conception
SGP IAS
Project development / Planning grant
Chose the relevant indicators Collect baseline data
(in concept note)
Incorporate data in project proposal
Report data on SGP database
Project
implementation
Evolution of the
indicators in project
reports (before each
disbursement)
UNDP adaptation Chose the relevant indicators Collect baseline data
(in concept note)
Incorporate data in project proposal
Evolution of the
indicators in project
reports (before each
disbursement)
VRA
Prepare 4 tentative
Organize VRA session and collect
questions, based on the main quantitative and qualitative data
climate risk for the
Incorporate data in project proposal
community
Organize mid-term
VRA session and
collect quantitative
and qualitative data
(include in Mid-Term
report)
Volunteerism
Collect baseline info for
project note
Evolution of the
indicators in project
reports (before each
disbursement)
Participation of
volunteers is
highlighted in activity
reports
National
Portfolio
Collect baseline data
Incorporate data in project proposal
Project closing
Final participatory
project evaluation;
Include final
measurements in
final report and K&M
document
Final participatory
project evaluation;
Include final
measurements in
final report and K&M
document
Organize final VRA
session and collect
quantitative and
qualitative data;
Include data in final
report and K&M
document
Final participatory
project evaluation;
Include final
measurements in
final report and K&M
document
Write case studies / share with PMU and ALM
Fill out SGP database
Quarterly portfolio report (incorporate all the added indicators / measurement from the projects)
Yearly report
Aggregate indicators
Global Portfolio APR / PIR : Annual project report / Project Implementation report



More and more, we are asked to prove our
results.
M&E is ESSENTIAL to show that our projects
achieve their objectives.
This is why we need to assess the challenges
of our system, and try to improve it.




How do you implement our M&E ?
How do you explain it to the proponents /
communities, and support them in
implementing ?
What are the challenges you are facing ?
What do you need to overcome these
challenges ?



Quantitative aspect of VRA ?
Consolidate our 4 set of indicators into 1,
without losing any of the information ?
Incorporate project outcome indicators ?