Transcript Indicator - Mosaic.net International, Inc.

Managing for Change:
The Art of Results Based
Management
Module 3 - Risk and Performance
Measurement
Mosaic Workshop - Ottawa
July 14th - 18th, 2008
Enablers and Constraints…
• Are forces affecting
any part of your
project both
positively and
negatively
Why Identify Assumptions?
• Programs/projects are implemented in a
dynamic environment
• Factors external to the project are often
the cause of failure
• Make explicit the conditions external to
the program/project upon which its
internal logic is dependent
Test Your Design Logic
OUTCOME
Participating citizens
increasingly challenge
intolerance, racism and
hatred in their
communities.
OUTPUT
Individuals and groups
gain awareness of
racism and
discrimination.
Internal Logic
What conditions are
you assuming will
exist in the
operation’s
environment?
External Logic
Risk
•
•
Risk is the uncertainty that surrounds
project results.
It is the likelihood and impact of an
event with the potential to influence
the achievement of results.
Risk Analysis Matrix
Project Example…
Internal Staff Training on Public
Consultation Practice
Outcome:
Local Government Authority
staff consistently apply public
consultation guidelines in the
implementation of their unit
responsibilities; they do so
with a solid grasp of the
subject matter and with a
command of the leading
techniques of engagement
1. What could hamper
progress?
2. What is the likelihood of a
problem?
3. What effect would the
problem have on the
Outcome?
Effects
Significant
(3)
Moderate
(2)
Minor
(1)
Monitor Risks
Risk Mitigation
Required
Risk Mitigation
Required
Acceptable
Risks
Monitor Risks
Risk Mitigation
Required
Acceptable
Risks
Acceptable
Risks
Monitor Risks
Low (1)
Medium (2)
High (3)
Likelihood
Risk Mitigation & Monitoring
•
•
Risk mitigation strategies should reduce
the likelihood of an undesirable event, or
minimize the effect the event could have
on the project.
Risk areas that remain above the level
of acceptable risk should be monitored.
Workshop Exercise - Risk
Management
1.
2.
3.
4.
5.
Develop your matrix
Pick one output and one outcome.
Define your assumptions.
Define your risks and and conduct risk analysis
Consider mitigation and monitoring strategies
Assumptions
Assumptions, Risk Identification and Mitigation Strategy Template
Risks
Risk
Ranking/Rating
Likelihood/Effect
(L/M/H) (L/M/H)
Mitigation Strategy
k Identification and Mitigation Strategy Template
cted Result
er LFA)
ut:
omes:
ct:
Assumptions
Risks
Risk
Ranking/Rating
Likelihood/Effect
(L/M/H) (L/M/H)
Mitigation Strategy
Risk Identification and Mitigation Strategy Template
Expected Result
(as per LFA)
Output:
Outcomes:
Impact:
Assumptions
Risk
Ranking/Rating
Likelihood/Effect
(L/M/H) (L/M/H)
Mitigation Strategy
Keeping Track of Results
• Shifting from the design and planning side of
RBM to information gathering
Monitoring and Evaluation
- The Distinction
•
Monitoring
A continuous self-assessment of progress -collecting data
and comparing current performance with planned activities
and results.
•
Evaluation
Independent (often external), periodic, strategically focused
assessment of a project’s continuing relevance, management,
governance, results, coverage, external relationships,
sustainability, etc.
Keeping Track of Results…
• Means paying attention to …
– Indicators
– Sources of Information
– Methods and Frequency
– Person(s) to do the Work
– Who needs to know
Indicators…
• Point to information you
need to determine if the
project is creating the
results you want.
• They can be both
quantitative and
qualitative.
• They should help the
project make wise
management decisions.
• They should be cost
effective to use.
Quantitative Indicators
Expressed as a percentage, ratio, proportion, comparison, or number
that can be analyzed statistically
Examples?
–Number of households represented on Village
Development Committees
–Ratio of men to women in senior management roles
in the partner organization
–Proportion of households using servicable latrines
–Number of community organizations committing
resources to address hate crimes in ‘x’ community
Quantitative Indicators Your Turn
Climate Change Conference –
Stakeholder Dialogue on
Provincial Action Plan
Outcome - Public discourse on
the Province’s Climate Change
Action Plan Update peaks
through community organized
events
?
Qualitative Indicators
Expressed as a change or comparison between two states of
understanding or experience that can be analyzed for patterns
Relies on people’s judgment or perception
Examples:
–Villager (men and women) stories of how their lives
have changed following completion literacy training
–Change in the level of rapport between local
government and higher levels of government.
–Change in community perceptions of the role/efficacy
of Village Development Committees
–Before-after comparison in the coverage of race
relations issues in the local media
Qualitative Indicators Your Turn
Climate Change Conference –
Stakeholder Dialogue on
Provincial Action Plan
Outcome - Public discourse on
the Province’s Climate Change
Action Plan Update peaks
through community organized
events
?
Indicators, Baseline, and
Target
Meaning
Example
Indicator
The information to be
collected
Proportion of school
children routinely washing
their hands after using the
latrine/washroom
Baseline
What the indicator would
tell you at the beginning
1 in 5
Target
What you want the
indicator to tell you at a
specific point in the future
4 in 5
Qualitative Indicators - Setting
out Markers of Progress
• Sometimes you cannot meaningfully track
progress by counting “units” at the beginning
(baseline), middle and end (e.g. ratio of local
government units/service areas that have
institutionalized public involvement as a way
of working)
• You may need to set out…
– Stages of completion, or
– Markers of progress
Qualitative Indicators - Setting
out Markers of Progress
• Consider the policy formulation…
– Indicator is ‘Comparison of actual to planned progress in
updating ‘x’ policy’
• You could say…
– Baseline - ‘0’ updated policy
– Target - ‘1’ updated policy
• But how useful is that from a management
point of view?
Qualitative Indicators - Setting
out Markers of Progress
•
Would be more useful to identify progress markers.
For example:
a)
b)
c)
d)
e)
f)
•
•
•
Established need for updated policy
Policy research underway
Policy options formulated
Discussion and approvals underway
Updated policy approved
Updated policy enacted
Baseline might be ‘a’
Yr 1 target might be ‘c’
Yr 3 target might ‘e’
Indicator Selection Checklist
•
Validity - Does it measure the result?
Yes
No
•
Is it cost-effective to collect the information?
Yes
No
•
Does it help understand how the program or
project is affecting men and women, and/or
specific sub-groups of people differently?
Yes
No
Yes
No
Yes
No
•
Does it provide useful information for
management decisions?
•
Does it communicate well to stakeholders
(public, policy makers, donors)?
Test Case #1
Indicator:
a)
Change in the average amount of
production lost between field and
market among participating farmers
b)
Proportion of participant farmers
who can describe how they will
replicate handling procedures
following the training
New skills through training - Output
Participant farmers understand post harvest handling
procedures and are willing to try these out on their own after
the training.
•
Validity - Does it measure the result?
Yes
No
•
Is it cost-effective to collect the information?
Yes
No
•
Does it help understand how the program or
project is affecting men and women, and/or
specific sub-groups of people differently?
Yes
No
Yes
No
Yes
No
•
Does it provide useful information for
management decisions?
•
Does it communicate well to stakeholders
(public, policy makers, donors)?
Test Case # 2
Climate Change Conference – Stakeholder
Dialogue on Provincial Action Plan
Outcome - Public discourse on the Province’s
Climate Change Action Plan Update peaks through
community organized events
Indicator:
Pick your favourite from your group’s list
of quantitative and qualitative indicators.
Test it…
•
Validity - Does it measure the result?
Yes
No
•
Is it cost-effective to collect the information?
Yes
No
•
Does it help understand how the program or
project is affecting men and women, and/or
specific sub-groups of people differently?
Yes
No
Yes
No
Yes
No
•
Does it provide useful information for
management decisions?
•
Does it communicate well to stakeholders
(public, policy makers, donors)?
Test Case #3
Household Participation in Community Decisionmaking - Outcome
Households participate more fully in community decisionmaking and improvement initiatives; they help each other
out when in trouble
Indicator: Reported change in the way
households respond to common
hardships (e.g. abusive
relationships, financial shortfalls,
childcare, infirmity)
•
Validity - Does it measure the result?
Yes
No
•
Is it cost-effective to collect the information?
Yes
No
•
Does it help understand how the program or
project is affecting men and women, and/or
specific sub-groups of people differently?
Yes
No
Yes
No
Yes
No
•
Does it provide useful information for
management decisions?
•
Does it communicate well to stakeholders
(public, policy makers, donors)?
Performance Measurement
Plan
Results
Levels
Impact
Outcome
Output
Indicators
Base
-Line
Target
Data Source/ Report
Method
Frequency
& Resp.
Exercise
• Prepare a Matrix of the Performance
Measurement Framework
• Select at least one output and one outcome.
• Brainstorm a total of 3 indicators per result: 1
quantitative and 1 qualitative indicator and 1
other indicator.
• Develop your sources of information,
methods, responsibility and frequency.