Transcript The Monitoring and Evaluation of Government Policy

Monitoring and evaluating policy across the spheres
and functions of government
University of Cape Town
Department of Statistical Sciences
Funding was contributed by the Conflict and Governance
Facility, a project of National Treasury, which is funded by
the European Union under the European Programme for
Reconstruction and Development
M&E

Monitoring: A continuing function that uses systematic
collection of data on specified indicators to provide
management/stakeholders with indications of the extent
of progress and achievement of objectives

Evaluation: The systematic and objective assessment
of an ongoing or completed project, programme or
policy, its design, implementation and results. The aim
is to determine the relevance and fulfilment of
objectives, efficiency, effectiveness, impact and
sustainability.
Adapted from OECD; http://www.oecd.org/dataoecd/29/21/2754804.pdf 17 Oct 2005
M&E
Motherhood and apple pie? Is there
anything to discuss?
 Does access to data lead to better
information, lead to better decision
making, lead to better performance?

Evidence of what?
Monitoring involves selecting an indicator that can
provide evidence of progress towards some
objective (value judgements)
Assumptions: causality and the links between
actions and results.
To provide evidence:
(1)Articulate the framework within which
monitoring will occur.
(2)Clarify the logic that will be applied in making
the claim for evidence (of progress).
M&E of Policy in South Africa: Alignment across the
Spheres and Functions of Government
Purpose:




To develop a critical appraisal of the current system of
monitoring and evaluation established by government
around the Programme of Action (POA)
To propose a set of indicators that can be used to
appraise performance.
To investigate the alignment of M&E processes across
national and provincial spheres of government.To investigate the existence of appropriate data sets
that can be used to calibrate indicators.
Goal setting and prioritisation
MCDA
 Problem Structuring : defining the problem
arena, including stakeholders
 Specifying criteria (in M&E these could be
referred to as objectives, goals or
outcomes)
 Development of alternative courses of
action
 Preference modelling
 Assigning weights to the criteria
A strategic framework for M&E

Articulating Goals/ Values/ High level
Objectives (or, put simply “what is the
problem?”)
 Developing a ‘theory of change’ on how
interventions will tackle the problems within the
problem context
 Determining what activities will be undertaken
to address the problems.
 Outcomes: if all interventions are successful
where will we be? The answer is the (medium
to long-term) outcome. NB: Check that
activities
outcomes
objectives/
goals.
Strategic M&E
Problem
Ind
Objectives/
Goals
Inputs
Theory
of
Change
Activities:
Programmes
projects, etc
Outputs
Does outcome
lead to goal being
fulfilled?
Ind
Ind
Ind
Outcome
Ind
High level objectives for the Province
A Home for ALL
To enhance
economic
participation,
shared growth
and
development
To enhance spatial
integration and
provide sustainable
service delivery
To ensure the
integrity of the
physical
environment
To develop
social and
human
capital
To practice
holistic
governance
and
administratn
A set of desired outcomes for each
objective
A Home for ALL
To enhance
economic
participation,
shared growth and
development
To enhance spatial
integration and provide
sustainable service
delivery
Enhanced broad-based economic
participation
Enhanced provincial
economic growth
Pro-poor, shared economic
growth
To ensure the
integrity of the
physical
environment
To develop
social and
human capital
% black ownership of
companies
% real GDPR growth
per capita p.a.
Share of poorest
quintile in provincial
consumption (MDG).
Skills profile which matches the needs
of the economy
Adequate spatial integration and
transport infrastructure to support a
growing and integrated economy
Significantly increased employment
% of Science,
Engineering and
Technology graduates
% of individual income
spent on transport
Unemployment rate
To practice
holistic
governance and
administration
To develop
social and
human capital
High levels of
social capital in
communities
Safe communities
An improved health profile
To ensure the
integrity of the
physical
environment
Improved human capital
and educational status
An effective social safety net
Sustainable access to safe water resources
Sustainable access to safe energy resources
To enhance spatial
integration and provide
sustainable service
delivery
An environment which promotes
sustainable livelihoods
A physical environment which
enhances quality of life
Basic services and adequate
bulk infrastructure which leads
to improved quality of life
Spatially integrated,
sustainable human settlements
To enhance
economic
participation,
shared growth and
development
Significantly increased employment
Adequate spatial integration and transport
infrastructure to support a growing and integrated
economy
Skills profile which matches the needs of the economy
Pro-poor, shared economic growth
Enhanced provincial economic growth
Enhanced broad-based economic participation
A Home for ALL
To practice
holistic
governance and
administration
Government operates in a
holistic manner
Constitutional rights are
protected
Government is effective,
efficient and has good
customer (citizen) relations
There is political stability
and peace
There is democracy and
accountability
There is adherence to the
rule of law
Government is ethical and
free of corruption
National value tree
Build more subsidised
housing, medium density housing closer to
work & provide
serviced stands
Multipurpose centres, e-govt, etc. (49)
Improve state capacity for growth & development (47)
Better interaction between govt& people (10, 50, 71)
Better inter-govt. cooperation
Integrate traditional leadership
Better implementation, monitoring, evaluation
Better access to govt information
Improve provision of information (49)
Strengthen democratic institutions incl. courts
Improve services in government
Create a compassionate govt service (9)
Reduce Corruption (10)
Help consolidate peace,
security & development in
southern Africa, Africa &
globe & Position SA as an
effective global player
Improve local govt capacity (48)
Reduce death from road accidents
Provide efficient, safe road &rail transport (9, 26, 52)
Improve
safety &
security
Economic cooperation in southern Africa
Realise NEPAD, strengthen AU, SADC (62)
Efficient logistics and infrastructure (59)
Strengthen democracy esp. Zimbabwe, DRC, Angola, also Congo,
Swaziland also Tanzania, Kenya, Burundi, Rwanda, Ghana,
Senegal, Ivory Coast, Sudan, Ethiopia, Somalia, Mahgreb) (60, 61)
Cooperation with South
Strengthen SA’s Early Warning Mechanism
Pursue multilateral global approach. Democratise UN, IMF, WB,
etc. Improve relations with China (64)
Cooperation with industrialised nations
Address social roots of criminality
Improve efficiency of justice system (9,55)
Reduce crime & m. death from crime
Promote social
cohesion (14) & a nonracist, non-sexist,
democratic society
Intervene in crime areas-enhance normal communal life (53)
Reduce awaiting trial numbers
Strengthen prosecution & Scorpions, special courts
Improve coordination among relevant agencies
Build community, good citizenship, solidarity
Alleviate
poverty
More, better trained police
Protect borders (illegal immigration, crime, terrorism,
marine resources)
Improve anti-corruption structures
Build a healthier
nation
Expand primary school nutrition progm (43)
Expand primary health care (42)
Comprehensive care & treatment of HIV & AIDS.
Create a single & integrated
economy that benefits all.
Complete tertiary infrastructure improvements (42)
Recruit & retain health personnel
Improve services in health facilities
Reduce preventable causes of death-TB, diabetes, malnutrition,
maternal deaths, HIV/AIDS, malaria (9)
Provide comprehensive social security (10) and ensure people
access them (6)
Improve household & community assets
Continue economic
growth (6, 14, 22)
All children with decent classrooms
Reduce teacher/pupil ratio
Spend in favour of students from poor households
Expand school nutrition program.
Free health service to disabled
Introduce national health insurance
Improve UIF
Ensure people access grants
Allocate more resources to welfare grants for 3 yrs (44)
Provide free basic water & electricity
Speed up delivery of water, sanitation, phones,
health, electricity, housing, education
Provide efficient, safe transport (9,26, 52)
Ensure low interest & inflation rates
Facilitate broad-based BEE
Speed up rural development & urban renewal
Increase public & private of investment (24-26)
Provide the skills re-quired by the economy (27,29,32-36,43,45)
Invest in road, rail & air transport, telecommunications & energy
Facilitate economic activity within the second economy, Pro-mote
involvement of marginalised in economic activity (27-39)
Improve opportunities for self-employment &
Promote sustainable community livelihoods (27)
Promote growth of small & micro-enterprises (30,31)
Facilitate creation of jobs, skills and work experience (27,29,3236,43,45)
Create new jobs
Reduce
unemployment
Ensure involvement of communities in local economic
initiatives
Provide efficient, safe transport (9,26, 52)
Complete land restitution
Continue land reform
Comprehensive assistance to emerging farmers,
expand agricultural support programs (37-39)
Support micro-loan financing
Reform support agencies
Change to financial system and institutions
Skills training
Learnerships
Education system geared for productive work
Encourage labour intensive methods
Deal with negative impact of casualisation
Extend Expanded Public Works Program (28-29)
A society that truly cares & a better life for all (as in the Bill of Rights& Constitution)
Broaden access to
services & constitutional rights and
provide good
governance
A society that truly cares & a better life for all (as in
the Bill of Rights& Constitution)
Create a single &
integrated
economy that
benefits all.
Reduce
unemployment
Build a
healthier
nation
Continue
economic
growth
Promote social
cohesion & a
non-racist, nonsexist, democratic society
Alleviate
poverty
Improve
safety &
security
Help consolidate
peace, security &
development in
southern Africa, Africa &
globe & Position SA as
an effective global player
Broaden access
to services &
constitutional
rights and provide
good governance
Provide efficient, safe transport
Improve opportunities
for self-employment &
Promote sustainable
community livelihoods
Ensure involvement of
communities in local economic
initiatives
Comprehensive assistance to
emerging farmers, expand
agricultural support programs
Continue land reform
Promote growth of small
& micro-enterprises
Complete land restitution
Change to financial system and
institutions
Reform support agencies
Facilitate creation of
jobs, skills and work
experience
Support micro-loan financing
Education system geared for
productive work
Learnerships
Create new jobs
Skills training
Encourage labour intensive
methods
Deal with negative impact of
casualisation
Extend Expanded Public Works
Program
A society that truly cares & a better life for all (as in the
Bill of Rights & Constitution)
Create a single &
integrated
economy that
benefits all.
Reduce
unemployment
Facilitate economic
activity within the
second economy, Promote involvement of
marginalised in
economic activity
A society that truly cares & a better life for all (as in the
Bill of Rights& Constitution)
Build a
healthier
nation
Reduce preventable causes of
death-TB, diabetes,
malnutrition, maternal deaths,
HIV/AIDS, malaria
Comprehensive care & treatment
of HIV & AIDS.
Recruit & retain
health personnel
Expand primary health care
Improve services in
health facilities
Expand primary school nutrition
program
Complete tertiary infrastructure improvements
A society that truly cares & a better life for all (as in the
Bill of Rights& Constitution)
Broaden access to
services & constitutional
rights and provide good
governance
Improve
state
capacity
for growth
& development
Better
Better inter- Integrate
interaction govt.
traditional
between
cooperation leadership
govt &
people
Better
implementation,
monitoring,
evaluation
Better
Strengthen Improve Create a
Reduce
Improve
access to provision of democratic services compassio- corruption
govt
information institutions in govt nate govt
information
incl. courts
service
Improve local govt
capacity
Multipurpose centres,
e-govt, etc.
A society that truly cares & a better life for
all (as in the Bill of Rights & Constitution)
Create a single &
integrated economy
that benefits all.
Improve household &
community assets
Speed up delivery of
water, sanitation,
phones, health,
electricity, housing,
education
Comprehensive
assistance to
emerging farmers,
expand agricultural
support programs
Continue land reform
Complete land
restitution
Improve opportunities
for self-employment &
Promote sustainable
community livelihoods
Alleviate
poverty
Provide free basic
water & electricity
Reduce
unemployment
(e) Proportion of HH with access
to secure tenure (owned or
rented) (Target 10 Indicator 32)
(a) Proportion of population with access to an improved water source (both
urban and rural) (Target 10 Indicator 30)
(b) Proportion of population with access to improved sanitation (Target 10
Indicator 31)
(c) % of gdp spent on sanitation infrastructure
(d) Proportion of households with access to electricity
Links to GW M&E system
Build more
subsidised housing,
medium density
housing closer to
work& provide
serviced stands
The following is the heading and
definition for ALL of the indicators
mentioned on this page:
Housing: These indicators measure the
marginality of human living conditions
(a) Percentage of population housed in informal/formal
dwellings.
(b) Percentage of the population which is homeless.
Build more subsidised
housing, medium density housing closer to
work & provide
serviced stands
Expand primary school nutrition progm (43)
Expand primary health care (42)
Comprehensive care & treatment of HIV & AIDS.
Create a single & integrated
economy that benefits all.
Complete tertiary infrastructure improvements (42)
Recruit & retain health personnel
Improve services in health facilities
Reduce preventable causes of death-TB, diabetes, malnutrition,
maternal deaths, HIV/AIDS, malaria (9)
Provide comprehensive social security (10) and ensure people
access them (6)
Improve household & community assets
Continue economic
growth (6, 14, 22)
All children with decent classrooms
Reduce teacher/pupil ratio
Spend in favour of students from poor households
Expand school nutrition program.
Free health service to disabled
Introduce national health insurance
Improve UIF
Ensure people access grants
Allocate more resources to welfare grants for 3 yrs (44)
Provide free basic water & electricity
Speed up delivery of water, sanitation, phones,
health, electricity, housing, education
Provide efficient, safe transport (9,26, 52)
Ensure low interest & inflation rates
Facilitate broad-based BEE
Speed up rural development & urban renewal
Increase public & private of investment (24-26)
Provide the skills re-quired by the economy (27,29,32-36,43,45)
Invest in road, rail & air transport, telecommunications & energy
Facilitate economic activity within the second economy, Pro-mote
involvement of marginalised in economic activity (27-39)
Improve opportunities for self-employment &
Promote sustainable community livelihoods (27)
Promote growth of small & micro-enterprises (30,31)
Facilitate creation of jobs, skills and work experience (27,29,3236,43,45)
Create new jobs
Reduce
unemployment
Ensure involvement of communities in local economic
initiatives
Provide efficient, safe transport (9,26, 52)
Complete land restitution
Continue land reform
Comprehensive assistance to emerging farmers,
expand agricultural support programs (37-39)
Support micro-loan financing
Reform support agencies
Change to financial system and institutions
Skills training
Learnerships
Education system geared for productive work
Encourage labour intensive methods
Deal with negative impact of casualisation
Extend Expanded Public Works Program (28-29)
A society that truly cares & a better life for all (as enshrined in the Bill of Rights, constitution)
Build a healthier
nation
Alleviate
poverty
Measurement issues
Simple vs composite indicators
 Parsimonious models
 Robust or sensitive to change
 Local or global views
 Error (measurement, sampling, statistical
bias, interpretation)
 Measuring (step 1) and valuing (step 2)

Some conclusions…






Pervasive lack of clarity on high level
objectives… value trees might be a useful tool
Priority. What’s that?
Danger of over-engineering
Danger of separating M&E from strategic
planning (activities may not be aligned with
desired outcomes)
Tremendous benefits of awareness created by
MTEF…. but
Lack of quantitative/ problem solving/ modeling
skills
Strategic M&E
There is a strong need for a common
strategic planning process which is
individually tailored by each department/
govt entity within the context of clear
national, provincial, local objectives.