Transcript 4.-How-is-a-policy-supposed-to-workx

BCURE Evidence-Informed Decision-Making Capacity Building Workshop
1st and 2nd June 2015
Pretoria, South Africa
How is a Policy Supposed to Work? –
Theory of Change Analysis
Philip Davies
International Initiative for Impact Evaluation [3ie]
Philip Davies
www.3ieimpact.org
Some Features of Evidence
• Evidence is information or data that supports, or
rejects, a conclusion
• Evidence is any thing that increases the estimate of
the probability of the truthfulness of a proposition
• Evidence is particularistic and generalisable
• Often disagreement agreement on what counts as
‘evidence’
• Evidence is always contestable/contested
• Not all research is of equal value/sufficient quality
• Single studies can misrepresent the balance of
evidence
• Hence, the need for systematic reviews/synthesis of
evidence
Philip Davies
www.3ieimpact.org
Quantitative Evidence
• Social Surveys – cross sectional and time series
• Longitudinal Studies – cohort and panel data
• Experimental Designs (random allocation)
• Quasi-Experimental Designs (Matched samples,
Interrupted Time Series, Regression analysis)
• Economic Methods - Cost-Benefit and CostEffectiveness Analysis
• Statistical / Econometric Modelling
Philip Davies
www.3ieimpact.org
Qualitative Evidence
• In-Depth interviews
• Focus Groups
• Other Consultative Designs
• Observational and Participant-Observational
Studies
• Ethnography
• Documentary Analysis
• Oral Histories
• Case Studies
Philip Davies
www.3ieimpact.org
Theory of Change/Logic Model/Programme Theory
• How is a policy/programme supposed to work?
• What activities, mechanisms, people, outputs
have to be in place?
• And in what sequence – what is the causal
chain?
• What resources are required – and are
available?
• What data are required – and are available?
• Is the policy/programme feasible/achievable?
Philip Davies
www.3ieimpact.org
Building a Theory of Change:
From Inputs to Outcomes
Philip Davies
www.3ieimpact.org
Constituent Features of a Theory of Change
Assumptions?
Philip Davies
www.3ieimpact.org
Constituent Features of a Theory of Change
Data Required •Counterfactual data
•Administrative data
•Survey data, statistics
•Cost/benefit data
•Administrative data
•Performance data
•Qualitative data
•Administrative data
•Performance data
•Costs/benefits data
•Surveys, statistics,
demographic data
•Qualitative data
•Costs/benefits data
•Systematic review data
•Documentary
Philip Davies
analysis
•Performance data
•Historical data
•Diversity data
•Qualitative data
•Effectiveness data
www.3ieimpact.org
•Performance data
•Effectiveness data
•Stakeholder data
•Qualitative data
•Costs/benefits data
•Stakeholder data
•Qualitative data
•Public opinion data
•Effectiveness data
Philip Davies
www.3ieimpact.org
Different Levels of Intervention
Philip Davies
www.3ieimpact.org
Macro Level Environment (Policy)
Government
Industry
Tax and Price
Legislation
Licensing
Advertising
(Availability)
Health Systems
Obesity/H
ealth
Eating
Physical Environment
Home
Bars/Pubs
Availability
Access
Supermarkets
Role Modeling
Peers
Cognition
Biological
Demographic
Philip Davies
Obesity/H
ealth
Eating
Social Environment
Family
Friends
Opportunities
Social
Support
Social Norms
Obesity/H
ealth
Eating
Individual Level
Lifestyle
Skills and
Behaviours
www.3ieimpact.org
Coping Ability
Obesity/H
ealth
Eating
Establishing the Policy Logic/Theory of Change
Basic Principles
• Map out the causal chain
• Test the underlying assumptions with evidence
• Understand context and culture
• Anticipate heterogeneity (variance) across the
population
• Rigorous evaluation of impact using an
appropriate counterfactual
• Use mixed methods of evidence gathering
Philip Davies
www.3ieimpact.org
Group Exercise
In groups, take a policy issue of your choice and:
• Map out how the policy is supposed to work, identifying:
i. What are the outputs and outcomes to be achieved?
ii. What people/agencies are likely to be involved?
iii.What mechanisms (training, planning, zoning,
financing, communications, delivery etc) are likely to
be involved?
iv.What time will be required to achieve the above
factors?
v.What are the likely costs involved, and where should
they fall?
Philip Davies
www.3ieimpact.org
Thank you
Philip Davies on
Email: [email protected]
+44 (0)207 958 8350
Visit www.3ieimpact.org
Philip Davies
www.3ieimpact.org