Transcript Document

Impact Evaluation Methods:
Causal Inference
Sebastian Martinez
Impact Evaluation Cluster, AFTRL
Slides by Paul J. Gertler & Sebastian Martinez
Motivation
►
►
“Traditional” M&E:
• Is the program being implemented as
designed?
• Could the operations be more efficient?
• Are the benefits getting to those intended?
 Monitoring trends
• Are indicators moving in the right direction?
  NO inherent Causality
Impact Evaluation:
 What was the effect of the program on
outcomes?
 Because of the program, are people better off?
 What would happen if we changed the
program?
  Causality
2
Policy
Intervention
Increase Access and
Quality in Early Child
Education
-Construction
Improve learning in
Science and Math in
high school
-Upgrade
Improve quality of
instruction in higher
education
Monitoring
Impact Evaluation
-New classrooms
-SES of students
- # of Meals
-Use of curriculum
-Increased
attendance
-health/growth
-Cognitive
Development
science
laboratories
-Training of
instructors
- # equipped labs
-# trained instructors
-Lab attendance &
use
-Learning
-Teacher
-
# of training
sessions
- # of internet
terminals
-Learning
-Feeding
-Quality
training
-Online courses
-Labor
market
- University
enrollment
-Attendance/drop
-Labor
out
market
3
Motivation
Objective in evaluation is to estimate the
CAUSAL effect of intervention X on
outcome Y
 What is the effect of a cash transfer on
household consumption?
► For causal inference we must understand
the data generation process
 For impact evaluation, this means
understanding the behavioral process
that generates the data
• how benefits are assigned
►
4
Causation versus Correlation
►
Recall: correlation is NOT causation
 Necessary but not sufficient condition
 Correlation: X and Y are related
• Change in X is related to a change in
Y
• And….
• A change in Y is related to a change
in X
 Causation – if we change X how much
does Y change
• A change in X is related to a change
in Y
5
• Not necessarily the other way around
Causation versus Correlation
Three criteria for causation:
 Independent variable precedes the
dependent variable.
 Independent variable is related to the
dependent variable.
 There are no third variables that could
explain why the independent variable is
related to the dependent variable
► External validity
 Generalizability: causal inference to
generalize outside the sample
population or setting
►
6
Motivation
The word cause is not in the vocabulary of
standard probability theory.
 Probability theory: two events are
mutually correlated, or dependent  if
we find one, we can expect to encounter
the other.
► Example age and income
► For impact evaluation, we supplement the
language of probability with a vocabulary
for causality.
►
7
Statistical Analysis &
Impact Evaluation
►
►
Statistical analysis: Typically involves inferring
the causal relationship between X and Y from
observational data
 Many challenges & complex statistics
Impact Evaluation:
 Retrospectively:
• same challenges as statistical analysis
 Prospectively:
• we generate the data ourselves through
the program’s design  evaluation
design
• makes things much easier!
8
How to assess impact
►
What is the effect of a cash transfer on
household consumption?
►
Formally, program impact is:
α = (Y | P=1) - (Y | P=0)
Compare same individual with & without
programs at same point in time
► So what’s the Problem?
►
9
Solving the evaluation
problem
Problem: we never observe the same
individual with and without program at
same point in time
► Need to estimate what would have
happened to the beneficiary if he or she
had not received benefits
► Counterfactual: what would have
happened without the program
► Difference between treated observation
and counterfactual is the estimated impact
►
10
Estimate effect of X on Y
►
Compare same individual with & without treatment
at same point in time (counterfactual):
sick 2 days
sick 10 days
Impact = 2 - 10 = - 8 days sick!
►
Program impact is outcome with program minus
outcome without program
11
Finding a good counterfactual
The treated observation and the
counterfactual:
 have identical factors/characteristics,
except for benefiting from the
intervention
 No other explanations for differences in
outcomes between the treated
observation and counterfactual
► The only reason for the difference in
outcomes is due to the intervention
►
12
Measuring Impact
Tool belt of Impact Evaluation Design
Options:
► Randomized Experiments
► Quasi-experiments
 Regression Discontinuity
 Difference in difference – panel data
 Other (using Instrumental Variables,
matching, etc)
► In all cases, these will involve knowing the
rule for assigning treatment
13
Choosing your design
For impact evaluation, we will identify the
“best” possible design given the
operational context
► Best possible design is the one that has
the fewest risks for contamination
 Omitted Variables (biased estimates)
 Selection (results not generalizable)
►
14
Case Study
Effect of cash transfers on consumption
► Estimate impact of cash transfer on
consumption per capita
 Make sure:
• Cash transfer comes before change in
consumption
• Cash transfer is correlated with
consumption
• Cash transfer is the only thing
changing consumption
► Example based on Oportunidades
15
►
Oportunidades
►
National anti-poverty program in Mexico (1997)
►
Cash transfers and in-kind benefits conditional on
school attendance and health care visits.
►
Transfer given preferably to mother of beneficiary
children.
►
Large program with large transfers:
 5 million beneficiary households in 2004
 Large transfers, capped at:
• $95 USD for HH with children through junior
high
• $159 USD for HH with children in high
school
16
Oportunidades Evaluation
►
►
Phasing in of intervention
 50,000 eligible rural communities
 Random sample of of 506 eligible communities
in 7 states - evaluation sample
Random assignment of benefits by community:
 320 treatment communities (14,446
households)
• First transfers distributed April 1998
 186 control communities (9,630 households)
• First transfers November 1999
17
Oportunidades Example
18
Common Counterfeit
Counterfactuals
1. Before and After:
2005
Sick 2 days
2007
Sick 15 days
Impact = 15 - 2 = 13 more days sick?
2. Enrolled /
Not Enrolled:
Sick 2 days
Sick 1 day
Impact = 2 - 1 = + 1 day sick?
19
“Counterfeit” Counterfactual
Number 1
►
Before and after:
 Assume we have data on
• Treatment households before the cash
transfer
• Treatment households after the cash
transfer
 Estimate “impact” of cash transfer on
household consumption:
• Compare consumption per capita before the
intervention to consumption per capita after
the intervention
• Difference in consumption per capita
between the two periods is “treatment”
20
Case 1: Before and After
►
Compare Y before and
CPC
after intervention
αi = (CPCit | T=1) (CPCi,t-1| T=0)
Before
After
A
Estimate of
counterfactual
(CPCi,t| T=0) = (CPCi,t-1|
T=0)
►
►
“Impact” = A-B
B
t-1
t
Time
21
Case 1: Before and After
Mean
Case 1 - Before and After
Control - Before Treatment - After
233.48
268.75
t-stat
16.3
Case 1 - Before and After
Multivariate Linear Regression
Linear Regression
Estimated Impact on CPC
35.27**
34.28**
(2.16)
(2.11)
** Significant at 1% level
22
Case 1: Before and After
►
Compare Y before and
CPC
after intervention
αi = (CPCit | T=1) - (CPCi,t-1| T=0)
►
Estimate of counterfactual
Before
After
(CPCi,t| T=0) = (CPCi,t-1| T=0)
A
►
►
“Impact” = A-B
Does not control for time
varying factors
 Recession: Impact = AC
 Boom: Impact = A-D
D?
B
C?
t-1
t
Time
23
“Counterfeit” Counterfactual
Number 2
►
Enrolled/Not Enrolled
 Voluntary Inscription to the program
 Assume we have a cross-section of postintervention data on:
• Households that did not enroll
• Households that enrolled
 Estimate “impact” of cash transfer on
household consumption:
• Compare consumption per capita of those
who did not enroll to consumption per capita
of those who enrolled
• Difference in consumption per capita
between the two groups is “treatment”
24
Case 2: Enrolled/Not Enrolled
Case 2 - Enrolled/Not Enrolled
Not Enrolled Enrolled t-stat
Mean CPC
290.16
268.7541 5.6
Case 2 - Enrolled/Not Enrolled
Linear Regression
Multivariate Linear Regression
Estimated Impact on CPC
-22.7**
-4.15
(3.78)
(4.05)
** Significant at 1% level
25
Those who did not enroll….
►
Impact estimate: αi = (Yit | P=1) - (Yj,t| P=0) ,
►
Counterfactual:
►
Examples:
 Those who choose not to enroll in program
 Those who were not offered the program
• Conditional Cash Transfer
• Job Training program
Cannot control for all reasons why some choose to
sign up & other didn’t
Reasons could be correlated with outcomes
We can control for observables…..
But are still left with the unobservables
►
►
►
►
(Yj,t| P=0) ≠ (Yi,t| P=0)
26
Impact Evaluation Example:
Two counterfeit counterfactuals
►
What is going on??
Estimated Impact
on CPC
Case 1 - Before and After
Linear
Multivariate Linear
Regression
Regression
Case 2 - Enrolled/Not Enrolled
Linear
Multivariate Linear
Regression
Regression
35.27**
34.28**
-22.7**
-4.15
(2.16)
(2.11)
(3.78)
(4.05)
** Significant at 1% level
►
►
►
Which of these do we believe?
Problem with Before-After:
 Can not control for other time-varying factors
Problem with Enrolled-Not Enrolled:
 Do no know why the treated are treated and the
others not
27
Solution to the Counterfeit
Counterfactual
Sick 2 days
Observe Y with treatment
Sick 10 days
ESTIMATE Y without treatment
Impact = 2 - 10 = - 8 days sick!
On AVERAGE,
is a good counterfactual for
28
Possible Solutions…
We need to understand the data
generation process
 How beneficiaries are selected and how
benefits are assigned
► Guarantee comparability of treatment and
control groups, so ONLY difference is the
intervention
►
29
Measuring Impact
Experimental design/randomization
► Quasi-experiments
 Regression Discontinuity
 Double differences (diff in diff)
 Other options
►
30