Transcript Document
Impact Evaluation Methods:
Causal Inference
Sebastian Martinez
Impact Evaluation Cluster, AFTRL
Slides by Paul J. Gertler & Sebastian Martinez
Motivation
►
►
“Traditional” M&E:
• Is the program being implemented as
designed?
• Could the operations be more efficient?
• Are the benefits getting to those intended?
Monitoring trends
• Are indicators moving in the right direction?
NO inherent Causality
Impact Evaluation:
What was the effect of the program on
outcomes?
Because of the program, are people better off?
What would happen if we changed the
program?
Causality
2
Policy
Intervention
Increase Access and
Quality in Early Child
Education
-Construction
Improve learning in
Science and Math in
high school
-Upgrade
Improve quality of
instruction in higher
education
Monitoring
Impact Evaluation
-New classrooms
-SES of students
- # of Meals
-Use of curriculum
-Increased
attendance
-health/growth
-Cognitive
Development
science
laboratories
-Training of
instructors
- # equipped labs
-# trained instructors
-Lab attendance &
use
-Learning
-Teacher
-
# of training
sessions
- # of internet
terminals
-Learning
-Feeding
-Quality
training
-Online courses
-Labor
market
- University
enrollment
-Attendance/drop
-Labor
out
market
3
Motivation
Objective in evaluation is to estimate the
CAUSAL effect of intervention X on
outcome Y
What is the effect of a cash transfer on
household consumption?
► For causal inference we must understand
the data generation process
For impact evaluation, this means
understanding the behavioral process
that generates the data
• how benefits are assigned
►
4
Causation versus Correlation
►
Recall: correlation is NOT causation
Necessary but not sufficient condition
Correlation: X and Y are related
• Change in X is related to a change in
Y
• And….
• A change in Y is related to a change
in X
Causation – if we change X how much
does Y change
• A change in X is related to a change
in Y
5
• Not necessarily the other way around
Causation versus Correlation
Three criteria for causation:
Independent variable precedes the
dependent variable.
Independent variable is related to the
dependent variable.
There are no third variables that could
explain why the independent variable is
related to the dependent variable
► External validity
Generalizability: causal inference to
generalize outside the sample
population or setting
►
6
Motivation
The word cause is not in the vocabulary of
standard probability theory.
Probability theory: two events are
mutually correlated, or dependent if
we find one, we can expect to encounter
the other.
► Example age and income
► For impact evaluation, we supplement the
language of probability with a vocabulary
for causality.
►
7
Statistical Analysis &
Impact Evaluation
►
►
Statistical analysis: Typically involves inferring
the causal relationship between X and Y from
observational data
Many challenges & complex statistics
Impact Evaluation:
Retrospectively:
• same challenges as statistical analysis
Prospectively:
• we generate the data ourselves through
the program’s design evaluation
design
• makes things much easier!
8
How to assess impact
►
What is the effect of a cash transfer on
household consumption?
►
Formally, program impact is:
α = (Y | P=1) - (Y | P=0)
Compare same individual with & without
programs at same point in time
► So what’s the Problem?
►
9
Solving the evaluation
problem
Problem: we never observe the same
individual with and without program at
same point in time
► Need to estimate what would have
happened to the beneficiary if he or she
had not received benefits
► Counterfactual: what would have
happened without the program
► Difference between treated observation
and counterfactual is the estimated impact
►
10
Estimate effect of X on Y
►
Compare same individual with & without treatment
at same point in time (counterfactual):
sick 2 days
sick 10 days
Impact = 2 - 10 = - 8 days sick!
►
Program impact is outcome with program minus
outcome without program
11
Finding a good counterfactual
The treated observation and the
counterfactual:
have identical factors/characteristics,
except for benefiting from the
intervention
No other explanations for differences in
outcomes between the treated
observation and counterfactual
► The only reason for the difference in
outcomes is due to the intervention
►
12
Measuring Impact
Tool belt of Impact Evaluation Design
Options:
► Randomized Experiments
► Quasi-experiments
Regression Discontinuity
Difference in difference – panel data
Other (using Instrumental Variables,
matching, etc)
► In all cases, these will involve knowing the
rule for assigning treatment
13
Choosing your design
For impact evaluation, we will identify the
“best” possible design given the
operational context
► Best possible design is the one that has
the fewest risks for contamination
Omitted Variables (biased estimates)
Selection (results not generalizable)
►
14
Case Study
Effect of cash transfers on consumption
► Estimate impact of cash transfer on
consumption per capita
Make sure:
• Cash transfer comes before change in
consumption
• Cash transfer is correlated with
consumption
• Cash transfer is the only thing
changing consumption
► Example based on Oportunidades
15
►
Oportunidades
►
National anti-poverty program in Mexico (1997)
►
Cash transfers and in-kind benefits conditional on
school attendance and health care visits.
►
Transfer given preferably to mother of beneficiary
children.
►
Large program with large transfers:
5 million beneficiary households in 2004
Large transfers, capped at:
• $95 USD for HH with children through junior
high
• $159 USD for HH with children in high
school
16
Oportunidades Evaluation
►
►
Phasing in of intervention
50,000 eligible rural communities
Random sample of of 506 eligible communities
in 7 states - evaluation sample
Random assignment of benefits by community:
320 treatment communities (14,446
households)
• First transfers distributed April 1998
186 control communities (9,630 households)
• First transfers November 1999
17
Oportunidades Example
18
Common Counterfeit
Counterfactuals
1. Before and After:
2005
Sick 2 days
2007
Sick 15 days
Impact = 15 - 2 = 13 more days sick?
2. Enrolled /
Not Enrolled:
Sick 2 days
Sick 1 day
Impact = 2 - 1 = + 1 day sick?
19
“Counterfeit” Counterfactual
Number 1
►
Before and after:
Assume we have data on
• Treatment households before the cash
transfer
• Treatment households after the cash
transfer
Estimate “impact” of cash transfer on
household consumption:
• Compare consumption per capita before the
intervention to consumption per capita after
the intervention
• Difference in consumption per capita
between the two periods is “treatment”
20
Case 1: Before and After
►
Compare Y before and
CPC
after intervention
αi = (CPCit | T=1) (CPCi,t-1| T=0)
Before
After
A
Estimate of
counterfactual
(CPCi,t| T=0) = (CPCi,t-1|
T=0)
►
►
“Impact” = A-B
B
t-1
t
Time
21
Case 1: Before and After
Mean
Case 1 - Before and After
Control - Before Treatment - After
233.48
268.75
t-stat
16.3
Case 1 - Before and After
Multivariate Linear Regression
Linear Regression
Estimated Impact on CPC
35.27**
34.28**
(2.16)
(2.11)
** Significant at 1% level
22
Case 1: Before and After
►
Compare Y before and
CPC
after intervention
αi = (CPCit | T=1) - (CPCi,t-1| T=0)
►
Estimate of counterfactual
Before
After
(CPCi,t| T=0) = (CPCi,t-1| T=0)
A
►
►
“Impact” = A-B
Does not control for time
varying factors
Recession: Impact = AC
Boom: Impact = A-D
D?
B
C?
t-1
t
Time
23
“Counterfeit” Counterfactual
Number 2
►
Enrolled/Not Enrolled
Voluntary Inscription to the program
Assume we have a cross-section of postintervention data on:
• Households that did not enroll
• Households that enrolled
Estimate “impact” of cash transfer on
household consumption:
• Compare consumption per capita of those
who did not enroll to consumption per capita
of those who enrolled
• Difference in consumption per capita
between the two groups is “treatment”
24
Case 2: Enrolled/Not Enrolled
Case 2 - Enrolled/Not Enrolled
Not Enrolled Enrolled t-stat
Mean CPC
290.16
268.7541 5.6
Case 2 - Enrolled/Not Enrolled
Linear Regression
Multivariate Linear Regression
Estimated Impact on CPC
-22.7**
-4.15
(3.78)
(4.05)
** Significant at 1% level
25
Those who did not enroll….
►
Impact estimate: αi = (Yit | P=1) - (Yj,t| P=0) ,
►
Counterfactual:
►
Examples:
Those who choose not to enroll in program
Those who were not offered the program
• Conditional Cash Transfer
• Job Training program
Cannot control for all reasons why some choose to
sign up & other didn’t
Reasons could be correlated with outcomes
We can control for observables…..
But are still left with the unobservables
►
►
►
►
(Yj,t| P=0) ≠ (Yi,t| P=0)
26
Impact Evaluation Example:
Two counterfeit counterfactuals
►
What is going on??
Estimated Impact
on CPC
Case 1 - Before and After
Linear
Multivariate Linear
Regression
Regression
Case 2 - Enrolled/Not Enrolled
Linear
Multivariate Linear
Regression
Regression
35.27**
34.28**
-22.7**
-4.15
(2.16)
(2.11)
(3.78)
(4.05)
** Significant at 1% level
►
►
►
Which of these do we believe?
Problem with Before-After:
Can not control for other time-varying factors
Problem with Enrolled-Not Enrolled:
Do no know why the treated are treated and the
others not
27
Solution to the Counterfeit
Counterfactual
Sick 2 days
Observe Y with treatment
Sick 10 days
ESTIMATE Y without treatment
Impact = 2 - 10 = - 8 days sick!
On AVERAGE,
is a good counterfactual for
28
Possible Solutions…
We need to understand the data
generation process
How beneficiaries are selected and how
benefits are assigned
► Guarantee comparability of treatment and
control groups, so ONLY difference is the
intervention
►
29
Measuring Impact
Experimental design/randomization
► Quasi-experiments
Regression Discontinuity
Double differences (diff in diff)
Other options
►
30