Transcript The Political economy of economic research With a case

Meta analysis
The case-study of aid effectiveness
European Journal of Political Economy.
1st issue 2008 p 1-24
Martin Paldam
http://www.martin.paldam.dk
Click on to Working Papers
Joint work with Hristos Doucouliagos,
Deakin university, Melbourne, Australia
Primary studies of to estimate an effect
Data: The primary data
Your work in ½-1 year: Less than 1 man-year of
work. Reflects: You ideas + have read.
Meta studies of the literature on an effect
Data: The literature
The AEL-AAL 250 studies: 200 man-years of work.
Reflects: The ideas of everybody in the field.
Forensic economics: Quantitative studies of literature: Where are we? and how did we get there?
The talk discusses the political economy
of some of the results of 8 meta studies of:
AEL, Aid Effectiveness Literature, 2005-7
4 studies: 2 out ,1 accepted and 1 R&R
AAL, Aid Allocation Literature 2007-9.
2 new WPs, 2 in partly first draft
Meta studies often give embarrassing results.
Strong reactions of referees:
Most negative and positive I have experienced.
M1 studies of the effect μ, giving M2 > M1
estimates of μ, that can be calibrated to same
scale (partial correlations).
The full-set, [M]F, and the best-set, [M]B.
Coded with a C-vector for each estimate of all
possible characteristics:
Our case: An C-vector for 60 possible model
controls, estimation methods, data, author, journal
Funnel plots, MRAs, …
Meta studies: Three questions to a literature
1.
Does the result converge to something that
we can consider the true value?
2.
Are there breakthroughs (structural jumps)
which can be identified and explained:
Does journal quality, estimators, etc matter?
3.
Does the distribution of the results point to
biases
Three general results (all embarrasing):
(a) Amazing variation in results:
Always signifiantly different results
(b) Estimators rarely important:
Profession obsessed with estimators?
Sociology of our profession: Macho test
(c) Publication biases common:
Publication biases: Look at funnel plot.
All estimates plotted over precision ln N
Case:
The price elasticity of beer.
Intuition: Negative but not large.
Literature: Average finding surprisingly large
Study the funnels
FATs testing for asymmetries
Correct distribution to find true average
Several formulas, fortunately robust!
Tom Stanley (+ Chris Doucouliagos)
New paper in Oxford Bulletin
Monte Carlo experiments
The aid effectiveness discussion
Great subject for Meta study:
(1) Emotional: One side is the side of the angels!
(2) Strong interests: Stay on the gravy train!
(3) A basic statistical fact that has to be overcome
The zero-correlation result
Zero-correlation result  two great games:
The control variables game: Find a plus set
The causality game: Find a big negative bias!
Zero correlation between aid and growth:
for all aid recipients
Period
N
Cor
Period
N
Cor
1960-65
92
-0.12
1985-90
143
-0.12
1965-70
103
-0.00
1990-95
169
-0.00
1970-75
111
-0.01
1995-00
178
0.09
1975-80
122
0.06
2000-05
175
-0.02
1980-85
134
0.09
Aver.
1227
-0.00
The causality game: A literature on each arrow
The reverse causality flow R.
We wish a negative coefficient
Mechanism
Sign
R1: Aid to poor + poor grow less
Neg, small
R2: Disasters  low growth + aid
Neg, small
R3: Growth  good projects  aid
Plus ?
R4: Growth  future markets  aid
Plus, small
Sum: Theoretically
Meta study: 30 papers, 211 estimates
???
Plus, tiny
Hence: The data is a big problem.
They point to aid ineffectiveness
and it is not obvious
that it is due to simultaneity
My own primary studies: very little
(everybody know)
Shift perspective: to research
Point of talk:
Economics assumes that all humans have
priors/interests. Why not us?
Also economists also have priors/interests.
We operate on the market for economic research.
It may not be a perfect market.
Our small talk at lunch tables, in bars etc.
Often assumes that journals have biases, that
referee processes are …
Limitation of discussion: (a) Empirical + (b) macro
Ad (a): We analyze M studies of the same effect
Ad (b): Data is limited relative to the amount of
research. Thus data mining problem
What can we prove?
Meta studies claim they can prove a great deal.
Not for individual studies, but for specific literature
Our studies typically find asymmetries pointing to
priors. (In a moment)
We like to believe that research is a process that
search for truth that converges to the truth.
Assume: Truth is the true value of μ.
Process in individual/paper
Process on market, market incentives:
Are the incentives truth-finding consistent?
Process for individual researcher, X
X search for a value of μ, till he is satisfied.
The paper is thus the result of a
stopping rule in Xs search process:
X stops when he has found a μ that:
(a) Is in accord with his priors or his interests
(b) Is publishable
(c) Can be defended statistically
Thus: When I stop have I
found truth or confirmed my priors?
Process on market:
Innovation + replication generates trust in results
Innovation: Theory, estimation technique, data.
Innovation easy to publish (?)
Replication:
Independent: Other authors on new data
Dependent (1): Same author on new data
Dependent (2): New author on new data
Macro: Normally overlapping data so
only: Marginally independent
Thus: Effect on estimate of extra data
Data mining:
The number of estimates on subsets of the same
data is large relative to the number of observations
Ex: Phillips curves. Estimated on the w, p, u data
for 30 countries over the last 50 years?
Guess: 5 mio estimated?
Ex: Money demand,…
Ex: Growth regressions:
Sala-i-Martin alone about 90 million
Consequence:
Type I errors reduced: Rejecting true model
Type II errors increased: Accepting false models
Hence in heavily mined fields:
There must be many type II errors
Thus independent replication necessary.
And meta studies highly needed
Not problem of each researcher; but the collective.
We all read up some of the literature and join the
mining collective.
We fish in the common pond of df ’s.
A double tragedy of the common.
1. It is the standard tragedy that we exhaust the df.
2. It is also a tragedy that nothing visible happens
we can just go on and on!
The Aid Effectiveness Literature, AEL, studies:
μ = ∂g/∂h, conditional on everything our profession has thought of – it is a great del.
Micro base. Average LDC growth 1.5%.
Projects based on cost-benefit (growth
contribution): Social rate of return 10%.
Thus, h = H/Y ≈ 7.5%  0.75 pp growth
It should be highly visible in data,
but as we have seen it is not.
Thus a puzzle: It is the AEL paper generator:
In 2006 aid exceeded $ 100 bill + AEL paper nr
100 came out. No agreement on results.
Data: Aid started in mid 1960s. Now ap 145 data
per year: and 6000 annual observations.
Average to 5 years: about 1000.
Published regressions 1,025, made 25,000?
Alternative: Sum of N is 25,000
Likely that false models have appeared
Before we look at results look at the data,
Simple regressions between aid and growth:
The zero correlation result
Growth first
No lag
Aid first
All
Const
Coef
p-val
Coef
p-val
Coef
p-val
1.816
0.000
1.579 0.000 1.504 0.000
Effect -0.039 0.023 -0.010 0.935 0.003 0.364
N
895
1,008
876
R2
Box Const
0.006
1.843
0.000
0.000
0.000
1.676 0.000 1.578 0.000
Effect -0.052 0.007 -0.022 0.207 -0.010 0.559
N
841
945
839
R2
0.009
0.002
0.000
Thus, the AEL starts from a zero correlation, and
put structure on this result till something appears.
Model is the same as the Barro-growth regression:
git = α + μhit + (γx1it + ... + γxnit) + uit
or
git = α + μhit + δzit + ωhitzit + (γx1it + ... + γxnit) + uit
Researchers have tried 60 x’es and 10 z’s
Many millions possible permutations, each gives a
different estimate of μ. As average is zero half are
positive, and 5% are significant. What to choose?
A tool: The funnel plot
Prior
Bias
Inside or outside
(1) For results
Polishing
(2) Ideology
Accordingly
Authors, referees,
journals
Authors, journals?
(3) Goodness
Accordingly Authors, journals
(4) Interests
Accordingly Institutions 
authors, journals?
(5) History
Path
dependent
Author history +
“clubs”
In the AEL: Everything goes together to generate:
The reluctancy bias
Researchers and journals are reluctant to
publish negative results
Proof follows
Let us look at the 5 priors – one at a time:
Polishing:
We want to display our goods as well as possible.
Then they are easier to sell to journals
Career + feel well.
Strong incentives:
What do we expect to see?
Easier to polish in small samples:
Study t-ratios as a function of df: t = t(N)
If random ln t proportional to ln N:
The MST: ln│ti│= α0 + α1 ln Ni + ui
test: α1 < 0  polishing
Often found in meta studies, in the AEL also.
Ideology:
An ideology that predicts the size (sign) of μ
 authors with that ideology find that size (sign).
In the AEL:
a. Libertarians (Friedman, Bauer): Aid  larger
public sectors  planning  socialism  harms
b. “New-left”: Aid from capitalist states  capitalism and exploitation  harms
Both OK (not many authors) especially early
Goodness:
Common finding: We all want to look good
and most want to be politically correct
Shown as asymmetry of funnel plot: The FAT.
Part of the funnel is missing.
In the AEL: Aid aims at doing good (+ …)
So to show that it fails is bad.
Nice to be on the side of the angels: Bono,
Jeff Sachs, Gordon Brown, Koffi Anan, etc.
Causes reluctancy.
The FAT: εi = β0 + β1si + υi, where εi is the
standardized effect, and si is its standard error.
Also, look at the funnels
We look at: μ = μ(N) and μ = μ(t)
in a moment
Interests:
Normally Ok: there are many interests.
In the AEL: diffuse interests on the one side.
And the “Aid Industry” on the other side.
It has a turnover of $ 100 bill. It has a
composition with includes a bureaucracy +
political parties + NGOs + business + unions.
It gives about 10% in consultancy fees
+ 0.25-0.5 % to research.
The aid industry want aid to work
Many of those working in the AEL are working
for/financed by the aid industry
Problem: Many does not write so!
Gives reluctancy as well
Psychology: Alignment of priors to interests
Obs: Goodness + interests give same result in this
case. Hence we expect clear effects.
History:
50% are in one paper only.
The rest are in more + many additional links.
People are z > 0.5 committed after one paper to
find the same result. Our guess z = 0.9
Also, same department as, writing PhD under, …
Very significant: Fighting schools
Reluctancy:
Asymmetry of missing negative values
How should it look? Should be visible on μ = μ(N).
Sorting out μ = μ(N) and μ = μ(t)
Problem:
Learning by doing:
μ = μ(t) should slope upward.
Look at the two graphs:
Problem: No learning by doing,
See graphical interpretation, next slide
Run: μNt = α + β ln N + γ t + ε
Multicollinearity: N goes up for t rising
α
β on ln N
0.31 (7.1)
-0.043 (-4.7)
0.19 (9.7)
0.28 (5.9)
-0.026 (-2.1)
γ on t
N
-0.00027 (-4.4)
538
-0.00015 (-1.9)
Thus:
Reluctancy confirmed: Is it goodness or interests?
Test: Use (poorly measured) interest variable
Effect of interest:
Always sign as expected, not always significant:
It is not a big effect, but it is there!
Correct funnel for asymmetry:
Net result insignificant
Thus the literature has not showed that aid
works after 100 papers over 40 years.
Depressing
Set of MRAs – very bulky see paper!
I just give some highlights:
A
B
Reluctancy + interests found
Polishing found
C No effect of quality of publication.
D No structural shifts due to new theory
E
ODA give slightly better results than EDA
F
No effect of new estimators
G Clear effect of new data
Bias in process: Better for career to develop new
estimators than new data. An incentive that is not
truth finding consistent.
The end:
We behave as predicted
by our theories
Also economists are human!