Transcript Quantifying the Impact of Social Science Research

Quantifying the Impact of
Social Science
Development Research:
Is It Possible?
Kunal Sen
IDPM and BWPI, University of
Manchester
Based on paper: Literature Review on
Rates of Return to Research, available
on DFID R4D website.
Quantifying the impact of research:
the rate of return to research





Similar to any other investment by the public sector, research is expected
to yield benefits that are in excess of the costs of funding research.
The rate of return to research is one important way that net benefits to
funding research can be measured.
To calculate the rate of return to research, the present value of the
current and future benefits of the research is compared to the total costs
of the research, and an internal rate of return is calculated to equalise the
revenue stream with the cost outlays.
This internal rate of return is the rate of return to research.
The higher the rate of return to research, the higher is the expected net
payoffs from research, and the stronger case for investing in research as
compared to other types of public investment. Or for investing in one type
of research versus another.
TWO QUESTIONS
WHAT DO WE KNOW ABOUT THE RATE
OF RETURN TO DIFFERENT TYPES OF
SOCIAL SCIENCE DEVELOPMENT
RESEARCH?
 TO WHAT EXTENT IS IT POSSIBLE TO
CALCULATE RATES OF RETURN TO
DIFFERENT TYPES OF DEVELOPMENT
RESEARCH?

The Causal Chain from Research
to Impact
1.
2.
3.
Did the research influence policy
thinking/decisions/processes
(the
attribution problem)
Did the policy intervention/change/reform
lead to the observed outcome (the
identification problem)
Can the benefits of outcome(s) be
quantified? (the measurement problem)
Figure 2. The Results Chain for Different Types of Research
The Attribution Problem
Research
Programmes/Centres
The Identification
Problem
Policy Change/
Intervention/Reform
Outcomes-TangibleIntangible
Impact
Communication
Of Research Findings
The Measurement Problem
Contextual Factors
The Attribution Problem
The attribution problem can be broken down to the
following components:
a) how well defined is the set of research users?
b) the counter-factual: will the policy change have
occurred without the research taking place?
c) how important are contextual factors and
exogenous events in influencing policy,
independent of the research?
The Identification Problem
Since developmental outcomes may occur due to many reasons,
and policy interventions is one possible cause of such outcomes
among many others, it is often difficult to precisely identify
whether the policy intervention can be causally related to the
outcome in question.
There are three different aspects to the identification problem:
a) selection bias;
b) omitted variable bias;
c) Reverse causality.
The Measurement Problem
An important requirement in the application of the rate of return
approach is that all benefits, past, present and future, can be
quantified and expressed in the same unit of value.
This leads to five problems in the measurement of these benefits:
a) valuing multiple outputs;
b) valuing intangible outcomes;
c) time-scale of measurement;
d) the degree of uncertainty on the size of the impact;
e) measuring effects, where there are macro-changes or strong
spillover effects.
Figure 3. Differential Paybacks to Two Projects in Health Research
Figure 4. Rates of Return to Research in Different Time Dimensions
Project 2
Project 1
Cumulative Net Benefit
Time
Figure 5. The Degree of Uncertainty associated with Returns to Research
Type of Research 1
Type of Research 2
Number of
Projects
Number of
Projects
20%
Rate of return to research
20%
Rate of return to research
Methodologies to quantify the impact of
policy change/intervention
Simulation models
 Regression based methods
 Case studies
 Randomised control trials

The Results Chain for Different Types of
Research
Type of
Research
Agriculture
Health
Social Policy
Economic
Policy
Attribution
Identification
Relatively
Relatively easy.
straightforward
Relatively
Easier for
straightforward clinical trials; less
straightforward
for
multidimensional
measures of
health outcomes
Complex,
Moderate degree
depends on
of difficulty.
contextual
factors
Complex,
Moderate
depends on
degree of
contextual
difficulty.
factors
Measurement
Relatively easy
Relatively easy
Moderately
difficult
Moderately
difficult
The Results Chain for Different Types of
Research – contd.
Type of
Attribution
Research
Infrastructure Complex,
depends on
contextual
factors
Governance
Climate
Change
Identification
Moderate
degree of
difficulty –
difficult to
establish
causality
Very complex, High degree of
depends on
difficulty.
contextual
factors
Very complex, High degree of
depends on
difficulty.
contextual
factors
Measurement
Moderately
difficult.
Very difficult.
Very difficul
What do we know about the rates of
return to different types of social science
research?



Usable rates of return to research (RORs) exist
– agriculture and health research
Proxy rates of return do not exist, but there are
credible ways to calculate RORs – infrastructure
research, economic and social policy research
Proxy Rates of return do not exist, and there are
no credible ways to calculate RORs governance research, climate change research.
So can we calculate the rates of return to
different types of social science
research?



A non-starter for research which lead to
intangible outcomes, where the time-scale of
outcomes is very long and where the
identification problem is particularly challenging–
governance and climate change research.
Possible for economic and social policy research
– but the informational requirements for doing so
are very high.
Already exists for agriculture and health
research.
How to improve our ability to
measure the impact of research




In general, there is a need for investing in improved methodologies
that tackle the identification problem (but not necessarily a focus on
randomised control trials only).
Investing in monitoring and evaluation processes at the start of the
research programme to address the attribution problem – creating
baselines and using case-studies to track the impact of research.
Looking at best practice on how to address the attribution problem –
e.g. Fred Carden’s work in IDRC.
A limited use of methodologies such as willingness to pay where
there are clear tangible benefits of research to address the
measurement problem.