Transcript Document

SAI’s role in development and use of key
indicators for R&D evaluation: a quantitative
example and some concluding remarks
INTOSAI Working Group on Key National Indicators
Ville Vehkasalo & Timo Oksanen, 23.4.2013, Krakow
Presentation outline
Our stance on indicator development
Example of how to use key indicators in
quantitative R&D evaluation
Qualitative evaluation possibilities
Concluding remarks; incorporation into the White
Paper on KNI
2
About SAI’s role in indicator development
Depending on the national mandates, the SAI’s role
can be active or passive – or something in between
However, an active role in indicator development can
endanger SAI’s independency and objectiveness
The NAO of Finland has not participated in Finland’s
KNI development
Therefore, we have kept an outsider’s view to Finnish
KNI-system
3
Example: how can we use key indicators in
quantitative R&D evaluation?
EU’s Regional Development Fund (ERDF) aims
to achieve the following objectives in 2007–2013:
1) to enhance regional R&D and innovation
capacities
2) to stimulate innovation and entrepreneurship in
all sectors of the regional and local economy
3) to promote entrepreneurship, in particular by
facilitating the economic exploitation of new ideas
and fostering the creation of new firms.
Source: Regulation (EC) No 1080/2006
4
Example
Cost of the ERDF program in Finland, 2007–
2013: 1,7 billion euros (EU funding)
The effects of ERDF program are monitored
using these indicators:
1) number of new firms
2) number of jobs
3) unemployment rate
4) employment rate
5
Example
5) regional GDP increase relative to the whole
economy
6) share of exports in firms’ turnover
7) share of R&D activities in GDP
8) average educational level.
Source: ERDF Program of Southern Finland 2007–2013
6
Example
The number of new firms is included in Finland in
Figures, which contains key statistical data about
Finland on 25 different statistical topics, produced
by Statistics Finland
This statistic is not included in Findicator, the
official key indicator compilation
7
Example
How can we measure the effects of the 2007–
2013 ERDF program in Finland?
Counterfactual: what would have happened
without the program?
We need a control group that was not subjected
to the program
But in 2007–2013, the whole country is included
in the ERDF program
8
Example
However, in the earlier ERDF program, 2000–
2006, small parts of Southern Finland were not
included in the program
Therefore, we can compare the development in
these new municipalities to those in Southern
Finland that had been included earlier (old
municipalities), in order to control for economywide fluctuations that may also affect start-ups
Population changes can be accounted for by
using per capita figures
9
Example
Straightforward comparison is out of the question,
as old and new municipalities have systematic
differences:
new firms per 1000 capita,
population-weighted means
year 2005
year 2011
old municipalities
5,04
5,25
new municipalities
7,05
7,37
Even before joining the program, new areas had
higher rates of firm creation
10
Example
In order to control for unobservable
characteristics, we have to use panel data: the
same municipalities before and after the policy
change
Specifically, we use the number of new firms from
2005 (before) and 2011 (after) in each of these
municipalities
Small sample: only 31 new municipalities vs. 34
old ones (N = 65)
11
Example
We use the difference from 2005 to 2011, Dy =
y2011 – y2005, as the independent variable
Differencing wipes out time-invariant
characteristics, such as proximity to a larger city
Regression Dy = a + b new_munic
Coefficient estimates are:
coef.
robust s.e.
p-value
new_munic
-0,095
0,352
0,788
constant
0,150
0,200
0,455
12
Example
new_munic estimate has wrong sign but it is
statistically insignificant
Previous estimates are unweighted, i.e. small and
large municipalities get the same weight, or
importance, in the results
Alternatively we can use weights that measure
the size of the municipality, for instance
population levels
13
Example
If we use 2005 population levels as weights, we
get these estimates:
coef.
robust s.e.
p-value
new_munic
0,110
0,171
0,525
constant
0,205
0,123
0,101
Again, can not reject null hypothesis
14
Example
Average change of +0,31 in the intervention
group differs from zero (p = 0,014) but it would be
misleading to attribute this to the program
We had an average change of +0,2 in the
municipalities that were included earlier, i.e. even
without this “new” program
The ERDF program did not cause the observed
increase of 0,31 in the number of new firms
15
Example
This example is a bit unrealistic (sample too
small, etc.) but it illustrates the basic quantitative
evaluation framework:
1) Gather relevant data on intervention and control
groups, before and after the intervention
2) Use simple difference-in-differences regression
or standard panel data methods
3) Present your results with careful interpretation
16
Qualitative methods
Quantitative methods are useful in assessing
program effectiveness
In addition, there are various qualitative
approaches to R&D evaluation, such as
interviews and participant observation
Possible explanations to why or how something
happened/did not happen as planned
General conclusions not possible
17
R&D subproject conclusions (1): Evaluating
specific programs and interventions
Evaluation of R&D programs is difficult, but not
impossible
Finding relevant data can be tricky
Not possible to evaluate all programs; must have
control groups
Without proper analysis, indicators are of limited
use in program evaluation
18
R&D subproject conclusions (2): Evaluating the
whole R&D system as a part of modern society
Problems are threefold: normative, causative and
conceptual
Lack of clear, strategic whole-of-society vision
communicated by the government (normative)
Lack of understanding and knowledge about the
general impacts of R&D system on modern
economies (causative)
What would and could be the role of SAIs and
Key National Indicators of R&D in all of this?
(conceptual)
19
R&D subproject: Incorporation into the White
Paper on KNI
WG Secretariat can freely use our reports in
preparing/editing the White Paper on KNI
For instance, our reports could be useful in
augmenting the section Principles and
Guidelines, subsection Guidelines for knowledgebased economies, where the evaluation of R&D
programs is already mentioned
20
R&D subproject: List of reports
Utilising R&D knowledge at R&D policymaking in Finland:
problems and promises, Helsinki 2011 (.ppt)
SAI’s role in development and use of key indicators for
R&D evaluation, Riga 2012 (.ppt)
SAI’s role in development and use of key indicators for
research and development (R&D) evaluation, 2012 (.doc)
SAI’s role in development and use of key indicators for
R&D evaluation: a quantitative example and some
concluding remarks, Krakow 2013 (.ppt)
21
Thank you!
[email protected]
[email protected]
http://www.vtv.fi/en
22