Transcript Document

SAI’s role in development and use of key
indicators for R&D evaluation
Timo Oksanen 3.4.2012
About SAI’s role in indicator development
Depending on the national mandates, the SAI’s role
can be active or passive – or something in between
However, an active role in indicator development can
endanger SAI’s independency and objectiveness
The NAO of Finland has not participated in Finland’s
KNI development
Therefore, we have kept an outsider’s view to Finnish
KNI-system
2
Findicator: The place to find up-to-date
information on:
demographic developments in Finland
international crisis management
public expenditure
obesity among Finns
income differences
fish catches
Available to all at www.findicator.fi
3
Findicator is a comprehensive databank
Includes approximately 100 indicators of social
progress
Up-to-date and relevant information on important
societal issues
Indicators selected in consultation with user
groups and data providers
Grouped thematically and by policy issue in line
with Government Programme
4
R&D key indicators
In the Finnish KNI collection (findicator.fi), there
are only two indicators that measure R&D
outcome and effort:
Time series of patent applications
Time series of R&D expenditures, by sector
5
Patent applications in Finland, 1972 - 2010
5000
4500
number of patent applications
4000
Foreign
3500
3000
2500
2000
Finnish businesses
1500
1000
500
Finnish private
0
1970
1975
1980
1985
1990
1995
2000
2005
2010
6
R&D expenditures by sector, 1971 - 2010
8000
7000
6000
universities
4000
government+npo
business
3000
2000
1000
0
19
71
19
73
19
75
19
77
19
79
19
81
19
83
19
85
19
87
19
89
19
91
19
93
19
95
19
97
19
99
20
01
20
03
20
05
20
07
20
09
million euros
5000
7
On the basis of our R&D audits it can be said that…
1. A lot of work has been done (Slides 7-9)
http://www.tekes.fi/u/Better_results_more_value.pdf
2. Some things are just happening (slide 10)
3. A lot of things has still to be done….(slides 11-18). To put it
shortly:
”Systemic change is a highly topical issue both in Finland
and internationally. Climate change, aging of the
population, advancing sustainable development as well as
structural transformations in the economy and the related
need to find new growth areas are current grand
challenges. Responding to these challenges is not possible
through individual small-scale reforms but calls for systemlevel changes”.
http://www.tekes.fi/u/systeemisen_muutoksen_haasteet.pd
f
8
Indicators related to economy and economic renewal
Phenomenom
Indicators
National prosperity
GDP per capita
Overall productivity of the economy
Total Factor Productivity TFP
Productivity renewal indicator
Foreign Direct Investments
Share of Foreign Direct Investments per GDP
Strengthening of intangible assets
Position in global value-networks
Share of new innovative products and services from business turnover.
Volume and share of intangible investments
Exports of knowledge-intensive sectors
Continuous improvement of
competitiveness
Development of turnover in knowledge intensive sectors (or alternatively
in KI jobs)
Collaboration, networks and knowledge
flows
Share of public and private organisations having collaborated in
innovation projects
Capability to innovate
Development of patenting, registered trademarks and designs (EPO /
USPTO / TRIAD)
Investments in R&D&I
Share of R&D&I expenditure in business turnover
Government direct & indirect support to business R&D
Foreign direct investments in Finnish R&D&I
Human resources for R&D&I
Availability of highly educated workforce
General conditions and incentives for
R&D&I
GDP share of VC investments at different growth stages
9
9
Indicators related to well-being
Phenomenom
Indicators
Health and quality of life
Life expectancy
Well being in working life
Share of 25 to 64-year-olds very or fairly satisfied with
their current job
Healthy and safe living environment
No indicator selected yet
New knowledge and competence associated with
well-being
Scientific publications among the top 10% most cited
publications worldwide as % of total scientific publications
of the country
New products, processes, services and social innovations
Innovations and systemic changes supporting wellbeing
High-quality and innovative well being services
Productivity of the social and health services of
municipalities and federations of municipalities
Quality and extent of R&D&I activities directed
towards well-being
Share of public organisations involved in health and wellbeing related R&D&I activities
Interorganisational collaboration related to well-being Mobility of researchers in the fields of health and wellin value networks and the strengthening of flows of
being
know-how
R&D&I investments on well being
Private and public R&D expenditure on well-being, health
care and working life
Knowledge and human resources
No indicator at present
10
Supportive operational environment
Health and social care costs
10
Indicators related to knowledge, education and culture
Phenomenom
Indicators
Competences and opportunities for life-long learning
Education and active citizenship
Education level of population
Interest in science, research and technology
Active and diverse cultural life
Value added in the cultural sector
Openness, diversity and networks
Citizen participation
Active and vital cultural life
Share of foreign nationals in the human resources of science and
technology
OECD international student assessment - PISA
Scientific publications within 10% of the most cited publications
worldwide as % of total scientific publications of the country
Patent applications by institutes of higher education and public
R&D-institutes
Participation of population aged 18+ in lifelong learning
No indicator selected at present
Internationalisation and openness in research activities
No indicator selected at present
Scientific research and education
Share of doctors of the Human Resources in Science and
Technology
No indicator selected at present
The quality and efficiency of the educational system
The quality and efficiency of higher education and
research
Knowledge as a resource for the economy and society
Disseminating research information to citizens and the
use of society
Research and innovation activities related to culture
International mobility and cooperation in research
Investments in competences and human resources
Investments in competences and human resources
Investments in general education and adult education
No indicator selected at present
Researcher mobility (inwards and outwards)
Investments in R&D activities in the public sector and in the
higher education sector
Research personnel's share of workforce
Costs from adult education
Investments in the culture related to research and
The Government R&D funding based on societal objective:
11
innovation
culture
Investments in international cooperation and networking The share R&D expenditure from abroad in the Higher11Education
and Government sectors
Levels of the implementation
Indicators
• Visual level = Graphics and
visualisation of indicators
• Technical level = platform and
connections, technical
implementation of
administration
• Definition level = Detailed
indicator definition
• Data level = sourcing and
updating of data
• Administrative level covers all
the levels above defining the
responsibilities and funding
Visual level
Activity level
Technical level
Definition level
Data level
Administrative level
• Activity level = e.g. availability of
the information behind the
indicators
Data
12
12
Concluding remarks (1)
There is a clear need for discussion on research and
innovation and the related societal development and
challenges
The framework should allow for the decision makers to
quickly assess the Finnish research and innovation
development against the main international trends and key
benchmark nations
The question of phenomena selection contains a political
aspect
The set of indicators (including the impacts chains) needs
continuous update on the basis of newest knowledge
As well, it should adapt to the changes in societal priorities
and goals
International cooperation is needed to make existing and
forthcoming efforts comparable
More knowledge needs to be gained on analyzing the
impacts of research and innovation on well-being and the
society
13
13
Concluding remarks (2)
Factors that need to be considered particularly when setting target values
to indicators include:
Accumulation factor: how much emphasis is put on the current
level of standing, what has been achieved and built over the past
years or decades
Improvement and development factor: as compared to relation
to peers or selected benchmarks
Systemic factor: how much emphasis is given to the existence
and functioning of the research and innovation system (or
ecosystem) as a whole, instead of its elements alone.
Global factor: how much emphasis is given to Finnish position
and impact in global value chains, EU frameworks or in addressing
global challenges, as compared to our success and impact within
the country
Relevance factor: how well is the current research and
innovation activity tuned for, prepared or adaptive to address the
societal challenges, such as ageing of population, economic
recession, etc.
14
14
A step to wider scope: where are we now?
(Fred Gault: Social impacts of the development of the STI
Indicators, 2011)
Indicator development is a dynamic activity. The OECD
Innovation Strategy included a Measurement Agenda which is
now being implemented. It includes intentions to:
i. Improve the measurement of broader innovation and its link
to macro-economic performance;
ii. Invest in high-quality and more comprehensive data
infrastructure to measure the determinants and impacts of
innovation;
iii. Recognize the role of innovation in the public sector and
promote its measurement;
iv. Promote the design of new statistical methods and
interdisciplinary approaches to data collection; and,
v. Promote the measurement of social goals and social
impacts of innovation”.
15
Special issues…(1)
Policy impacts (Gault, 2011)
As the indicators expand and policy makers recognize that
innovation is not an isolated event, more attention is being given
to the framework conditions and the policy mix that helps the
system to work better. As more micro data analysis is done, the
important result that the propensity to innovate in firms is higher
than the propensity to do R&D will have more influence on policy.
16
Special issues…(2)
Funding of Finnish universities 2012 
The model comprises in three main parts: education, research,
and other education and science policy objectives. A total of 75%
of the core funding will be allocated on the basis of a formula for
education and research, of which 41% is based on educational
factors and 34% on research factors. The remaining 25% of the
core funding is based on education and science policy objectives.
Education-based funding criteria (as % of core funding): Master’s
degrees awarded by the university (15 %), Bachelor’s degrees (9
%), the number of students completing a minimum of 55 credits
(11 %, of which 3 % based on data produced by the student
feedback system from 2015), credits completed in open university
and non-degree studies (2 %), the number of degrees awarded to
foreigners by the university (1 %), incoming and outgoing
international student exchanges in the university (2 %) and the
number of job-holding graduates (1 %).
17
… universities…(3).
The research-based funding criteria: doctoral degrees
awarded (9 %), publications (13 %, of which 10 %
international refereed publications and 3% other scientific
publications - from 2015 the number of the Finnish
Publication Forum classification levels 2 and 3 publications
instead of international refereed publications and the
number of level 1 publications instead of other scientific
publications); competed research funding (9 %, of which 3
% international competed research funding and 6% other
competed research funding); doctoral degrees awarded to
foreigners (1 %); and foreign teaching and research
personnel (2 %).
18
SAI’s role in development…?
Attribution/contribution/independence/capacitybuilding/resources:
So far the NAO of Finland has not participated in
Finland’s KNI development
The Future: it is difficult or even impossible to
separate the impact of SAI from other inputs and
activities.
19
The Role of the SAI with the STI-indicators?
i. besserwisser?
ii. Facilitator?
iii. Risk Manager?
iv. Measurement Technician?
v. Capacity
Builder?
(workshops 2011,
brainstorms 2012)
- from measurement of results for
contributor of the knowledge
management of the whole R&D
- from peaces to the whole (the
life cycle of the knowledge and
knowledge management)
- from analyses to synthesises,
- from outputs to processes and
capabilities of actors
- from mechanical synthesises to
real problem solving
vi. Problem solver? - …by co-operation with the R&D
actors
vi. Link between
R&D actors
and desicion
making?
20
The last steps at 2012: Findicators: An official
network to coordinate the development of
indicators (1)
to make propositions about developing the
services and infrastructures of indicators (and
knowledge management) in the government of
Finland
to follow-up international development in the field
to co-operate with other developmental projects
in this field, etc.
21
The spesific lines of developing R&D indicators
in near future 2012 (2)
http://www.tekes.fi/u/Better_results_more_value.p
df
The Academy of Finland and Tekes will report
until September 2012 to the Research and
Innovation Council of Finland about the main
lines of implementing the Framework (slides 911 above) (resources, administrative
responsibilities and task, pilots (focusing single
indicators) etc.
are politics able to utilise this kind of information?
how? when? why? where? how much it costs?
22