Transcript ERC - Inist

A concept for inferring « Frontier Research »
in research project proposals
Dirk Holste
Edgar Schiebel
Thomas Scherngell
Marianne Hörlesberger
Ivana Roche
Dominique Besagni
Claire François
Pascal Cuxac
ISSI 2011 – Durban, South Africa, July 4-7
Plan of the presentation
• Introducing the concept
• The ERC framework
• Scientometrics & Text mining
• Frontier research & Indicators
• The indicators
• Proposal selection function
• Discussion
2
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
ISSI 2011 - Durban, South Africa, July 4-7
Introducing the concept
Concept for inferring attributes of frontier research in
peer-reviewed research project proposals under the
scheme of the European Research Council (ERC)

Design, implement, test and refine a bibliometric/scientometric
model that investigates whether submitted and/or selected grant
applications fulfill the requirements of frontier research as this notion
was enounced by the High Level Expert Group of the ERC

Build and compare outcomes of the model with the review decision
in order to obtain further insight and to bring a reflection on the
influence of frontier research in the peer-review process
3
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
ISSI 2011 - Durban, South Africa, July 4-7
The ERC framework (1/4)
First European funding body to support investigator-driven
(frontier) research through open and direct competition
• Main goals:
– Scientific excellence as the only scientific selection criteria
– Major grants for the truly best and creative researchers and their ideas, to
identify and explore new opportunities and directions in all fields of
research
• Scientific domain clusters (panels):
– Physical and Engineering Sciences (PE)  10 panels
– Life Sciences (LS)  9 panels
– Social Sciences and Humanities (SH)  6 panels
• Grant Application schemes:
– Starting independent researcher grants (StGs)
– Advanced investigator grants (AdGs)
proposals submitted by principal investigator (PI)
to every project is allocated a main panel that becomes its “home discipline”
4
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
ISSI 2011 - Durban, South Africa, July 4-7
The ERC framework (2/4)
• StGs  Up-and-coming investigators: bring means and
opportunities for young investigators to develop independent
careers
• AdGs  Established senior researchers: foster pioneering, far-
reaching research endeavors, with high-risk/high-impact potential,
break of established disciplinary boundaries or exploration of new
productive lines of enquiry, methodology or techniques
• Funded through EU’s Framework Programme FP7
• Annual calls: 1st in 2007
5
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
ISSI 2011 - Durban, South Africa, July 4-7
The ERC framework (3/4)
• ERC annual budget evolution (2007-2013):
1800
1500
Mio. €
1200
900
600
300
0
20
07
20
08
20
09
20
10
20
11
20
12
20
13
• Rate of selected proposals:
– StGs (2009)  10% (244 out of 2,503 submitted proposals)
– AdGs (2009)  15% (244 out of 1,584 submitted proposals)
• Selection process:
– 25 expert panels
– members selected by the Scientific Council
– two-step peer-review process
6
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
ISSI 2011 - Durban, South Africa, July 4-7
The ERC framework (4/4)
The issue of the frontier research for the ERC
 the ERC’s High Level Expert Group enounced key attributes of frontier research:
Standing at the forefront of creating new knowledge… responsible for fundamental
discoveries… achieving occasional revolutionary breakthroughs
Intrinsically risky
Reducing the barriers between basic and applied research
Pursues questions irrespective of established disciplinary boundaries
The peer-review process shall indentify such activities /researchers out
of an immense number of submitted applications
7
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
ISSI 2011 - Durban, South Africa, July 4-7
Scientometrics & Text mining
• Identification of 4 key attributes according to the understanding
of the ERC High Level Expert Group’s notion of frontier
research:
•
•
•
•
Novelty
Risk
Applicability
Interdisciplinarity
• Formalisation of the key attributes in terms of indicators build
by combining scientometric and text mining approaches:
8
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
ISSI 2011 - Durban, South Africa, July 4-7
Frontier research & Indicators
Data related to
• Research project
Risk (independence,
originality/uniqueness)
Correspondence
Starting (StGs) or Advanced (AdGs) grants
Grant Applications
Data related to
• Principal investigator
Peer-review Step2
External
databases
Peer-review Step1
ERC
database
Scientometric
and
content-based
indicators
Data extraction,
text-mining
Novelty (1-timeliness:
cited bibliography;
2-proximity: near to
emerging research
areas)
Interdisciplinarity
(across disciplinary
barriers)
Pasteuresqueness
(potential applicability)
Data related to
• Research environment
Comparison: Is there congruence between our
results and peer-review decision? Are key
attributes correctly represented?
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
Selection function
model: discrete choice
model based on
indicators
9
ISSI 2011 - Durban, South Africa, July 4-7
Timeliness indicator
It is used to infer the « innovative degree » of the proposal
through the bibliographic references cited by the PI in his(her)
proposal
• Data source: ERC data
• Hypothesis:
– the more recent references are, the more likely the work is at the frontier
research
• Calculation:
– this indicator focuses on the time elapsed since the publication of the
proposal cited references and Timeliness is obtained from the difference
between the year of the proposal submission and the year of publication
of each cited reference in the proposal
10
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
ISSI 2011 - Durban, South Africa, July 4-7
Proximity indicator
• It is employed to infer the « innovative degree » of the proposal
through the dynamic change of the scientific landscape
corresponding to the proposal’s allocated panel
• Data source: ERC data & bibliographic DB
• Hypothesis:
– the closer a proposal is to regions of positive dynamic change, the more
innovative it is
• Calculation:
– raw data are obtained from bibliographic DB and ERC panel description to
identify and extract discriminating terminological information
– a clustering step produces a cluster map that groups similar references and
represents the publication landscape corresponding to the considered panel
– a diachronic analysis is used to study the evolution of the landscape between
two time periods
– clusters are ranked with respect to its innovativeness and the similarity of each
proposal to its nearest clusters is calculated
11
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
ISSI 2011 - Durban, South Africa, July 4-7
Risk indicator
• It is used to infer the « personal risk » of the PI in executing the
proposed research
• Data source: ERC data & bibliographic DB
• Hypothesis:
– references serve as one informative source for getting a « bibliometric
research profile »
– the lower the overlap between two reference profiles, the more risky is the
proposal for the PI
• Calculation:
– comparison between the profile of the proposal and, on one hand, the
profile of the past research of the PI and, on the other hand, the profile
of the research work in the « home » panel of the proposal
12
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
ISSI 2011 - Durban, South Africa, July 4-7
Pasteuresqueness indicator
• It is used to infer the applicability of expected results of the
proposal by considering evidence for immediate or intended
application
• Data source: ERC data & bibliographic DB & patent DB
• Pasteuresqueness… What does it mean?!
• Calculation:
Pure basic
Research
Use-inspired basic
research
(Bohr)
(Pasteur)
Frontier
Research
– patents granted by PI
– information related to industry
involvement
– self-citations published in journals
with prescribed « applicability »
Relevance for the advancement of
knowledge
– this neologism comes from the Pasteur’s Quadrant which gives a
categorization of scientific issues that seek both fundamental
understanding and social benefit
Pure applied
research
(Edison)
Relevance for immediate applications
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
13
ISSI 2011 - Durban, South Africa, July 4-7
Interdisciplinarity indicator
• It is used to infer self-consistently the presence, in the proposal,
of characteristic terminologies which belong to several ERC
panels different than the proposal’s « home » panel
• Data source: ERC data
• Hypothesis:
– the higher the occurrence of keywords belonging to different ERC panels,
the more interdisciplinary the proposal is considered
– the higher this number of different involved ERC panels, the more
interdisciplinary the proposal is considered
• Calculation:
– keyword labelling according to its statistical frequency of occurrence across
all panels
– assessment of the concentration in the proposal of keywords labelled as
belonging to different ERC panels
14
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
ISSI 2011 - Durban, South Africa, July 4-7
Proposal selection function (1/2)
We use a statistical discrete-choice model (DCM) to estimate the influence of
indicators on the selection probability of a research project proposal to be funded
Statistically, we model a dependent variable that takes on the form:
 1 proposal is accepted
Yi  
 0 otherwise
i , j = 1, … , n
The selection probability of a proposal to be accepted, Pr(Yi = 1),
depends on the set of indicators summarized in a matrix X such that
Pr(Yi  1)  F ( X i   )
Pr(Yi  0)  1  F ( X i   )
where Xi is a set of k observed factors for proposal i ,
β is the estimated k-by-1 parameter vector reflecting the impact of changes in Xi
on the probability Pr(Yi = 1)
15
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
ISSI 2011 - Durban, South Africa, July 4-7
Proposal selection function (2/2)
The logistic distribution is chosen for model estimation leading
to the empirical model given by:
expX i  
Pr(Yi  1)   ( X i   ) 
1- expX i  
The parameter estimation is based on Maximum-Likelihood
procedures.
16
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
ISSI 2011 - Durban, South Africa, July 4-7
Discussion
The developped concept…
• is ambitious  difficult, by employing bibliometric methods, to faithfully
•
•
•
•
represent attributes of frontier research and validly quantify them to evaluate
the grant decision
cannot substitute expertise  intends to bring us a bibliometric model
in which indicators are expected to have a positive effect on the decision
probability for ERC grant applications
does not rely on performance measures  combination of content
analysis and scientometric methods using textual information present in the
proposals
can be reused  focused on the ERC grant scheme but the concept might
be applicable more generally
results in a methodology  allowing the grant agency to monitor, from a
bibliometric perspective, the peer-review process and providing a basis for its
further refinement
17
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
ISSI 2011 - Durban, South Africa, July 4-7
Acknowledgements
This work was partially funded by the « Ideas » specific Programme of the
EU’s 7th Framework Programme for Research and Technological
Development (project reference no. 240765)
Project website: http://www.ait.ac.at/dbf
18
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
ISSI 2011 - Durban, South Africa, July 4-7
Thank you
ouy knahT
[dirk.holste; edgar.schiebel; thomas.scherngell; marianne.horlesberger]@ait.ac.at
[ivana.roche; dominique.besagni; claire.francois; pascal.cuxac]@inist.fr
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
19
ISSI 2011 - Durban, South Africa, July 4-7
20
ISSI 2011 - A concept for inferring "Frontier Research" in research project proposals - Durban, South Africa, July 4-7
Timeliness indicator (2/2)
ERC
database
Research
project
proposal
Extraction of
publication date
of references
Empirical
distribution of
publication
dates
Calculation of
TIMELINESS
indicator
Data pre-processing and text-mining
21
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
ISSI 2011 - Durban, South Africa, July 4-7
Proximity indicator (2/2)
Panel
description
Bibliographic
(PASCAL)
database
query
T1
Construction of two
indexed corpora
(time window T1, T2)
DB of
references
DB of
references
Ranking of
Diachronic
clusters by
cluster
analysis T1,T2 innovativeness
T2
ERC
database
Data
from
proposals
Extraction of
terminological
information
Position of
proposal in T2
cluster map
Calculation of
PROXIMITY
indicator
ERC
database
Translate main
panels into
database queries
Data pre-processing and text-mining
22
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
ISSI 2011 - Durban, South Africa, July 4-7
Risk indicator (2/2)
Distance between
references (d1)
ERC
database
References of PI’s
research proposal
External
databases
Reference profile in
the field of the
submitted proposal
Calculation of
RISK
indicator
References of PI’s
past research
Distance between
references (d2)
Data pre-processing and parsing
23
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
ISSI 2011 - Durban, South Africa, July 4-7
Pasteuresqueness indicator (2/2)
ERC
database
Date pertinent
to PI’s
curriculum vitae
Grants, funding,
acknowledgments of
collaborations
List of selfcitations (PI)
List of journals
publishing
External
databases
Journals
and
their scopes
Calculation of
PASTEURESQUENESS
indicator
Patents
submitted
by/granted to PI
Patents citing
PI’s work
Classification of
“fundamental” and
“applied” publications
Data pre-processing and text-mining
24
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
ISSI 2011 - Durban, South Africa, July 4-7
Extraction
of ERC
panel
structure
ERC
database
Extraction of
keywords
from
proposals
Matching of
proposals with panels
and analysis of field
diversity
Matching
keywords to
some “home
discipline”
Analysis of
concentration of
“home keywords”
in proposals
Calculation of
INTERDISCIPLINARITY
indicator
Interdisciplinarity indicator (2/2)
Data pre-processing and text-mining
25
A concept for inferring "Frontier Research" in research project proposals - Holste et al.
ISSI 2011 - Durban, South Africa, July 4-7