Transcript Session_1_-_Peter_Fisch

Evaluation of the
Framework Programme
Tools and Challenges
EuroCRIS Seminar
Brussels, 13 September 2010
Dr. Peter Fisch
European Commission
1
Roadmap
European Research Evaluation Network
Mandate, Composition, Activities
Monitoring FP7
Third Monitoring Report
Evaluation of FP7
Interim Evaluation of FP7
13/09/2010
2
European Research
Evaluation Network
•
Discussion forum on RTD evaluation
•
Established in 1997
•
Meetings twice a year, usually in the country of
EU presidency
•
Bottom-up agenda setting
3
European Research
Evaluation Network
Composition
•
EU Member States, Candidate Countries,
Associated Countries
•
Two members per country
• “academia”/“administration”
• “supply”/”demand”
•
Nominated by National Governments
•
Renewal in regular intervals
4
European Research
Evaluation Network
Activities
•
Exchange of information on European and
National initiatives
•
Presentations on “novel” approaches
•
Examples:
• Contribution to FP evaluation work
• Long-term impact studies
• Sharing of “local” experiences
5
FP7 Monitoring System

Move from “ad hoc” campaigns using external
experts (FP6) towards a systematic internal
monitoring (FP7)

Annual analysis based on a core set of
indicators

Flexible system to develop as FP7 will become
more “mature” (outputs)

Important information source for FP7
evaluations (notably the ongoing interim
evaluation)
6
FP7 Monitoring
Structure of 2009 Report
•
FP7 Implementation Overview
• Data / indicators on key aspects
•
FP7 Implementation Special Focus
• Novelties (ERC, JTI, Art 185, RSFF)
• Selected fields (International, Sustainable
Development, Marie Curie, EURATOM)
•
Simplification
• Measures taken
• NCP Survey
•
Achievements
• Very first findings
7
FP7 Monitoring
Core Indicators (1)
Promotion of FP7
1.1
Number of attendees at launch days
1.2
Number of information days
1.3
Commission organised meetings of NCPs
Performance of the calls
2.1
Success rate (overall) by priority area and funding scheme
2.2
Success rate for different types of organisation by priority area and
funding scheme
2.3
Success rate for different types of organisation by priority area and
funding scheme & success rates per country
Performance of the proposal evaluation and redress procedures
3.1
Overall quality assessment of the proposal evaluators on the FP proposal
evaluation process (evaluators survey)
3.2
Assessment of quality by the evaluators between the FP evaluation
process and other equivalent systems (evaluators survey)
3.3
Time to contract/grant
3.4
Percentage of experts reimbursed within the specified 45 days
3.5
Redress cases upheld (i.e. leading to a re-evaluation) – numbers and
percentages
Quality of on-going research projects
4.1
Average results of independent project review process by priority area
4.2
Percentage of projects by priority area covered by reviews
8
FP7 Monitoring
Core Indicators (2)
Project performance by outputs
5.1
Average number of project publications per project by priority area and
funding scheme
5.2
Average number of other forms of dissemination activities per project by
priority area and funding scheme
5.3
Average number of different types of intellectual property protection per
project by priority area and funding scheme
FP activity
6.1
Total number of active projects by priority area
6.2
Average financial size of projects by priority area and funding scheme
6.3
Participation by types of organisation by priority area funding scheme
6.4
Participation totals per country
Achieving gender equality
7.1
Number of male and female coordinators in proposals
7.2
Number of male and female coordinators in projects
7.3
Gender breakdown (by seniority) of project participants
7.4
Percentage of male and female members in Advisory Groups and
Programme Committees
9
FP7 Monitoring
Core Indicators (3)
Observing sound ethical principles in FP research
8.1
Number of projects going through the review process/ % by area/
programme
8.2
Number of ethical reviews where the result showed sufficient or
insufficient attention had been given
8.3
Number of projects stopped as a results of the ethical review
8.4
Number of screenings by services
Performance of International Cooperation activities
9.1
Total numbers of participations of 3rd countries by priority area and
funding scheme
9.2
Success rates of 3rd countries in calls by priority area and funding
scheme
9.3
EC contribution to 3rd countries
9.4
Number of international outgoing / incoming fellowships
Simplification of the FP
10.1 Do stakeholders perceive that the FP is getting simpler to use in terms of
financial and administrative procedures?
10.2 How do stakeholders find the ease of use of the FP compared to similar
international research actions and large national schemes?
10.3 Are there any aspects of FP procedures which are adversely affecting to
a significant extent the quality of research carried out and the quality of
participation in the FP?
10
FP7 Monitoring
Key Data (1)
Absolute figures (2007 - 2009):
• 41.000 proposals received
• 234.000 applicants
• 9.100 proposals retained
• 51.000 participants
• 15 billion € EU contribution
11
FP7 Monitoring
Key Data (2)
Organisations:
• Universities 30%
• “Industry” 25%
• Research Organisations 23%
Gender:
• 20.5% female “contact persons for scientific
aspects”
• 36.1% female “fellows” in Marie Curie actions
12
WOMEN IN CONTACT PERSON ROLES
Key Data (3)
Gene
ral
Society
Policies
SSH
INCO
Health
Regions
Food
Potential
Environment
ERC
Marie Curie
Fission
Infrastructures
Nanotech
JTIs
Fusion
Security
SMEs
Energy
Space
Transport
0,0%
5,0%
10,0%
15,0%
20,0%
25,0%
30,0%
35,0%
40,0%
45,0%
50,0%
Contact Person Contact Person for Scientific Aspects
13
FP Evaluation System
Basics
• Embedded in the Commission Evaluation system
• Evaluations to be carried out by the services
responsible for an activity as part of the
management responsibilities
• Multi-layer system consisting of thematic
evaluations at programme level, studies to analyse
general issues and evaluations at FP level
• Expert Group Report as “top of the iceberg”
14
FP6 Ex-Post Evaluation
Groundwork
• Monitoring, Project Database (CORDA)
• Output indicators
• Self assessments
• Thematic evaluation studies and reports
• Horizontal evaluation studies
• National Impact Assessments
• Feedbacks, surveys
13/09/2010
• …
15
FP7 Interim Evaluation
Basics
• FP7 Decision:
– Interim Evaluation “no later than” 2010
• To cover FP as a whole
• Specific reviews in some areas
– (ERC, RSFF, INFSO …)
• To be carried out by a group of external experts
• Meetings from March to September 2010
• Final Report expected in October 2010
13/09/2010
16
FP7 Interim Evaluation
Expert Group
Name
First Name
Nationality
Gender
ACHESON
Helena
IE
F
ANNERBERG
Rolf
SE
M
Chair
BEGG
Iain
UK
M
Rapporteur
BORRÁS
Susana
ES
F
HALLÉN
Arvid
NO
M
MAIMETS
Toivo
EE
M
MUSTONEN
Riitta
FI
F
RAFFLER
Hartmut
DE
M
SWINGS
Jean-Pierre
BE/USA
M
YLIHONKO
Kristiina
FI
F
13/09/2010
17
FP7 Interim Evaluation
Key Questions
• General objectives achieved?
• How to improve impact of FP on ERA and other
policies?
• FP7 role in positioning Europe on the global
map?
• Efficiency of novel measures (ERC, JTI, ...)?
• How to better address interdisciplinary “grand
challenges”?
• Simplification measures effective?
• Progress on issues raised in FP6 evaluation?
13/09/2010
18
Contact
Dr. Peter Fisch
Head of Unit “Evaluation and Monitoring of programmes”
European Commission – DG Research A.3
SDME 2/41
1049 Brussels
[email protected]
http://ec.europa.eu/research/evaluations
19