Transcript Tips for Using This Template

Building a Scientific Basis for
Research Evaluation
Rebecca F. Rosen, PhD
Copyright © 2012
American Institutes
for Research.
All rights reserved.
Senior Researcher
Research Trends Seminar
October 17, 2012
Outline
• Science of science policy
• A proposed conceptual framework
• Empirical approaches:
 NSF Engineering Dashboard
 ASTRA – Australia
 HELIOS – France
• Final thoughts
2
Outline
• Science of science policy
• A proposed conceptual framework
• Empirical approaches:
 NSF Engineering Dashboard
 ASTRA – Australia
 HELIOS – France
• Final thoughts
3
The emergence of a science of
science policy
• Jack Marburger’s challenge (2005)
• Science of Science & Innovation Policy Program at the
National Science Foundation (2007)
 An emerging, highly interdisciplinary research field
• Science of Science Policy Interagency Task Group
publishes a “Federal Research Roadmap” (2008):
 The data infrastructure is inadequate for decisionmaking
• STAR METRICS (2010)
4
Why a science of science policy?
• Evidence-based investments
• Good metrics = good incentives
• Science is networked and global
• Build a bridge between researchers and
policymakers
• Researchers ask the right questions
• The adjacent possible: leverage existing
and new research and expertise
• New tools to describe & measure communication
5
The timing is right:
6
A conceptual framework for a
science of science policy
7
Getting the right framework matters
• What you measure is what you get
 Poor incentives
 Falsification
• Usefulness
• Effectiveness
8
A proposed conceptual framework
9
Adapted from Ian Foster, University of Chicago
A framework to drive person-centric
data collection
WHO is doing the research
WHAT is the topic of their research
HOW are the researchers funded
WHERE do they work
With WHOM do they work
What are their PRODUCTS
10
Challenge – The data infrastructure
didn’t exist
However, some of the data do exist
11
Empirical Approaches
Leveraging existing data to begin
describing results of the scientific
enterprise
12
An empirical approach
• Enhance the utility of enterprise data
• Identify authoritative “core” data elements
• Develop an Application Programming
Interface (API)
 Data platform that provides programmatic
access to public (or private) agency
information
• Develop a tool to demonstrate value of
API
13
Topic modeling: Enhancing the value
of existing data
Automatically learned topics (e.g.):
…
t6. conflict violence war international military …
t7. model method data estimation variables …
t8. parameter method point local estimates …
t9. optimization uncertainty optimal stochastic …
t10. surface surfaces interfaces interface …
NSF
proposals
Topic Model:
- Use words from
(all) text
- Learn T topics
t11. speech sound acoustic recognition human …
t12. museum public exhibit center informal outreach
t13. particles particle colloidal granular material …
t14. ocean marine scientist oceanography …
…
t49
t18
t114
t305
14
David Newman - UC Irvine
Topic tags for
each and every
proposal
Stepwise empirical approach
• Enhance the utility of enterprise data
• Identify authoritative “core” data elements
• Develop an Application Programming
Interface (API)
 Data platform that provides flexible,
programmatic access to public (or private)
agency information
• Develop a tool to demonstrate value of
API
15
16
Stepwise empirical approach
• Enhance the utility of enterprise data
• Identify authoritative “core” data elements
• Develop an Application Programming
Interface (API)
 Data platform that provides programmatic
access to public (or private) agency
information
• Develop a tool to demonstrate value of
API
17
Outline
• Science of science policy
• A proposed conceptual framework
• Empirical approaches:
 NSF Engineering Dashboard
 ASTRA – Australia
 HELIOS – France
• Final thoughts
18
19
20
21
22
23
Outline
• Science of science policy
• A proposed conceptual framework
• Empirical approaches:
 NSF Engineering Dashboard
 ASTRA – Australia
 HELIOS – France
• Final thoughts
24
Linking administrative and grant
funding data in Australia
25
Outline
• Science of science policy
• A proposed conceptual framework
• Empirical approaches:
 NSF Engineering Dashboard
 ASTRA – Australia
 HELIOS – France
• Final thoughts
26
Describing public-private
partnerships in France
People
People
27
What does getting it right mean?
• A community driven empirical data
framework should be:
 Timely
 Generalizable and replicable
 Low cost, high quality
• The utility of “Big Data”:
 Disambiguated data on individuals
- Comparison groups
 New text mining approaches to describe
and measure communication
 ??
28
Final thoughts
29
Policy makers can engage SciSIP
communities:
•
Patent Network Dataverse; Fleming at Harvard and Berkeley
•
Medline-Patent Disambiguation; Torvik & Smalheiser at U
Illinois)
•
COMETS (Connecting Outcome Measures in Entrepreneurship
Technology and Science); Zucker & Darby at UCLA
30
The power of open research
communities
• Internet and data technology can
transform effectiveness of science:
 Informing policy
 Communicating science to the public
 Enabling scientific collaborations
• Interoperability is key
• Publishers are an important part of the
community
31
THANK YOU!
Rebecca F. Rosen, PhD
E-Mail: [email protected]
1000 Thomas Jefferson Street NW
Washington, DC 20007
General Information: 202-403-5000
TTY: 887-334-3499
Website: www.air.org
32
33