Transcript Thomas KARL, American Meteorological Society

Ensuring scientific integrity
of peer-review articles
related to underlying data
and analysis
IFMS Global Meeting 1
In association with the 90th Annual AMS Meeting
Atlanta, Georgia - USA
19 January 2010
Thomas R. Karl, L.H.D.
President, American Meteorological Society
&
Director of the National Climatic Data Center
National Oceanic and Atmospheric Administration
[email protected]
NAS Workshop on
IFMS Global Meeting 1
Data Integration
Outline
• Issue Context:
• Stolen emails from Univ. of East Anglia Climate Research Unit
• Should we preserve underlying data and
software and if so how?
Ensuring scientific integrity of peer-review articles related to underlying data and analysis
2
NAS Workshop on
IFMS Global Meeting 1
Data Integration
Issue Context
Stolen emails from Univ. of East Anglia Climate Research Unit
• Raised the following issues in regard to the integrity of peerreview science:
• Availability of underlying original data used in the article
• Availability of software and algorithms used to transform
the original data into products used in the article
Ensuring scientific integrity of peer-review articles related to underlying data and analysis
3
NAS Workshop on
IFMS Global Meeting 1
Data Integration
Should we preserve underlying data and
software and if so how?
Scientific credibility: rests on reproducibility of results
• Two approaches
1. Responsibility for reproducibility lies with the scientific
process of repeatability of results
2. Responsibility for reproducibility lies with the scientific
journal policies
• A number of implications
Ensuring scientific integrity of peer-review articles related to underlying data and analysis
4
NAS Workshop on
IFMS Global Meeting 1
Data Integration
Should we preserve underlying data and
software and if so how?
Approach 1: Responsibility for reproducibility lies with the scientific
process of repeatability of results
U.S. Climate Change
Science Program
Strategic Plan
July 2003
“..actual accuracy is only known after
comparison of independent measurements
and analysis from multiple laboratories.”
“Each climate variable will be measured
using independent observations and
examined with independent data analysis”
http://www.climatescience.gov/Library/stratplan2003/def
ault.htm
AMS Policy statement
Impact of CRU Hacking on the AMS
Statement on Climate Change
November 2009
“The beauty of science is that it depends on
independent verification and replication as
part of the process of confirming research
results.”
This process …is tied intrinsically to the
procedures leading to publication …in the
peer-reviewed literature…”
http://www.ametsoc.org/policy/climatechangeclarify.html
Ensuring scientific integrity of peer-review articles related to underlying data and analysis
5
NAS Workshop on
IFMS Global Meeting 1
Data Integration
Should we preserve underlying data and
software and if so how?
Approach 2: Responsibility for reproducibility lies with the scientific
journal policies
• Issues:
• What preservation policies should journals have?
•
International, national or journal specific?
• What underlying data should be preserved?
•
Original data, reprocessed data, associated metadata
• What underlying software should be preserved?
•
Standards for software, version control
• How should the underlying data and software be
preserved?
•
•
•
By the Journal, Principal Investigator, a Data Center?
How long to preserve
Who pays (provider or requestor)?
Ensuring scientific integrity of peer-review articles related to underlying data and analysis
6
NAS Workshop on
IFMS Global Meeting 1
Data Integration
Should we preserve underlying data and
software and if so how?
• The AMS scientific journals currently do not have a policy on
preserving underlying data and software used in an article
• AMS does have the infrastructure in place to host a permanent
repository for data and software (called Electronic Supplement (ES))
• The journal Science does have a policy:
All data necessary to understand, assess, and extend the conclusions of the
manuscript must be available to any reader of /Science/. After publication, all
reasonable requests for materials must be fulfilled. Any restrictions on the availability of data or materials, including
fees and original data obtained from other sources (Materials Transfer Agreements), must be disclosed to the
editors upon submission. Fossils or other rare specimens must be deposited in a public museum or repository and
available for research.
/Science/ supports the efforts of databases that aggregate published data for the use of the scientific community.
Therefore, appropriate data sets (including microarray data, protein or DNA sequences, atomic coordinates or
electron microscopy maps for macromolecular structures, and climate data) must
be deposited in an
approved database, and an accession number or a specific access address
must be included in the published paper. We encourage compliance with MIBBI guidelines
(Minimum Information for Biological and Biomedical Investigations).
Ensuring scientific integrity of peer-review articles related to underlying data and analysis
7
NAS Workshop
on
Integration
of Model
IFMS Global Meeting 1
Data Integration
Output
Workshop
Discussion
Ensuring scientific integrity of peer-review articles related to underlying data and analysis
8