Reproducibility
and Replicability
in Science
Committee on Reproducibility and Replicability in Science
Board on Behavioral, Cognitive, and Sensory Sciences
Committee on National Statistics
Division of Behavioral and Social Sciences and Education
Nuclear and Radiation Studies Board
Division on Earth and Life Studies
Board on Mathematical Sciences and Analytics
Committee on Applied and Theoretical Statistics
Division on Engineering and Physical Sciences
Board on Research Data and Information
Committee on Science, Engineering, Medicine, and Public Policy
Policy and Global Affairs
A Consensus Study Report of
THE NATIONAL ACADEMIES PRESS
Washington, DC
www.nap.edu
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
This activity was supported by contracts between the National Academy of Sciences and Alfred P. Sloan Foundation (G-2018-10102) and the National Science Foundation (1743856). Support for the work of the Board on Behavioral, Cognitive, and Sensory Sciences is provided primarily by a grant from the National Science Foundation (Award No. BCS-1729167). Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project.
International Standard Book Number-13: 978-0-309-48616-3
International Standard Book Number-10: 0-309-48616-5
Digital Object Identifier: https://doi.org/10.17226/25303
Library of Congress Control Number: 2019946492
Additional copies of this publication are available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
Copyright 2019 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: National Academies of Sciences, Engineering, and Medicine. (2019). Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. https://doi.org/10.17226/25303.
The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president.
The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. C. D. Mote, Jr., is president.
The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.
The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The National Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.
Learn more about the National Academies of Sciences, Engineering, and Medicine at www.nationalacademies.org.
Consensus Study Reports published by the National Academies of Sciences, Engineering, and Medicine document the evidence-based consensus on the study’s statement of task by an authoring committee of experts. Reports typically include findings, conclusions, and recommendations based on information gathered by the committee and the committee’s deliberations. Each report has been subjected to a rigorous and independent peer-review process and it represents the position of the National Academies on the statement of task.
Proceedings published by the National Academies of Sciences, Engineering, and Medicine chronicle the presentations and discussions at a workshop, symposium, or other event convened by the National Academies. The statements and opinions contained in proceedings are those of the participants and are not endorsed by other participants, the planning committee, or the National Academies.
For information about other products and activities of the National Academies, please visit www.nationalacademies.org/about/whatwedo.
COMMITTEE ON REPRODUCIBILITY AND REPLICABILITY IN SCIENCE
HARVEY V. FINEBERG1 (Chair), Gordon and Betty Moore Foundation
DAVID B. ALLISON,1 Indiana University, Bloomington
LORENA A. BARBA, The George Washington University
DIANNE CHONG,2 Boeing Research and Technology (retired)
DAVID DONOHO,3,4 Stanford University
JULIANA FREIRE, New York University
GERALD GABRIELSE,3 Northwestern University
CONSTANTINE GATSONIS, Brown University
EDWARD HALL, Harvard University
THOMAS H. JORDAN,3 University of Southern California
DIETRAM A. SCHEUFELE, University of Wisconsin–Madison
VICTORIA STODDEN, University of Illinois at Urbana–Champaign
SIMINE VAZIRE,5 University of California, Davis
TIMOTHY D. WILSON, University of Virginia
WENDY WOOD, University of Southern California and INSEAD-Sorbonne
JENNIFER HEIMBERG, Study Director
THOMAS ARRISON, Program Director
MICHAEL COHEN, Senior Program Officer
MICHELLE SCHWALBE, Director, Board on Mathematical Sciences and Analytics
ADRIENNE STITH BUTLER, Associate Board Director
BARBARA A. WANCHISEN, Director, Board on Behavioral, Cognitive, and Sensory Sciences
TINA WINTERS, Associate Program Officer
REBECCA MORGAN, Senior Librarian
THELMA COX, Program Coordinator (beginning January 2019)
LESLEY WEBB, Program Assistant (September 2018 through January 2019)
GARRET TYSON, Program Assistant (September 2017 through August 2018)
ERIN HAMMERS FORSTAG, Consultant Writer
___________________
1 Member of the National Academy of Medicine.
2 Member of the National Academy of Engineering.
3 Member of the National Academy of Sciences.
4 Resigned from the committee July 24, 2018.
5 Resigned from the committee October 11, 2018.
BOARD ON BEHAVIORAL, COGNITIVE, AND SENSORY SCIENCES
Division of Behavioral and Social Sciences and Education
SUSAN FISKE1 (Chair), Princeton University
JOHN BAUGH, Washington University in St. Louis
LAURA CARTENSEN,2 Stanford University
JUDY DUBNO, Medical University of South Carolina
JENNIFER EBERHARDT,1 Stanford University
WILSON S. GEISLER,1 The University of Texas at Austin
MICHELE GELFAND, University of Maryland, College Park
NANCY G. KANWISHER,1 Massachusetts Institute of Technology
JANICE KIECOLT-GLASER,2 The Ohio State University College of Medicine
WILLIAM (BILL) MAURER, University of California, Irvine
TERRIE E. MOFFITT,2 Duke University
ELIZABETH A. PHELPS, Harvard University
STEVEN E. PETERSEN, Washington University in St. Louis
DANA SMALL, Yale University
TIMOTHY J. STRAUMAN, Duke University
BARBARBA A. WANCHISEN, Director
___________________
1 Member of the National Academy of Sciences.
2 Member of the National Academy of Medicine.
COMMITTEE ON NATIONAL STATISTICS
Division of Behavioral and Social Sciences and Education
ROBERT M. GROVES1,2 (Chair), Georgetown University
MARY ELLEN BOCK, Purdue University
ANNE C. CASE,1 Princeton University
MICHAEL E. CHERNEW,1 Harvard Medical School
JANET M. CURRIE,1 Princeton University
DONALD A. DILLMAN, Washington State University
DIANA FARRELL, JPMorgan Chase Institute
DANIEL KIFER, The Pennsylvania State University
THOMAS L. MESENBOURG, U.S. Census Bureau (retired)
SARAH M. NUSSER, Iowa State University
COLM O’MUIRCHEARTAIGH, University of Chicago
JEROME P. REITER, Duke University
ROBERTO RIGOBON, Massachusetts Institute of Technology
JUDITH A. SELTZER, University of California, Los Angeles
C. MATTHEW SNIPP, Stanford University
BRIAN HARRIS-KOJETIN, Director
CONSTANCE F. CITRO, Senior Scholar
___________________
1 Member of the National Academy of Medicine.
2 Member of the National Academy of Sciences.
NUCLEAR AND RADIATION STUDIES BOARD
Division on Earth and Life Studies
GEORGE APOSTOLAKIS1 (Chair), Massachusetts Institute of Technology
JAMES A. BRINK (Vice Chair), Massachusetts General Hospital
SALLY A. AMUNDSON, Columbia University Medical Center
STEVEN M. BECKER, Old Dominion University
AMY J. BERRINGTON DE GONZÁLEZ, National Cancer Institute
PAUL T. DICKMAN, Argonne National Laboratory
TISSA H. ILLANGASEKARE, Colorado School of Mines
CAROL M. JANTZEN, Savannah River National Laboratory (retired)
BONNIE D. JENKINS, The Brookings Institution
ALLISON M. MACFARLANE, The George Washington University
NANCY JO NICHOLAS, Los Alamos National Laboratory
R. JULIAN PRESTON, Environmental Protection Agency
HENRY D. ROYAL, Washington University School of Medicine in St. Louis
WILLIAM H. TOBEY, Harvard University
SERGEY V. YUDINTSEV, Russian Academy of Sciences
CHARLES FERGUSON, Director
___________________
1 Member of the National Academy of Engineering.
BOARD ON MATHEMATICAL SCIENCES AND ANALYTICS
Division on Engineering and Physical Sciences
STEPHEN M. ROBINSON1 (Chair), University of Wisconsin–Madison
JOHN R. BIRGE,1 University of Chicago
W. PETER CHERRY,1 Independent Consultant, Ann Arbor, MI
DAVID S. C. CHU, Institute for Defense Analyses
RONALD R. COIFMAN,2 Yale University
JAMES H. CURRY, University of Colorado
MARK L. GREEN, University of California, Los Angeles
SHAWNDRA HILL, Microsoft Research
LYDIA KAVRAKI,3 Rice University
TAMARA KOLDA, Sandia National Laboratories
JOSEPH A. LANGSAM, University of Maryland, College Park
DAVID MAIER, Portland State University
LOIS C. McINNES, Argonne National Laboratory
JILL C. PIPHER, Brown University
ELIZABETH A. THOMPSON,1 University of Washington
CLAIRE J. TOMLIN,1 University of California, Berkeley
LANCE A. WALLER, Emory University
KAREN WILLCOX, The University of Texas at Austin
DAVID D. YAO,1 Columbia University
MICHELLE K. SCHWALBE, Director
___________________
1 Member of the National Academy of Engineering.
2 Member of the National Academy of Sciences.
3 Member of the National Academy of Medicine.
COMMITTEE ON APPLIED AND THEORETICAL STATISTICS
Division on Engineering and Physical Sciences
ALFRED O. HERO, III (Chair), University of Michigan
KATHERINE BENNETT ENSOR, Rice University
ALICIA L. CARRIQUIRY,1 Iowa State University
RONG CHEN, Rutgers University
MICHAEL J. DANIELS, University of Florida
AMY H. HERRING, Duke University
TIM HESTERBERG, Google, Inc.
NICHOLAS HORTON, Amherst College
DAVID MADIGAN, Columbia University
XIAO-LI MENG, Harvard University
JOSÉ M.F. MOURA,2 Carnegie Mellon University
RAQUEL PRADO, University of California, Santa Cruz
NANCY M. REID,3 University of Toronto
CYNTHIA RUDIN, Duke University
AARTI SINGH, Carnegie Mellon University
ALYSON GABBARD WILSON, North Carolina State University
BENJAMIN WENDER, Director
___________________
1 Member of the National Academy of Medicine.
2 Member of the National Academy of Engineering.
3 Member of the National Academy of Sciences.
BOARD ON RESEARCH DATA AND INFORMATION
Policy and Global Affairs
ALEXA MCCRAY1 (Chair), Harvard Medical School
AMY BRAND, Massachusetts Institute of Technology
STUART FELDMAN, Schmidt Philanthropies
SALMAN HABIB, Argonne National Laboratory
JAMES HENDLER, Rensselaer Polytechnic Institute
ELLIOT MAXWELL, e-Maxwell and Associates
BAREND MONS, Leiden University Medical Center
SARAH M. NUSSER, Iowa State University
MICHAEL STEBBINS, Science Advisors, LLC
BONNIE CARROLL, Information International Associates (CODATA Secretary General)
JOHN HILDEBRAND,2 University of Arizona (NAS Foreign Secretary)
PAUL UHLIR, Data Policy and Management (CODATA Executive Committee Member)
GEORGE STRAWN, Director
___________________
1 Member of the National Academy of Medicine.
2 Member of the National Academy of Sciences.
COMMITTEE ON SCIENCE, ENGINEERING, MEDICINE, AND PUBLIC POLICY
Policy and Global Affairs
ALAN I. LESHNER (2020)1 (Chair), American Association for the Advancement of Science
Ex-Officio Members:
VICTOR J. DZAU1 (2020), President, National Academy of Medicine
MARCIA McNUTT2 (2022), President, National Academy of Sciences
C. D. (DAN) MOTE, JR.3 (2019), President, National Academy of Engineering
Members:
CYNTHIA BARNHART,3 Massachusetts Institute of Technology
CLAIRE D. BRINDIS,1 University of California, San Francisco
DAVID E. DANIEL,3 University of Texas at Dallas
KATHARINE G. FRASE,3 IBM Corporation (retired)
JOHN G. HILDEBRAND,2 University of Arizona
DAVID KORN,1 Harvard Medical School
RICHARD A. MESERVE,3 Covington & Burling LLP
J. SANFORD SCHWARTZ,1 University of Pennsylvania
CHRISTOPHER A. SIMS,2 Princeton University
ROBERT F. SPROULL,3 University of Massachusetts at Amherst
JAMES M. TIEN,2 University of Miami
ZENA WERB,1,2 University of California, San Francisco
MICHAEL S. WITHERELL,2 Lawrence Berkeley National Laboratory
SUSAN M. WOLF,1 University of Minnesota
PETER G. WOLYNES,2 Rice University
ANNE-MARIE MAZZA, Senior Director
___________________
1 Member of the National Academy of Medicine.
2 Member of the National Academy of Sciences.
3 Member of the National Academy of Engineering.
Acknowledgments
This Consensus Study Report reflects the invaluable contributions of many individuals, including those who served on the committee, the staff of the National Academies of Sciences, Engineering, and Medicine, and many other experts. This report was made possible by funding from the National Science Foundation and sponsorships from the Alfred P. Sloan Foundation. We thank Daniel Goroff of the Alfred P. Sloan Foundation for supporting the study financially to allow for commissioned papers, expanded dissemination activities, and providing insight to the committee.
The committee is also grateful for the efforts of the following authors who prepared background papers for the committee’s use in drafting the report:
- Rosemary Bush, Perspectives on Reproducibility and Replication of Results in Climate Science
- Emily Howell, Public Perceptions of Scientific Uncertainty and Media Reporting of Reproducibility and Replication in Science
- Xihong Lin, Reproducibility and Replicability in Large Scale Genetic Studies
- Anne Plant and Robert Hanisch, Reproducibility and Replicability in Science, a Metrology Perspective
- Lars Vilhuber, Reproducibility and Replicability in Economics
This Consensus Study Report was reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise. The purpose of this independent review is to provide candid and critical comments that
will assist the institution in making its published report as sound as possible and to make certain that the report meets institutional standards for objectivity, evidence, and responsiveness to the charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process.
We thank the following individuals for their review of this report: R. Stephen Berry, Department of Chemistry and James Franck Institute, The University of Chicago; Kenneth Bollen, Department of Psychology and Neuroscience and Department of Sociology, University of North Carolina at Chapel Hill; Mary Sue Coleman, Office of the President, Association of American Universities; David L. Donoho, Department of Statistics, Stanford University; Stuart Firestein, Department of Biological Sciences, Columbia University; Steven N. Goodman, Division of Epidemiology and Clinical Research, Stanford University School of Medicine; Paul L. Joskow, Alfred P. Sloan Foundation and Department of Economics, Massachusetts Institute of Technology; Louis J. Lanzerotti, Department of Physics, New Jersey Institute of Technology; Don Monroe, science and technology writer, Massachusetts; Brian Nosek, Department of Psychology and Center for Open Science, University of Virginia; Roger D. Peng, Bloomberg School of Public Health, Johns Hopkins University; Gianluca Setti, Department of Electronics and Telecommunications, Polytecnic of Turin; and Eric-Jan Wagenmakers, Department of Psychological Methods, University of Amsterdam.
Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the content of the report nor did they see the final draft of the report before its release. The review of this report was overseen by Robert M. Groves, Department of Mathematics and Statistics, Department of Sociology, and Office of the Provost, Georgetown University, and Julia M. Phillips, Sandia National Laboratories (retired). They were responsible for making certain that an independent examination of this report was carried out in accordance with the standards of the National Academies and that all review comments were carefully considered. Responsibility for the final content rests entirely with the authoring committee and the National Academies.
Preface
When Sleeper filled theaters in 1973, stick margarine was widely advertised as the healthier alternative to butter. In just a couple of decades, evidence began to accumulate that partially hydrogenated (trans) fats found in hard margarines were worse for heart health than the saturated fat found in butter (not that either is particularly heart healthy). More recently, adults who for years have been ingesting daily doses of baby aspirin, with the aim of reducing the risk of heart attack, are now being advised not to bother. The latest studies failed to confirm earlier findings that had suggested real benefits from daily aspirin. Perhaps we should caution Woody Allen’s Dr. Aragon to be a little less certain of “what we now know to be true.”
Science is based on a conviction that the natural world adheres to certain principles, grounded in an underlying and consistent reality. However, human capacity to discern those truths of nature, including human
behavior, is imperfect. We rely on science to reveal what is knowable of nature, and typically, that knowledge has some level of uncertainty attached to it. Repeated findings of comparable results tend to confirm the veracity of an original scientific conclusion, and, by the same token, repeated failures to confirm throw the original conclusions into doubt. When a scientific study becomes the basis of policy or has a direct or indirect impact on human well-being, scientific reliability becomes more than an academic question.
This Consensus Study was prompted by concerns about the reproducibility and replicability of scientific research. The National Science Foundation (NSF) had entered into discussions with the National Academies of Sciences, Engineering, and Medicine about a study on reproducibility and replicability in the social sciences when Congress enacted a provision of law that expanded the scope of the study to all science and engineering. The Alfred P. Sloan Foundation then joined in support of this work, with special interest in the efficiency of scientific research, and aided in the dissemination of the findings, conclusions, and recommendations of the study.
To carry out the task, the National Academies appointed a committee of 15 members representing a wide range of expertise: methodology and statistics, philosophy of science, science communication, behavioral and social sciences, earth and life sciences, physical sciences, computational science, engineering, academic leadership, journal editors, and industry expertise in quality control. Individuals with expertise pertaining to reproducibility and replicability of research results across a variety of fields were included as well. In conducting its study, the committee reviewed the research literature on reproducibility and replicability, held 12 meetings at which it heard from a wide range of stakeholders in the research enterprise and deliberated to reach the findings, conclusions, and recommendations presented in this report.
I have had the privilege of chairing this diverse panel of experts, and I thank all of the members of the committee for their intensive effort and collaborative spirit in crafting this report. We were aided by a remarkably talented study director, Jennifer Heimberg, and an able group of staff, including Thomas Arrison, Adrienne Stith Butler, Michelle Schwalbe, Tina Winters, Michael Cohen, Rebecca Morgan, Thelma Cox, Lesley Webb, and Garret Tyson. We also offer special thanks to Erin Hammers Forstag, who served as consultant writer on this project, and Eugenia Grohman, who edited earlier versions of this manuscript. We are most grateful to NSF and to the Alfred P. Sloan Foundation for their generous support of this undertaking.
We hope the ideas and guidance offered here prove useful to Congress, public and private funders of scientific research, scientists and research institutions, journal editors and authors, and the interested public. Science and technology shape our world in both dramatic and mundane ways. We all have a stake in ensuring that scientists adhere to the highest standards of practice, understand and express the uncertainty inherent in their conclusions, and continue to strengthen the interconnected web of scientific knowledge—the principal driver of progress in the modern world.
Harvey V. Fineberg, Chair
Committee on Reproducibility and Replicability in Science
This page intentionally left blank.
Contents
2 SCIENTIFIC METHODS AND KNOWLEDGE
Core Principles and Assumptions of Scientific Inquiry
Science Is a Communal Enterprise
Science Aims for Refined Degrees of Confidence, Rather Than Complete Certainty
Scientific Knowledge Is Durable and Mutable
Statistical Inference and Hypothesis Testing
3 UNDERSTANDING REPRODUCIBILITY AND REPLICABILITY
The Evolving Practices of Science
Defining Reproducibility and Replicability
Widespread Use of Computational Methods
Resources and Costs of Reproducibility
The Extent of Non-Reproducibility
Sources of Non-Reproducibility
Obsolescence of Digital Artifacts
Flawed Attempts to Reproduce Others’ Research
Barriers in the Culture of Research
The Extent of Non-Replicability
Perspectives of Researchers Who Have Studied Replicability
Non-Replicability That Is Potentially Helpful to Science
Unhelpful Sources of Non-Replicability
Inappropriate Statistical Inference
Incomplete Reporting of a Study
6 IMPROVING REPRODUCIBILITY AND REPLICABILITY
Strengthening Research Practices: Broad Efforts and Responsibilities
Improving Knowledge and the Use of Statistical Significance Testing
Efforts to Improve Reproducibility
Source Code and Data Version Control
Scientific Workflow-Management Systems
Overcoming Technological and Infrastructure Barriers to Reproducibility
Archival Repositories and Open Data Platforms
Code Hosting and Collaboration Platforms
Obsolescence of Data and Code Storage
Efforts to Improve Replicability
Journal Requirements, Badges, and Awards
Introducing Prepublication Checks for Errors and Anomalous Results
Encouraging the Publication of All Results
Additional Journal Initiatives
Research Funder Efforts to Encourage Replicability
Developing Effective Funder Mandates
Social Science Research Using Big Data
Public Perceptions of Reproducibility and Replicability
Public Understanding of Science
A Biographical Sketches of Committee Members and Staff
B Agendas of Open Committee Meetings
C Recommendations Grouped by Stakeholder