Reproducibility
and Replicability
in Science

Committee on Reproducibility and Replicability in Science

Board on Behavioral, Cognitive, and Sensory Sciences
Committee on National Statistics
Division of Behavioral and Social Sciences and Education

Nuclear and Radiation Studies Board
Division on Earth and Life Studies

Board on Mathematical Sciences and Analytics
Committee on Applied and Theoretical Statistics
Division on Engineering and Physical Sciences

Board on Research Data and Information
Committee on Science, Engineering, Medicine, and Public Policy
Policy and Global Affairs

A Consensus Study Report of

THE NATIONAL ACADEMIES PRESS
Washington, DC
www.nap.edu

THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001

This activity was supported by contracts between the National Academy of Sciences and Alfred P. Sloan Foundation (G-2018-10102) and the National Science Foundation (1743856). Support for the work of the Board on Behavioral, Cognitive, and Sensory Sciences is provided primarily by a grant from the National Science Foundation (Award No. BCS-1729167). Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project.

International Standard Book Number-13: 978-0-309-48616-3
International Standard Book Number-10: 0-309-48616-5
Digital Object Identifier: https://doi.org/10.17226/25303
Library of Congress Control Number: 2019946492

Additional copies of this publication are available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.

Copyright 2019 by the National Academy of Sciences. All rights reserved.

Printed in the United States of America

Suggested citation: National Academies of Sciences, Engineering, and Medicine. (2019). Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. https://doi.org/10.17226/25303.

COMMITTEE ON REPRODUCIBILITY AND REPLICABILITY IN SCIENCE

HARVEY V. FINEBERG¹ (Chair), Gordon and Betty Moore Foundation

DAVID B. ALLISON,¹ Indiana University, Bloomington

LORENA A. BARBA, The George Washington University

DIANNE CHONG,² Boeing Research and Technology (retired)

DAVID DONOHO,^3,4 Stanford University

JULIANA FREIRE, New York University

GERALD GABRIELSE,³ Northwestern University

CONSTANTINE GATSONIS, Brown University

EDWARD HALL, Harvard University

THOMAS H. JORDAN,³ University of Southern California

DIETRAM A. SCHEUFELE, University of Wisconsin–Madison

VICTORIA STODDEN, University of Illinois at Urbana–Champaign

SIMINE VAZIRE,⁵ University of California, Davis

TIMOTHY D. WILSON, University of Virginia

WENDY WOOD, University of Southern California and INSEAD-Sorbonne

JENNIFER HEIMBERG, Study Director

THOMAS ARRISON, Program Director

MICHAEL COHEN, Senior Program Officer

MICHELLE SCHWALBE, Director, Board on Mathematical Sciences and Analytics

ADRIENNE STITH BUTLER, Associate Board Director

BARBARA A. WANCHISEN, Director, Board on Behavioral, Cognitive, and Sensory Sciences

TINA WINTERS, Associate Program Officer

REBECCA MORGAN, Senior Librarian

THELMA COX, Program Coordinator (beginning January 2019)

LESLEY WEBB, Program Assistant (September 2018 through January 2019)

GARRET TYSON, Program Assistant (September 2017 through August 2018)

ERIN HAMMERS FORSTAG, Consultant Writer

___________________

¹ Member of the National Academy of Medicine.

² Member of the National Academy of Engineering.

³ Member of the National Academy of Sciences.

⁴ Resigned from the committee July 24, 2018.

⁵ Resigned from the committee October 11, 2018.

BOARD ON BEHAVIORAL, COGNITIVE, AND SENSORY SCIENCES

Division of Behavioral and Social Sciences and Education

SUSAN FISKE¹ (Chair), Princeton University

JOHN BAUGH, Washington University in St. Louis

LAURA CARTENSEN,² Stanford University

JUDY DUBNO, Medical University of South Carolina

JENNIFER EBERHARDT,¹ Stanford University

WILSON S. GEISLER,¹ The University of Texas at Austin

MICHELE GELFAND, University of Maryland, College Park

NANCY G. KANWISHER,¹ Massachusetts Institute of Technology

JANICE KIECOLT-GLASER,² The Ohio State University College of Medicine

WILLIAM (BILL) MAURER, University of California, Irvine

TERRIE E. MOFFITT,² Duke University

ELIZABETH A. PHELPS, Harvard University

STEVEN E. PETERSEN, Washington University in St. Louis

DANA SMALL, Yale University

TIMOTHY J. STRAUMAN, Duke University

BARBARBA A. WANCHISEN, Director

___________________

¹ Member of the National Academy of Sciences.

² Member of the National Academy of Medicine.

COMMITTEE ON NATIONAL STATISTICS

Division of Behavioral and Social Sciences and Education

ROBERT M. GROVES^1,2 (Chair), Georgetown University

MARY ELLEN BOCK, Purdue University

ANNE C. CASE,¹ Princeton University

MICHAEL E. CHERNEW,¹ Harvard Medical School

JANET M. CURRIE,¹ Princeton University

DONALD A. DILLMAN, Washington State University

DIANA FARRELL, JPMorgan Chase Institute

DANIEL KIFER, The Pennsylvania State University

THOMAS L. MESENBOURG, U.S. Census Bureau (retired)

SARAH M. NUSSER, Iowa State University

COLM O’MUIRCHEARTAIGH, University of Chicago

JEROME P. REITER, Duke University

ROBERTO RIGOBON, Massachusetts Institute of Technology

JUDITH A. SELTZER, University of California, Los Angeles

C. MATTHEW SNIPP, Stanford University

BRIAN HARRIS-KOJETIN, Director

CONSTANCE F. CITRO, Senior Scholar

___________________

¹ Member of the National Academy of Medicine.

² Member of the National Academy of Sciences.

NUCLEAR AND RADIATION STUDIES BOARD

Division on Earth and Life Studies

GEORGE APOSTOLAKIS¹ (Chair), Massachusetts Institute of Technology

JAMES A. BRINK (Vice Chair), Massachusetts General Hospital

SALLY A. AMUNDSON, Columbia University Medical Center

STEVEN M. BECKER, Old Dominion University

AMY J. BERRINGTON DE GONZÁLEZ, National Cancer Institute

PAUL T. DICKMAN, Argonne National Laboratory

TISSA H. ILLANGASEKARE, Colorado School of Mines

CAROL M. JANTZEN, Savannah River National Laboratory (retired)

BONNIE D. JENKINS, The Brookings Institution

ALLISON M. MACFARLANE, The George Washington University

NANCY JO NICHOLAS, Los Alamos National Laboratory

R. JULIAN PRESTON, Environmental Protection Agency

HENRY D. ROYAL, Washington University School of Medicine in St. Louis

WILLIAM H. TOBEY, Harvard University

SERGEY V. YUDINTSEV, Russian Academy of Sciences

CHARLES FERGUSON, Director

___________________

¹ Member of the National Academy of Engineering.

BOARD ON MATHEMATICAL SCIENCES AND ANALYTICS

Division on Engineering and Physical Sciences

STEPHEN M. ROBINSON¹ (Chair), University of Wisconsin–Madison

JOHN R. BIRGE,¹ University of Chicago

W. PETER CHERRY,¹ Independent Consultant, Ann Arbor, MI

DAVID S. C. CHU, Institute for Defense Analyses

RONALD R. COIFMAN,² Yale University

JAMES H. CURRY, University of Colorado

MARK L. GREEN, University of California, Los Angeles

SHAWNDRA HILL, Microsoft Research

LYDIA KAVRAKI,³ Rice University

TAMARA KOLDA, Sandia National Laboratories

JOSEPH A. LANGSAM, University of Maryland, College Park

DAVID MAIER, Portland State University

LOIS C. McINNES, Argonne National Laboratory

JILL C. PIPHER, Brown University

ELIZABETH A. THOMPSON,¹ University of Washington

CLAIRE J. TOMLIN,¹ University of California, Berkeley

LANCE A. WALLER, Emory University

KAREN WILLCOX, The University of Texas at Austin

DAVID D. YAO,¹ Columbia University

MICHELLE K. SCHWALBE, Director

___________________

¹ Member of the National Academy of Engineering.

² Member of the National Academy of Sciences.

³ Member of the National Academy of Medicine.

COMMITTEE ON APPLIED AND THEORETICAL STATISTICS

Division on Engineering and Physical Sciences

ALFRED O. HERO, III (Chair), University of Michigan

KATHERINE BENNETT ENSOR, Rice University

ALICIA L. CARRIQUIRY,¹ Iowa State University

RONG CHEN, Rutgers University

MICHAEL J. DANIELS, University of Florida

AMY H. HERRING, Duke University

TIM HESTERBERG, Google, Inc.

NICHOLAS HORTON, Amherst College

DAVID MADIGAN, Columbia University

XIAO-LI MENG, Harvard University

JOSÉ M.F. MOURA,² Carnegie Mellon University

RAQUEL PRADO, University of California, Santa Cruz

NANCY M. REID,³ University of Toronto

CYNTHIA RUDIN, Duke University

AARTI SINGH, Carnegie Mellon University

ALYSON GABBARD WILSON, North Carolina State University

BENJAMIN WENDER, Director

___________________

¹ Member of the National Academy of Medicine.

² Member of the National Academy of Engineering.

³ Member of the National Academy of Sciences.

BOARD ON RESEARCH DATA AND INFORMATION

Policy and Global Affairs

ALEXA MCCRAY¹ (Chair), Harvard Medical School

AMY BRAND, Massachusetts Institute of Technology

STUART FELDMAN, Schmidt Philanthropies

SALMAN HABIB, Argonne National Laboratory

JAMES HENDLER, Rensselaer Polytechnic Institute

ELLIOT MAXWELL, e-Maxwell and Associates

BAREND MONS, Leiden University Medical Center

SARAH M. NUSSER, Iowa State University

MICHAEL STEBBINS, Science Advisors, LLC

BONNIE CARROLL, Information International Associates (CODATA Secretary General)

JOHN HILDEBRAND,² University of Arizona (NAS Foreign Secretary)

PAUL UHLIR, Data Policy and Management (CODATA Executive Committee Member)

GEORGE STRAWN, Director

___________________

¹ Member of the National Academy of Medicine.

² Member of the National Academy of Sciences.

COMMITTEE ON SCIENCE, ENGINEERING, MEDICINE, AND PUBLIC POLICY

Policy and Global Affairs

ALAN I. LESHNER (2020)¹ (Chair), American Association for the Advancement of Science

Ex-Officio Members:

VICTOR J. DZAU¹ (2020), President, National Academy of Medicine

MARCIA McNUTT² (2022), President, National Academy of Sciences

C. D. (DAN) MOTE, JR.³ (2019), President, National Academy of Engineering

Members:

CYNTHIA BARNHART,³ Massachusetts Institute of Technology

CLAIRE D. BRINDIS,¹ University of California, San Francisco

DAVID E. DANIEL,³ University of Texas at Dallas

KATHARINE G. FRASE,³ IBM Corporation (retired)

JOHN G. HILDEBRAND,² University of Arizona

DAVID KORN,¹ Harvard Medical School

RICHARD A. MESERVE,³ Covington & Burling LLP

J. SANFORD SCHWARTZ,¹ University of Pennsylvania

CHRISTOPHER A. SIMS,² Princeton University

ROBERT F. SPROULL,³ University of Massachusetts at Amherst

JAMES M. TIEN,² University of Miami

ZENA WERB,^1,2 University of California, San Francisco

MICHAEL S. WITHERELL,² Lawrence Berkeley National Laboratory

SUSAN M. WOLF,¹ University of Minnesota

PETER G. WOLYNES,² Rice University

ANNE-MARIE MAZZA, Senior Director

___________________

¹ Member of the National Academy of Medicine.

² Member of the National Academy of Sciences.

³ Member of the National Academy of Engineering.

Acknowledgments

This Consensus Study Report reflects the invaluable contributions of many individuals, including those who served on the committee, the staff of the National Academies of Sciences, Engineering, and Medicine, and many other experts. This report was made possible by funding from the National Science Foundation and sponsorships from the Alfred P. Sloan Foundation. We thank Daniel Goroff of the Alfred P. Sloan Foundation for supporting the study financially to allow for commissioned papers, expanded dissemination activities, and providing insight to the committee.

The committee is also grateful for the efforts of the following authors who prepared background papers for the committee’s use in drafting the report:

• Rosemary Bush, Perspectives on Reproducibility and Replication of Results in Climate Science
• Emily Howell, Public Perceptions of Scientific Uncertainty and Media Reporting of Reproducibility and Replication in Science
• Xihong Lin, Reproducibility and Replicability in Large Scale Genetic Studies
• Anne Plant and Robert Hanisch, Reproducibility and Replicability in Science, a Metrology Perspective
• Lars Vilhuber, Reproducibility and Replicability in Economics

This Consensus Study Report was reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise. The purpose of this independent review is to provide candid and critical comments that

will assist the institution in making its published report as sound as possible and to make certain that the report meets institutional standards for objectivity, evidence, and responsiveness to the charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process.

We thank the following individuals for their review of this report: R. Stephen Berry, Department of Chemistry and James Franck Institute, The University of Chicago; Kenneth Bollen, Department of Psychology and Neuroscience and Department of Sociology, University of North Carolina at Chapel Hill; Mary Sue Coleman, Office of the President, Association of American Universities; David L. Donoho, Department of Statistics, Stanford University; Stuart Firestein, Department of Biological Sciences, Columbia University; Steven N. Goodman, Division of Epidemiology and Clinical Research, Stanford University School of Medicine; Paul L. Joskow, Alfred P. Sloan Foundation and Department of Economics, Massachusetts Institute of Technology; Louis J. Lanzerotti, Department of Physics, New Jersey Institute of Technology; Don Monroe, science and technology writer, Massachusetts; Brian Nosek, Department of Psychology and Center for Open Science, University of Virginia; Roger D. Peng, Bloomberg School of Public Health, Johns Hopkins University; Gianluca Setti, Department of Electronics and Telecommunications, Polytecnic of Turin; and Eric-Jan Wagenmakers, Department of Psychological Methods, University of Amsterdam.

Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the content of the report nor did they see the final draft of the report before its release. The review of this report was overseen by Robert M. Groves, Department of Mathematics and Statistics, Department of Sociology, and Office of the Provost, Georgetown University, and Julia M. Phillips, Sandia National Laboratories (retired). They were responsible for making certain that an independent examination of this report was carried out in accordance with the standards of the National Academies and that all review comments were carefully considered. Responsibility for the final content rests entirely with the authoring committee and the National Academies.

Preface

When Sleeper filled theaters in 1973, stick margarine was widely advertised as the healthier alternative to butter. In just a couple of decades, evidence began to accumulate that partially hydrogenated (trans) fats found in hard margarines were worse for heart health than the saturated fat found in butter (not that either is particularly heart healthy). More recently, adults who for years have been ingesting daily doses of baby aspirin, with the aim of reducing the risk of heart attack, are now being advised not to bother. The latest studies failed to confirm earlier findings that had suggested real benefits from daily aspirin. Perhaps we should caution Woody Allen’s Dr. Aragon to be a little less certain of “what we now know to be true.”

Science is based on a conviction that the natural world adheres to certain principles, grounded in an underlying and consistent reality. However, human capacity to discern those truths of nature, including human

behavior, is imperfect. We rely on science to reveal what is knowable of nature, and typically, that knowledge has some level of uncertainty attached to it. Repeated findings of comparable results tend to confirm the veracity of an original scientific conclusion, and, by the same token, repeated failures to confirm throw the original conclusions into doubt. When a scientific study becomes the basis of policy or has a direct or indirect impact on human well-being, scientific reliability becomes more than an academic question.

This Consensus Study was prompted by concerns about the reproducibility and replicability of scientific research. The National Science Foundation (NSF) had entered into discussions with the National Academies of Sciences, Engineering, and Medicine about a study on reproducibility and replicability in the social sciences when Congress enacted a provision of law that expanded the scope of the study to all science and engineering. The Alfred P. Sloan Foundation then joined in support of this work, with special interest in the efficiency of scientific research, and aided in the dissemination of the findings, conclusions, and recommendations of the study.

To carry out the task, the National Academies appointed a committee of 15 members representing a wide range of expertise: methodology and statistics, philosophy of science, science communication, behavioral and social sciences, earth and life sciences, physical sciences, computational science, engineering, academic leadership, journal editors, and industry expertise in quality control. Individuals with expertise pertaining to reproducibility and replicability of research results across a variety of fields were included as well. In conducting its study, the committee reviewed the research literature on reproducibility and replicability, held 12 meetings at which it heard from a wide range of stakeholders in the research enterprise and deliberated to reach the findings, conclusions, and recommendations presented in this report.

I have had the privilege of chairing this diverse panel of experts, and I thank all of the members of the committee for their intensive effort and collaborative spirit in crafting this report. We were aided by a remarkably talented study director, Jennifer Heimberg, and an able group of staff, including Thomas Arrison, Adrienne Stith Butler, Michelle Schwalbe, Tina Winters, Michael Cohen, Rebecca Morgan, Thelma Cox, Lesley Webb, and Garret Tyson. We also offer special thanks to Erin Hammers Forstag, who served as consultant writer on this project, and Eugenia Grohman, who edited earlier versions of this manuscript. We are most grateful to NSF and to the Alfred P. Sloan Foundation for their generous support of this undertaking.

We hope the ideas and guidance offered here prove useful to Congress, public and private funders of scientific research, scientists and research institutions, journal editors and authors, and the interested public. Science and technology shape our world in both dramatic and mundane ways. We all have a stake in ensuring that scientists adhere to the highest standards of practice, understand and express the uncertainty inherent in their conclusions, and continue to strengthen the interconnected web of scientific knowledge—the principal driver of progress in the modern world.

Harvey V. Fineberg, Chair
Committee on Reproducibility and Replicability in Science

This page intentionally left blank.

Contents

EXECUTIVE SUMMARY

SUMMARY

1 INTRODUCTION

2 SCIENTIFIC METHODS AND KNOWLEDGE

What Is Science?

Core Principles and Assumptions of Scientific Inquiry

Nature Is Not Capricious

Knowledge Grows Through Exploration of the Limits of Existing Rules and Mutually Reinforcing Evidence

Science Is a Communal Enterprise

Science Aims for Refined Degrees of Confidence, Rather Than Complete Certainty

Scientific Knowledge Is Durable and Mutable

Statistical Inference and Hypothesis Testing

3 UNDERSTANDING REPRODUCIBILITY AND REPLICABILITY

The Evolving Practices of Science

Defining Reproducibility and Replicability

Precision of Measurement

Variations in Methods Employed in a Study

Rigor and Transparency

4 REPRODUCIBILITY

Widespread Use of Computational Methods

Nonpublic Data and Code

Resources and Costs of Reproducibility

Assessing Reproducibility

The Extent of Non-Reproducibility

Sources of Non-Reproducibility

Inadequate Recordkeeping

Nontransparent Reporting

Obsolescence of Digital Artifacts

Flawed Attempts to Reproduce Others’ Research

Barriers in the Culture of Research

5 REPLICABILITY

Assessing Replicability

The Extent of Non-Replicability

Assessments of Replicability

Perspectives of Researchers Who Have Studied Replicability

Surveys

Retraction Trends

Sources of Non-Replicability

Non-Replicability That Is Potentially Helpful to Science

Unhelpful Sources of Non-Replicability

Publication Bias

Misaligned Incentives

Inappropriate Statistical Inference

Poor Study Design

Errors

Incomplete Reporting of a Study

Fraud and Misconduct

6 IMPROVING REPRODUCIBILITY AND REPLICABILITY

Strengthening Research Practices: Broad Efforts and Responsibilities

Education and Training

Improving Knowledge and the Use of Statistical Significance Testing

Efforts to Improve Reproducibility

Recordkeeping

Source Code and Data Version Control

Scientific Workflow-Management Systems

Tools for Reproduction of Results

Publication Reproducibility Audits

Overcoming Technological and Infrastructure Barriers to Reproducibility

Archival Repositories and Open Data Platforms

Code Hosting and Collaboration Platforms

Digital Object Identifiers

Obsolescence of Data and Code Storage

Implementation Challenges

Efforts to Improve Replicability

Openness Guidelines

Journal Requirements, Badges, and Awards

Introducing Prepublication Checks for Errors and Anomalous Results

Preregistration of Studies

Encouraging the Publication of All Results

Additional Journal Initiatives

Research Funder Efforts to Encourage Replicability

Developing Effective Funder Mandates

7 CONFIDENCE IN SCIENCE

Research Synthesis

Geoscience

Genetics

Psychology

Social Science Research Using Big Data

Public Perceptions of Reproducibility and Replicability

Public Understanding of Science

Public Trust in Science

Media Coverage of Science

REFERENCES

APPENDIXES

A Biographical Sketches of Committee Members and Staff

B Agendas of Open Committee Meetings

C Recommendations Grouped by Stakeholder

D Using Bayes Analysis for Hypothesis Testing

E Conducting Replicable Surveys of Scientific Communities

This page intentionally left blank.