National Academies Press: OpenBook

Reproducibility and Replicability in Science (2019)

Chapter: Front Matter

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

Reproducibility
and Replicability
in Science

Committee on Reproducibility and Replicability in Science

Board on Behavioral, Cognitive, and Sensory Sciences
Committee on National Statistics
Division of Behavioral and Social Sciences and Education

Nuclear and Radiation Studies Board
Division on Earth and Life Studies

Board on Mathematical Sciences and Analytics
Committee on Applied and Theoretical Statistics
Division on Engineering and Physical Sciences

Board on Research Data and Information
Committee on Science, Engineering, Medicine, and Public Policy
Policy and Global Affairs

A Consensus Study Report of

images

THE NATIONAL ACADEMIES PRESS
Washington, DC
www.nap.edu

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001

This activity was supported by contracts between the National Academy of Sciences and Alfred P. Sloan Foundation (G-2018-10102) and the National Science Foundation (1743856). Support for the work of the Board on Behavioral, Cognitive, and Sensory Sciences is provided primarily by a grant from the National Science Foundation (Award No. BCS-1729167). Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project.

International Standard Book Number-13: 978-0-309-48616-3
International Standard Book Number-10: 0-309-48616-5
Digital Object Identifier: https://doi.org/10.17226/25303
Library of Congress Control Number: 2019946492

Additional copies of this publication are available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.

Copyright 2019 by the National Academy of Sciences. All rights reserved.

Printed in the United States of America

Suggested citation: National Academies of Sciences, Engineering, and Medicine. (2019). Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. https://doi.org/10.17226/25303.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

Image

The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president.

The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. C. D. Mote, Jr., is president.

The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.

The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The National Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.

Learn more about the National Academies of Sciences, Engineering, and Medicine at www.nationalacademies.org.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

Image

Consensus Study Reports published by the National Academies of Sciences, Engineering, and Medicine document the evidence-based consensus on the study’s statement of task by an authoring committee of experts. Reports typically include findings, conclusions, and recommendations based on information gathered by the committee and the committee’s deliberations. Each report has been subjected to a rigorous and independent peer-review process and it represents the position of the National Academies on the statement of task.

Proceedings published by the National Academies of Sciences, Engineering, and Medicine chronicle the presentations and discussions at a workshop, symposium, or other event convened by the National Academies. The statements and opinions contained in proceedings are those of the participants and are not endorsed by other participants, the planning committee, or the National Academies.

For information about other products and activities of the National Academies, please visit www.nationalacademies.org/about/whatwedo.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

COMMITTEE ON REPRODUCIBILITY AND REPLICABILITY IN SCIENCE

HARVEY V. FINEBERG1 (Chair), Gordon and Betty Moore Foundation

DAVID B. ALLISON,1 Indiana University, Bloomington

LORENA A. BARBA, The George Washington University

DIANNE CHONG,2 Boeing Research and Technology (retired)

DAVID DONOHO,3,4 Stanford University

JULIANA FREIRE, New York University

GERALD GABRIELSE,3 Northwestern University

CONSTANTINE GATSONIS, Brown University

EDWARD HALL, Harvard University

THOMAS H. JORDAN,3 University of Southern California

DIETRAM A. SCHEUFELE, University of Wisconsin–Madison

VICTORIA STODDEN, University of Illinois at Urbana–Champaign

SIMINE VAZIRE,5 University of California, Davis

TIMOTHY D. WILSON, University of Virginia

WENDY WOOD, University of Southern California and INSEAD-Sorbonne

JENNIFER HEIMBERG, Study Director

THOMAS ARRISON, Program Director

MICHAEL COHEN, Senior Program Officer

MICHELLE SCHWALBE, Director, Board on Mathematical Sciences and Analytics

ADRIENNE STITH BUTLER, Associate Board Director

BARBARA A. WANCHISEN, Director, Board on Behavioral, Cognitive, and Sensory Sciences

TINA WINTERS, Associate Program Officer

REBECCA MORGAN, Senior Librarian

THELMA COX, Program Coordinator (beginning January 2019)

LESLEY WEBB, Program Assistant (September 2018 through January 2019)

GARRET TYSON, Program Assistant (September 2017 through August 2018)

ERIN HAMMERS FORSTAG, Consultant Writer

___________________

1 Member of the National Academy of Medicine.

2 Member of the National Academy of Engineering.

3 Member of the National Academy of Sciences.

4 Resigned from the committee July 24, 2018.

5 Resigned from the committee October 11, 2018.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

BOARD ON BEHAVIORAL, COGNITIVE, AND SENSORY SCIENCES

Division of Behavioral and Social Sciences and Education

SUSAN FISKE1 (Chair), Princeton University

JOHN BAUGH, Washington University in St. Louis

LAURA CARTENSEN,2 Stanford University

JUDY DUBNO, Medical University of South Carolina

JENNIFER EBERHARDT,1 Stanford University

WILSON S. GEISLER,1 The University of Texas at Austin

MICHELE GELFAND, University of Maryland, College Park

NANCY G. KANWISHER,1 Massachusetts Institute of Technology

JANICE KIECOLT-GLASER,2 The Ohio State University College of Medicine

WILLIAM (BILL) MAURER, University of California, Irvine

TERRIE E. MOFFITT,2 Duke University

ELIZABETH A. PHELPS, Harvard University

STEVEN E. PETERSEN, Washington University in St. Louis

DANA SMALL, Yale University

TIMOTHY J. STRAUMAN, Duke University

BARBARBA A. WANCHISEN, Director

___________________

1 Member of the National Academy of Sciences.

2 Member of the National Academy of Medicine.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

COMMITTEE ON NATIONAL STATISTICS

Division of Behavioral and Social Sciences and Education

ROBERT M. GROVES1,2 (Chair), Georgetown University

MARY ELLEN BOCK, Purdue University

ANNE C. CASE,1 Princeton University

MICHAEL E. CHERNEW,1 Harvard Medical School

JANET M. CURRIE,1 Princeton University

DONALD A. DILLMAN, Washington State University

DIANA FARRELL, JPMorgan Chase Institute

DANIEL KIFER, The Pennsylvania State University

THOMAS L. MESENBOURG, U.S. Census Bureau (retired)

SARAH M. NUSSER, Iowa State University

COLM O’MUIRCHEARTAIGH, University of Chicago

JEROME P. REITER, Duke University

ROBERTO RIGOBON, Massachusetts Institute of Technology

JUDITH A. SELTZER, University of California, Los Angeles

C. MATTHEW SNIPP, Stanford University

BRIAN HARRIS-KOJETIN, Director

CONSTANCE F. CITRO, Senior Scholar

___________________

1 Member of the National Academy of Medicine.

2 Member of the National Academy of Sciences.

Page viii Cite
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

NUCLEAR AND RADIATION STUDIES BOARD

Division on Earth and Life Studies

GEORGE APOSTOLAKIS1 (Chair), Massachusetts Institute of Technology

JAMES A. BRINK (Vice Chair), Massachusetts General Hospital

SALLY A. AMUNDSON, Columbia University Medical Center

STEVEN M. BECKER, Old Dominion University

AMY J. BERRINGTON DE GONZÁLEZ, National Cancer Institute

PAUL T. DICKMAN, Argonne National Laboratory

TISSA H. ILLANGASEKARE, Colorado School of Mines

CAROL M. JANTZEN, Savannah River National Laboratory (retired)

BONNIE D. JENKINS, The Brookings Institution

ALLISON M. MACFARLANE, The George Washington University

NANCY JO NICHOLAS, Los Alamos National Laboratory

R. JULIAN PRESTON, Environmental Protection Agency

HENRY D. ROYAL, Washington University School of Medicine in St. Louis

WILLIAM H. TOBEY, Harvard University

SERGEY V. YUDINTSEV, Russian Academy of Sciences

CHARLES FERGUSON, Director

___________________

1 Member of the National Academy of Engineering.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

BOARD ON MATHEMATICAL SCIENCES AND ANALYTICS

Division on Engineering and Physical Sciences

STEPHEN M. ROBINSON1 (Chair), University of Wisconsin–Madison

JOHN R. BIRGE,1 University of Chicago

W. PETER CHERRY,1 Independent Consultant, Ann Arbor, MI

DAVID S. C. CHU, Institute for Defense Analyses

RONALD R. COIFMAN,2 Yale University

JAMES H. CURRY, University of Colorado

MARK L. GREEN, University of California, Los Angeles

SHAWNDRA HILL, Microsoft Research

LYDIA KAVRAKI,3 Rice University

TAMARA KOLDA, Sandia National Laboratories

JOSEPH A. LANGSAM, University of Maryland, College Park

DAVID MAIER, Portland State University

LOIS C. McINNES, Argonne National Laboratory

JILL C. PIPHER, Brown University

ELIZABETH A. THOMPSON,1 University of Washington

CLAIRE J. TOMLIN,1 University of California, Berkeley

LANCE A. WALLER, Emory University

KAREN WILLCOX, The University of Texas at Austin

DAVID D. YAO,1 Columbia University

MICHELLE K. SCHWALBE, Director

___________________

1 Member of the National Academy of Engineering.

2 Member of the National Academy of Sciences.

3 Member of the National Academy of Medicine.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

COMMITTEE ON APPLIED AND THEORETICAL STATISTICS

Division on Engineering and Physical Sciences

ALFRED O. HERO, III (Chair), University of Michigan

KATHERINE BENNETT ENSOR, Rice University

ALICIA L. CARRIQUIRY,1 Iowa State University

RONG CHEN, Rutgers University

MICHAEL J. DANIELS, University of Florida

AMY H. HERRING, Duke University

TIM HESTERBERG, Google, Inc.

NICHOLAS HORTON, Amherst College

DAVID MADIGAN, Columbia University

XIAO-LI MENG, Harvard University

JOSÉ M.F. MOURA,2 Carnegie Mellon University

RAQUEL PRADO, University of California, Santa Cruz

NANCY M. REID,3 University of Toronto

CYNTHIA RUDIN, Duke University

AARTI SINGH, Carnegie Mellon University

ALYSON GABBARD WILSON, North Carolina State University

BENJAMIN WENDER, Director

___________________

1 Member of the National Academy of Medicine.

2 Member of the National Academy of Engineering.

3 Member of the National Academy of Sciences.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

BOARD ON RESEARCH DATA AND INFORMATION

Policy and Global Affairs

ALEXA MCCRAY1 (Chair), Harvard Medical School

AMY BRAND, Massachusetts Institute of Technology

STUART FELDMAN, Schmidt Philanthropies

SALMAN HABIB, Argonne National Laboratory

JAMES HENDLER, Rensselaer Polytechnic Institute

ELLIOT MAXWELL, e-Maxwell and Associates

BAREND MONS, Leiden University Medical Center

SARAH M. NUSSER, Iowa State University

MICHAEL STEBBINS, Science Advisors, LLC

BONNIE CARROLL, Information International Associates (CODATA Secretary General)

JOHN HILDEBRAND,2 University of Arizona (NAS Foreign Secretary)

PAUL UHLIR, Data Policy and Management (CODATA Executive Committee Member)

GEORGE STRAWN, Director

___________________

1 Member of the National Academy of Medicine.

2 Member of the National Academy of Sciences.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

COMMITTEE ON SCIENCE, ENGINEERING, MEDICINE, AND PUBLIC POLICY

Policy and Global Affairs

ALAN I. LESHNER (2020)1 (Chair), American Association for the Advancement of Science

Ex-Officio Members:

VICTOR J. DZAU1 (2020), President, National Academy of Medicine

MARCIA McNUTT2 (2022), President, National Academy of Sciences

C. D. (DAN) MOTE, JR.3 (2019), President, National Academy of Engineering

Members:

CYNTHIA BARNHART,3 Massachusetts Institute of Technology

CLAIRE D. BRINDIS,1 University of California, San Francisco

DAVID E. DANIEL,3 University of Texas at Dallas

KATHARINE G. FRASE,3 IBM Corporation (retired)

JOHN G. HILDEBRAND,2 University of Arizona

DAVID KORN,1 Harvard Medical School

RICHARD A. MESERVE,3 Covington & Burling LLP

J. SANFORD SCHWARTZ,1 University of Pennsylvania

CHRISTOPHER A. SIMS,2 Princeton University

ROBERT F. SPROULL,3 University of Massachusetts at Amherst

JAMES M. TIEN,2 University of Miami

ZENA WERB,1,2 University of California, San Francisco

MICHAEL S. WITHERELL,2 Lawrence Berkeley National Laboratory

SUSAN M. WOLF,1 University of Minnesota

PETER G. WOLYNES,2 Rice University

ANNE-MARIE MAZZA, Senior Director

___________________

1 Member of the National Academy of Medicine.

2 Member of the National Academy of Sciences.

3 Member of the National Academy of Engineering.

Page xiii Cite
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

Acknowledgments

This Consensus Study Report reflects the invaluable contributions of many individuals, including those who served on the committee, the staff of the National Academies of Sciences, Engineering, and Medicine, and many other experts. This report was made possible by funding from the National Science Foundation and sponsorships from the Alfred P. Sloan Foundation. We thank Daniel Goroff of the Alfred P. Sloan Foundation for supporting the study financially to allow for commissioned papers, expanded dissemination activities, and providing insight to the committee.

The committee is also grateful for the efforts of the following authors who prepared background papers for the committee’s use in drafting the report:

  • Rosemary Bush, Perspectives on Reproducibility and Replication of Results in Climate Science
  • Emily Howell, Public Perceptions of Scientific Uncertainty and Media Reporting of Reproducibility and Replication in Science
  • Xihong Lin, Reproducibility and Replicability in Large Scale Genetic Studies
  • Anne Plant and Robert Hanisch, Reproducibility and Replicability in Science, a Metrology Perspective
  • Lars Vilhuber, Reproducibility and Replicability in Economics

This Consensus Study Report was reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise. The purpose of this independent review is to provide candid and critical comments that

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

will assist the institution in making its published report as sound as possible and to make certain that the report meets institutional standards for objectivity, evidence, and responsiveness to the charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process.

We thank the following individuals for their review of this report: R. Stephen Berry, Department of Chemistry and James Franck Institute, The University of Chicago; Kenneth Bollen, Department of Psychology and Neuroscience and Department of Sociology, University of North Carolina at Chapel Hill; Mary Sue Coleman, Office of the President, Association of American Universities; David L. Donoho, Department of Statistics, Stanford University; Stuart Firestein, Department of Biological Sciences, Columbia University; Steven N. Goodman, Division of Epidemiology and Clinical Research, Stanford University School of Medicine; Paul L. Joskow, Alfred P. Sloan Foundation and Department of Economics, Massachusetts Institute of Technology; Louis J. Lanzerotti, Department of Physics, New Jersey Institute of Technology; Don Monroe, science and technology writer, Massachusetts; Brian Nosek, Department of Psychology and Center for Open Science, University of Virginia; Roger D. Peng, Bloomberg School of Public Health, Johns Hopkins University; Gianluca Setti, Department of Electronics and Telecommunications, Polytecnic of Turin; and Eric-Jan Wagenmakers, Department of Psychological Methods, University of Amsterdam.

Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the content of the report nor did they see the final draft of the report before its release. The review of this report was overseen by Robert M. Groves, Department of Mathematics and Statistics, Department of Sociology, and Office of the Provost, Georgetown University, and Julia M. Phillips, Sandia National Laboratories (retired). They were responsible for making certain that an independent examination of this report was carried out in accordance with the standards of the National Academies and that all review comments were carefully considered. Responsibility for the final content rests entirely with the authoring committee and the National Academies.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

Preface

When Sleeper filled theaters in 1973, stick margarine was widely advertised as the healthier alternative to butter. In just a couple of decades, evidence began to accumulate that partially hydrogenated (trans) fats found in hard margarines were worse for heart health than the saturated fat found in butter (not that either is particularly heart healthy). More recently, adults who for years have been ingesting daily doses of baby aspirin, with the aim of reducing the risk of heart attack, are now being advised not to bother. The latest studies failed to confirm earlier findings that had suggested real benefits from daily aspirin. Perhaps we should caution Woody Allen’s Dr. Aragon to be a little less certain of “what we now know to be true.”

Science is based on a conviction that the natural world adheres to certain principles, grounded in an underlying and consistent reality. However, human capacity to discern those truths of nature, including human

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

behavior, is imperfect. We rely on science to reveal what is knowable of nature, and typically, that knowledge has some level of uncertainty attached to it. Repeated findings of comparable results tend to confirm the veracity of an original scientific conclusion, and, by the same token, repeated failures to confirm throw the original conclusions into doubt. When a scientific study becomes the basis of policy or has a direct or indirect impact on human well-being, scientific reliability becomes more than an academic question.

This Consensus Study was prompted by concerns about the reproducibility and replicability of scientific research. The National Science Foundation (NSF) had entered into discussions with the National Academies of Sciences, Engineering, and Medicine about a study on reproducibility and replicability in the social sciences when Congress enacted a provision of law that expanded the scope of the study to all science and engineering. The Alfred P. Sloan Foundation then joined in support of this work, with special interest in the efficiency of scientific research, and aided in the dissemination of the findings, conclusions, and recommendations of the study.

To carry out the task, the National Academies appointed a committee of 15 members representing a wide range of expertise: methodology and statistics, philosophy of science, science communication, behavioral and social sciences, earth and life sciences, physical sciences, computational science, engineering, academic leadership, journal editors, and industry expertise in quality control. Individuals with expertise pertaining to reproducibility and replicability of research results across a variety of fields were included as well. In conducting its study, the committee reviewed the research literature on reproducibility and replicability, held 12 meetings at which it heard from a wide range of stakeholders in the research enterprise and deliberated to reach the findings, conclusions, and recommendations presented in this report.

I have had the privilege of chairing this diverse panel of experts, and I thank all of the members of the committee for their intensive effort and collaborative spirit in crafting this report. We were aided by a remarkably talented study director, Jennifer Heimberg, and an able group of staff, including Thomas Arrison, Adrienne Stith Butler, Michelle Schwalbe, Tina Winters, Michael Cohen, Rebecca Morgan, Thelma Cox, Lesley Webb, and Garret Tyson. We also offer special thanks to Erin Hammers Forstag, who served as consultant writer on this project, and Eugenia Grohman, who edited earlier versions of this manuscript. We are most grateful to NSF and to the Alfred P. Sloan Foundation for their generous support of this undertaking.

Page xvii Cite
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

We hope the ideas and guidance offered here prove useful to Congress, public and private funders of scientific research, scientists and research institutions, journal editors and authors, and the interested public. Science and technology shape our world in both dramatic and mundane ways. We all have a stake in ensuring that scientists adhere to the highest standards of practice, understand and express the uncertainty inherent in their conclusions, and continue to strengthen the interconnected web of scientific knowledge—the principal driver of progress in the modern world.

Harvey V. Fineberg, Chair
Committee on Reproducibility and Replicability in Science

Page xviii Cite
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

This page intentionally left blank.

Page xxii Cite
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

This page intentionally left blank.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R1
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R2
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R3
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R4
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R5
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R6
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R7
Page viii Cite
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R8
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R9
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R10
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R11
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R12
Page xiii Cite
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R13
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R14
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R15
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R16
Page xvii Cite
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R17
Page xviii Cite
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R18
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R19
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R20
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R21
Page xxii Cite
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R22
Next: Executive Summary »
Reproducibility and Replicability in Science Get This Book
×
Buy Paperback | $65.00 Buy Ebook | $54.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

One of the pathways by which the scientific community confirms the validity of a new scientific discovery is by repeating the research that produced it. When a scientific effort fails to independently confirm the computations or results of a previous study, some fear that it may be a symptom of a lack of rigor in science, while others argue that such an observed inconsistency can be an important precursor to new discovery.

Concerns about reproducibility and replicability have been expressed in both scientific and popular media. As these concerns came to light, Congress requested that the National Academies of Sciences, Engineering, and Medicine conduct a study to assess the extent of issues related to reproducibility and replicability and to offer recommendations for improving rigor and transparency in scientific research.

Reproducibility and Replicability in Science defines reproducibility and replicability and examines the factors that may lead to non-reproducibility and non-replicability in research. Unlike the typical expectation of reproducibility between two computations, expectations about replicability are more nuanced, and in some cases a lack of replicability can aid the process of scientific discovery. This report provides recommendations to researchers, academic institutions, journals, and funders on steps they can take to improve reproducibility and replicability in science.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!