National Academies Press: OpenBook

Reproducibility and Replicability in Science (2019)

Chapter: Front Matter

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R1
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R2
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R3
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R4
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R5
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R6
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R7
Page viii Cite
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R8
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R9
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R10
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R11
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R12
Page xiii Cite
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R13
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R14
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R15
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R16
Page xvii Cite
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R17
Page xviii Cite
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R18
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R19
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R20
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R21
Page xxii Cite
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page R22

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Prepublication copy, uncorrected proofs Reproducibility and Replicability in Science Committee on Reproducibility and Replicability in Science Division of Behavioral and Social Sciences and Education Board on Behavioral, Cognitive, and Sensory Sciences Committee on National Statistics Division on Earth and Life Studies Nuclear and Radiation Studies Board Division on Engineering and Physical Sciences Board on Mathematical Sciences and Analytics Committee on Applied and Theoretical Statistics Policy and Global Affairs Board on Research Data and Information Committee on Science, Engineering, Medicine, and Public Policy A Consensus Study Report of

Prepublication copy, uncorrected proofs THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001 This activity was supported by contracts between the National Academy of Sciences and Alfred P. Sloan Foundation (G-2018-10102), National Science Foundation (1743856). Support for the work of the Board on Behavioral, Cognitive, and Sensory Sciences is provided primarily by a grant from the National Science Foundation (Award No. BCS-1729167). Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project. Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project. International Standard Book Number-13: 978-0-309-XXXXX-X International Standard Book Number-10: 0-309-XXXXX-X Digital Object Identifier: https://doi.org/10.17226/25303 Additional copies of this publication are available for sale from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334- 3313; http://www.nap.edu. Copyright 2019 by the National Academy of Sciences. All rights reserved. Printed in the United States of America Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: https://doi.org/10.17226/25303.

Prepublication copy, uncorrected proofs The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president. The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. C. D. Mote, Jr., is president. The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president. The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The National Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine. Learn more about the National Academies of Sciences, Engineering, and Medicine at www.nationalacademies.org.

Prepublication copy, uncorrected proofs Consensus Study Reports published by the National Academies of Sciences, Engineering, and Medicine document the evidence-based consensus on the study’s statement of task by an authoring committee of experts. Reports typically include findings, conclusions, and recommendations based on information gathered by the committee and the committee’s deliberations. Each report has been subjected to a rigorous and independent peer-review process and it represents the position of the National Academies on the statement of task. Proceedings published by the National Academies of Sciences, Engineering, and Medicine chronicle the presentations and discussions at a workshop, symposium, or other event convened by the National Academies. The statements and opinions contained in proceedings are those of the participants and are not endorsed by other participants, the planning committee, or the National Academies. For information about other products and activities of the National Academies, please visit www.nationalacademies.org/about/whatwedo.

Prepublication copy, uncorrected proofs COMMITTEE ON REPRODUCIBILITY AND REPLICABILITY IN SCIENCE HARVEY V. FINEBERG,1 (Chair), Gordon and Betty Moore Foundation DAVID B. ALLISON,1 School of Public Health-Bloomington, Indiana University LORENA A. BARBA, School of Engineering and Applied Science, George Washington University DIANNE CHONG,2 Boeing Research and Technology (Retired) DAVID DONOHO, 3, 4 Department of Statistics, Stanford University JULIANA FREIRE, Tandon School of Engineering, New York University GERALD GABRIELSE,3 Department of Physics, Northwestern University CONSTANTINE GATSONIS, Center for Statistical Sciences, Brown University EDWARD HALL, Department of Philosophy, Harvard University THOMAS H. JORDAN,3 Department of Earth Sciences, University of Southern California DIETRAM A. SCHEUFELE, Madison and Morgridge Institute for Research, University of Wisconsin-Madison VICTORIA STODDEN, Institute for Data Sciences and Engineering, University of Illinois at Urbana-Champaign SIMINE VAZIRE,5 Department of Psychology, University of California, Davis TIMOTHY D. WILSON, Department of Psychology, University of Virginia WENDY WOOD, Department of Psychology, University of Southern California and INSEAD- Sorbonne University Study Staff JENNIFER HEIMBERG, Study Director THOMAS ARRISON, Program Director MICHAEL COHEN, Senior Program Officer MICHELLE SCHWALBE, Director, Board on Mathematical Sciences and Analytics ADRIENNE STITH BUTLER, Associate Board Director BARBARA A. WANCHISEN, Director, Board on Behavioral, Cognitive, and Sensory Sciences TINA WINTERS, Associate Program Officer REBECCA MORGAN, Senior Librarian THELMA COX, Program Coordinator (beginning January 2019) LESLEY WEBB, Program Assistant (September 2018 through January 2019) GARRET TYSON, Program Assistant (September 2017 through August 2018) ERIN HAMMERS FORSTAG, Consultant Writer 1 Member of the National Academy of Medicine 2 Member of the National Academy of Engineering 3 Member of the National Academy of Sciences 4 Resigned from the committee on July 24, 2018 5 Resigned from the committee on October 11, 2018 v

Prepublication copy, uncorrected proofs Division of Behavioral and Social Sciences and Education BOARD ON BEHAVIORAL, COGNITIVE, AND SENSORY SCIENCES SUSAN FISKE,1 (Chair), Princeton University JOHN BAUGH, Washington University in St. Louis LAURA CARTENSEN,2 Stanford University JUDY DUBNO, Medical University of South Carolina JENNIFER EBERHARDT,1 Stanford University WILSON S. GEISLER, 1 The University of Texas MICHELLE GELFAND, University of Maryland NANCY G. KANWISHER,1 Massachusetts Institute of Technology JANICE KIECOLT-GLASER,2 Ohio State University College of Medicine WILLIAM (BILL) MAUER, University of California, Irvine TERRIE E. MOFFITT,2 Duke University ELIZABETH A PHELPS, Harvard University STEVEN E. PETERSEN, Washington University in St. Louis DANA SMALL, Yale University TIMOTHY J. STRAUMAN, Duke University BARBARBA WANCHISEN, Director 1 Member of the National Academy of Sciences 2 Member of the National Academy of Medicine vi

Prepublication copy, uncorrected proofs Division of Behavioral and Social Sciences and Education COMMITTEE ON NATIONAL STATISTICS ROBERT M. GROVES,1, 2 (Chair), Georgetown University MARY ELLEN BOCK, Purdue University ANNE C. CASE,1 Princeton University MICHAEL E. CHERNEW,1 Harvard Medical School JANET M. CURRIE,1 Princeton University DONALD A. DILLMAN, Washington State University DIANA FARRELL, JPMorgan Chase Institute DANIEL KIFER, The Pennsylvania State University THOMAS L. MESENBOURG, Retired, formerly U.S. Census Bureau SARAH M. NUSSER, Iowa State University COLM O'MUIRCHEARTAIGH, University of Chicago JEROME P. REITER, Duke University ROBERTO RIGOBON, Massachusetts Institute of Technology JUDITH A. SELTZER, University of California, Los Angeles C. MATTHEW SNIPP, Stanford University BRIAN HARRIS-KOJETIN, Director CONSTANCE F. CITRO, Senior Scholar 1 Member of the National Academy of Medicine 2 Member of the National Academy of Sciences vii

Prepublication copy, uncorrected proofs. Division on Earth and Life Studies NUCLEAR AND RADIATION STUDIES BOARD GEORGE APOSTOLAKIS,1 (Chair), Massachusetts Institute of Technology JAMES A. BRINK, (Vice-Chair), Massachusetts General Hospital SALLY A. AMUNDSON, Columbia University Medical Center STEVEN M. BECKER, Old Dominion University AMY J. BERRINGTON DE GONZÁLEZ, National Cancer Institute PAUL T. DICKMAN, Argonne National Laboratory TISSA H. ILLANGASEKARE, Colorado School of Mines CAROL M. JANTZEN, (retired) Savannah River National Laboratory BONNIE D. JENKINS, The Brookings Institution ALLISON M. MACFARLANE, The George Washington University NANCY JO NICHOLAS, Los Alamos National Laboratory R. JULIAN PRESTON, Environnemental Protection Agency HENRY D. ROYAL, Washington University School of Medicine WILLIAM H. TOBEY, Harvard University SERGEY V. YUDINTSEV, Russian Academy of Sciences CHARLES FERGUSON, Director 1 Member of the National Academy of Engineering viii

Prepublication copy, uncorrected proofs. Division on Engineering and Physical Sciences BOARD ON MATHEMATICAL SCIENCES AND ANALYTICS STEPHEN M. ROBINSON,1 (Chair), University of Wisconsin-Madison JOHN R. BIRGE,1 University of Chicago W. PETER CHERRY,1 Independent Consultant, Ann Arbor, MI DAVID S. C. CHU, Institute for Defense Analyses RONALD R. COIFMAN,2 Yale University JAMES H. CURRY, University of Colorado MARK L. GREEN, University of California, Los Angeles SHAWNDRA HILL, Microsoft Research TAMARA KOLDA, Sandia National Laboratories LYDIA KAVRAKI, 3 Rice University JOSEPH A. LANGSAM, University of Maryland, College Park DAVID MAIER, Portland State University LOIS C. MCINNES, Argonne National Laboratory JILL C. PIPHER, Brown University ELIZABETH A. THOMPSON,1 University of Washington CLAIRE J. TOMLIN,1 University of California, Berkeley LANCE A. WALLER, Emory University KAREN WILLCOX, University of Texas at Austin DAVID D. YAO,1 Columbia University MICHELLE K. SCHWALBE, Director 1 Member of the National Academy of Engineering 2 Member of the National Academy of Sciences 3 Member of the National Academy of Medicine ix

Prepublication copy, uncorrected proofs. Division of Engineering and Physical Sciences COMMITTEE ON APPLIED AND THEORETICAL STATISTICS ALFRED O. HERO, III, (Chair) University of Michigan KATHERINE BENNETT ENSOR, Rice University ALICIA L. CARRIQUIRY,1 Iowa State University RONG CHEN, Rutgers University MICHAEL J. DANIELS, University of Florida AMY H. HERRING, Duke University TIM HESTERBERG, Google, Inc. NICHOLAS HORTON, Amherst College DAVID MADIGAN, Columbia University XIAO-LI MENG, Harvard University JOSE M.F. MOURA,2 Carnegie Mellon University RAQUEL PRADO, University of California, Santa Cruz NANCY M. REID,3 University of Toronto CYNTHIA RUDIN, Duke University AARTI SINGH, Carnegie Mellon University ALYSON GABBARD WILSON, North Carolina State University BENJAMIN WENDER, Director 1 Member of the National Academy of Medicine 2 Member of the National Academy of Engineering 3 Member of the National Academy of Sciences x

Prepublication copy, uncorrected proofs. Policy and Global Affairs BOARD ON RESEARCH DATA AND INFORMATION ALEXA MCCRAY,1 (Chair), Harvard Medical School AMY BRAND, Massachusetts Institute of Technology STUART FELDMAN, Schmidt Philanthropies SALMAN HABIB, Argonne National Laboratory JAMES HENDLER, Rensselaer Polytechnic Institute ELLIOT MAXWELL, e-Maxwell & Associates BAREND MONS, University Medical Center SARAH NUSSER, Iowa State University MICHAEL STEBBINS, Science Advisors, LLC BONNIE CARROLL, Information International, Associates (CODATA Secretary General) JOHN HILDEBRAND,2 University of Arizona (NAS Foreign Secretary) PAUL UHLIR, Data Policy and Management (CODATA Executive Committee Member) GEORGE STRAWN, Director 1 Member of the National Academy of Medicine 2 Member of the National Academy of Sciences xi

Prepublication copy, uncorrected proofs. Policy and Global Affairs COMMITTEE ON SCIENCE, ENGINEERING, MEDICINE, AND PUBLIC POLICY ALAN I. LESHNER (2020),1 (Chair) American Association for the Advancement of Science Ex-officio Members: VICTOR J. DZAU,1 (2020), President, National Academy of Medicine MARCIA MCNUTT,2 (2022), President, National Academy of Sciences C. D. (DAN) MOTE, JR.,3 (2019), President, National Academy of Engineering Members: CYNTHIA BARNHART,3 Massachusetts Institute of Technology CLAIRE D. BRINDIS,1 University of California, San Francisco DAVID E. DANIEL,3 University of Texas at Dallas KATHARINE G. FRASE,3 International Business Machines Corporation (retired) JOHN G. HILDEBRAND,2 University of Arizona DAVID KORN,* Harvard Medical School RICHARD A. MESERVE, 3 Covington & Burling LLP J. SANFORD SCHWARTZ, 1 University of Pennsylvania CHRISTOPHER A. SIMS,2 Princeton University ROBERT F. SPROULL,3 University of Massachusetts at Amherst JAMES M. TIEN,2 University of Miami ZENA WERB,1, 2 University of California, San Francisco MICHAEL S. WITHERELL,2 Lawrence Berkeley National Laboratory SUSAN M. WOLF,1 University of Minnesota PETER G. WOLYNES,2 Rice University ANNE-MARIE MAZZA, Senior Director 1 Member of the National Academy of Medicine 2 Member of the National Academy of Sciences 3 Member of the National Academy of Engineering xii

Prepublication copy, uncorrected proofs.

Prepublication copy, uncorrected proofs. ACKNOWLEDGMENTS This Consensus Study report reflects the invaluable contributions of many individuals, including those who served on the committee, the staff of the National Academies of Sciences, Engineering, and Medicine, and many other experts. This report was made possible by funding from the National Science Foundation (NSF) and sponsorships from the Alfred P. Sloan Foundation. We thank Daniel Goroff of the Alfred P. Sloan Foundation, for supporting the study financially to allow for commissioned papers, expanded dissemination activities, and providing insight to the committee. The committee is also grateful for the efforts of the following authors who prepared background papers for the committee’s use in drafting the report: Rosemary Bush, Perspectives on Reproducibility and Replication of Results in Climate Science; Emily Howell, Public Perceptions of Scientific Uncertainty and Media Reporting of Reproducibility and Replication in Science; Xihong Lin, Reproducibility and Replicability in Large Scale Genetic Studies; Anne Plant and Robert Hanish, Reproducibility and Replicability in Science, a Metrology Perspective; and Lars Vilhuber, Reproducibility and Replicability in Economics. This Consensus Study report was reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to make certain that the report meets institutional standards for objectivity, evidence, and responsiveness to the charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We thank the following individuals for their review of this report: R. Stephen Berry, Department of Chemistry and James Franck Institute, The University of Chicago; Kenneth Bollen, Department of Psychology and Neuroscience and Department of Sociology, University of North Carolina; Mary Sue Coleman, Office of President, Association of American Universities; David L. Donoho, Department of Statistics, Stanford University; Stuart Firestein, Department of Biological Sciences, Columbia University; Steven N. Goodman, Division of Epidemiology and Clinical Research, Stanford University School of Medicine; Paul L. Joskow, President emeritus, Alfred P. Sloan Foundation and Department of Economics, emeritus, Massachusetts Institute of Technology; Louis J. Lanzerotti, Department of Physics, New Jersey Institute of Technology; Don Monroe, Science and technology writer, MA; Brian Nosek, Department of Psychology and Center for Open Science, University of Virginia; Roger D. Peng, Bloomberg School of Public Health, Johns Hopkins University; Gianluca Setti, Department of Electronics and Telecommunications, Polytecnic of Turin; and Eric-Jan Wagenmakers, Department of Psychological Methods, University of Amsterdam. Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the content of the report nor did they see the final draft of the report before its release. The review of this report was overseen by Robert M. Groves, Department of Mathematics and Statistics, and Sociology and Office of the Provost, Georgetown University, and Julia M. Phillips, retired, Sandia National Laboratories. They were responsible for xiv

Prepublication copy, uncorrected proofs. making certain that an independent examination of this report was carried out in accordance with the standards of the National Academies and that all review comments were carefully considered. Responsibility for the final content rests entirely with the authoring committee and the National Academies. xv

Prepublication copy, uncorrected proofs. PREFACE Dr. Melik: This morning for breakfast he requested something called "wheat germ, organic honey, and tiger's milk." Dr. Aragon: [chuckling] Oh, yes. Those are the charmed substances that some years ago were thought to contain life-preserving properties. Dr. Melik: You mean there was no deep fat? No steak or cream pies or . . . hot fudge? Dr. Aragon: Those were thought to be unhealthy . . . precisely the opposite of what we now know to be true. Dr. Melik: Incredible. Dialogue from Sleeper, 1973 film by Woody Allen set 200 years in the future. When Sleeper filled theaters in 1973, stick margarine was widely advertised as the healthier alternative to butter. In just a couple of decades, evidence began to accumulate that partially hydrogenated (trans) fats found in hard margarines were worse for heart health than the saturated fat found in butter (not that either is particularly heart healthy). More recently, adults who for years have been ingesting daily doses of baby aspirin, with the aim of reducing the risk of heart attack, are now being advised not to bother. The latest studies failed to confirm earlier findings that had suggested real benefits from daily aspirin. Perhaps we should caution Woody Allen’s Dr. Aragon to be a little less certain of “what we now know to be true.” Science is based on a conviction that the natural world adheres to certain principles, grounded in an underlying and consistent reality. However, human capacity to discern those truths of nature, including human behavior, is imperfect. We rely on science to reveal what is knowable of nature, and typically, that knowledge has some level of uncertainty attached to it. Repeated findings of comparable results tend to confirm the veracity of an original scientific conclusion, and, by the same token, repeated failures to confirm throw the original conclusions into doubt. When a scientific study becomes the basis of policy or has a direct or indirect impact on human well-being, scientific reliability becomes more than an academic question. This consensus study was prompted by concerns about the reproducibility and replicability of scientific research. The National Science Foundation (NSF) had entered into discussions with the National Academies of Sciences, Engineering, and Medicine about a study on reproducibility and replicability in the social sciences when Congress enacted a provision of law that expanded the scope of the study to all science and engineering. The Alfred P. Sloan Foundation then joined in support of this work, with special interest in the efficiency of scientific research and to aid in the dissemination of the findings, conclusions, and recommendations of the study. To carry out the task, the National Academies appointed a committee of 13 members representing a wide range of expertise: methodology and statistics, philosophy of science, science communication, behavioral and social sciences, earth and life sciences, physical sciences, computational science, engineering, academic leadership, journal editors, and industry expertise in quality control. Individuals with expertise pertaining to reproducibility and replicability of research results across a variety of fields were included as well. In conducting its study, the committee reviewed the research literature on reproducibility and replicability, held twelve meetings at which it heard from a wide range of stakeholders in the research enterprise, and xvi

Prepublication copy, uncorrected proofs. deliberated to reach the findings, conclusions, and recommendations presented in this report. I have had the privilege of chairing this diverse panel of experts, and I thank all the members of the committee for their intensive effort and collaborative spirit in crafting this report. We were aided by a remarkably talented study director, Jennifer Heimberg, and an able group of staff, including Thomas Arrison, Adrienne Stith Butler, Michelle Schwalbe, Tina Winters, Michael Cohen, Rebecca Morgan, Thelma Cox, Lesley Webb, and Garret Tyson. We also offer special thanks to Erin Hammers Forstag, who served as consultant writer on this project, and Eugenia Grohman, who edited earlier versions of this manuscript. We are most grateful to the NSF and to the Sloan Foundation for their generous support of this undertaking. We hope the ideas and guidance offered here prove useful to Congress, to public and private funders of scientific research, to scientists and research institutions, to journal editors and writers, and to the interested public. Science and technology shape our world in dramatic and in mundane ways. We all have a stake in ensuring that scientists adhere to the highest standards of practice, understand and express the uncertainty inherent in their conclusions, and continue to strengthen the interconnected web of scientific knowledge—the principal driver of progress in the modern world. Harvey V. Fineberg, Chair Committee on Reproducibility and Replicability in Science xvii

Prepublication copy, uncorrected proofs. CONTENTS Executive Summary 1 Summary 3 1 Introduction 17 2 Scientific Methods and Knowledge 21 What Is Science? 21 Core Values and Assumptions of Scientific Inquiry 23 Nature Is Not Capricious 23 Knowledge Grows through Exploration of the Limits of Existing Rules and Mutually Reinforcing Evidence 24 Science Is a Communal Enterprise 24 Science Aims for Refined Degrees of Confidence, Rather Than Complete Certainty 25 Scientific Knowledge Is Durable and Mutable 25 Statistical Inference and Hypothesis Testing 26 3 Understanding Reproducibility and Replicability 31 The Evolving Practices of Science 31 Defining Reproducibility and Replicability 33 Precision of Measurement 36 Variations in Methods Employed in a Study 40 Rigor and Transparency 42 4 Reproducibility 45 Widespread Use of Computational Methods 45 Non-Public Data and Code 46 Resources and Costs of Reproducibility 46 Assessing Reproducibility 48 The Extent of Non-Reproducibility 51 Sources of Non-Reproducibility 55 Inadequate Record-Keeping 56 Non-Transparent Reporting 57 Obsolescence of Digital Artifacts 57 Flawed Attempts to Reproduce Others’ Research 58 Barriers in the Culture of Research 58 5 Replicability 59 Assessing Replicability 59 xviii

Prepublication copy, uncorrected proofs. The Extent of Non-Replicability 62 Assessments of Replicability 62 Perspectives of Researchers Who Have Studied Replicability 68 Surveys 69 Retraction Trends 69 Sources of Non-Replicability 71 Non-Replicability that Is Potentially Helpful to Science 71 Unhelpful Sources of Non-Replicability 75 Publication Bias 76 Misaligned Incentives 79 Inappropriate Statistical Inference 79 Poor Study Design 81 Errors 81 Incomplete Reporting of a Study 82 Fraud and Misconduct 83 6 Improving Reproducibility and Replicability 87 Strengthening Research Practices: Broad Efforts and Responsibilities 87 Education and Training 89 Improving Knowledge and Use of Statistical Significance Testing 90 Efforts to Improve Reproducibility 91 Record-Keeping 91 Source Code and Data Version Control 93 Scientific Workflow-Management Systems 94 Tools for Reproduction of Results 95 Publication Reproducibility Audits 97 Overcoming Technological and Infrastructure Barriers to Reproducibility 97 Archival Repositories and Open Data Platforms 98 Code Hosting and Collaboration Platforms 100 Digital Object Identifiers 100 Obsolescence of Data and Code Storage 100 Implementation Challenges 102 Efforts to Improve Replicability 104 Openness Guidelines 104 Journal Requirements, Badges, and Awards 105 Introducing Prepublication Checks for Errors and Anomalous Results 107 Preregistration of Studies 108 Encouraging the Publication of All Results 109 Additional Journal Initiatives 111 Research Funder Efforts to Encourage Replicability 112 Developing Effective Funder Mandates 115 xix

Prepublication copy, uncorrected proofs. 7 Confidence in Science 117 Research Syntheses 117 Geoscience 119 Genetics 121 Psychology 122 Social Science Research Using Big Data 124 Public Perceptions of Reproducibility and Replicability 126 Public Understanding of Science 126 Public Trust in Science 127 Media Coverage of Science 128 References 131 Appendixes A Biographical Sketches of Committee Members and Staff 159 B Agendas of Open Committee Meetings 167 C Recommendations Grouped by Stakeholders 175 D Using Bayes Analysis for Hypothesis Testing 183 E Conducting Replicable Surveys of Scientific Communities 193 xx

Prepublication copy, uncorrected proofs.

Next: Executive Summary »
Reproducibility and Replicability in Science Get This Book
×
Buy Prepub | $69.00 Buy Paperback | $60.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

One of the pathways by which the scientific community confirms the validity of a new scientific discovery is by repeating the research that produced it. When a scientific effort fails to independently confirm the computations or results of a previous study, some fear that it may be a symptom of a lack of rigor in science, while others argue that such an observed inconsistency can be an important precursor to new discovery.

Concerns about reproducibility and replicability have been expressed in both scientific and popular media. As these concerns came to light, Congress requested that the National Academies of Sciences, Engineering, and Medicine conduct a study to assess the extent of issues related to reproducibility and replicability and to offer recommendations for improving rigor and transparency in scientific research.

Reproducibility and Replicability in Science defines reproducibility and replicability and examines the factors that may lead to non-reproducibility and non-replicability in research. Unlike the typical expectation of reproducibility between two computations, expectations about replicability are more nuanced, and in some cases a lack of replicability can aid the process of scientific discovery. This report provides recommendations to researchers, academic institutions, journals, and funders on steps they can take to improve reproducibility and replicability in science.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!