National Academies Press: OpenBook

Reproducibility and Replicability in Science (2019)

Chapter: Appendix C: Recommendations Grouped by Stakeholder

« Previous: Appendix B: Agendas for Committee Meetings
Suggested Citation:"Appendix C: Recommendations Grouped by Stakeholder." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 175
Suggested Citation:"Appendix C: Recommendations Grouped by Stakeholder." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 176
Suggested Citation:"Appendix C: Recommendations Grouped by Stakeholder." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 177
Suggested Citation:"Appendix C: Recommendations Grouped by Stakeholder." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 178
Suggested Citation:"Appendix C: Recommendations Grouped by Stakeholder." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 179
Suggested Citation:"Appendix C: Recommendations Grouped by Stakeholder." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 180
Suggested Citation:"Appendix C: Recommendations Grouped by Stakeholder." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 181
Suggested Citation:"Appendix C: Recommendations Grouped by Stakeholder." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 182

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Prepublication copy, uncorrected proofs. APPENDIX C: Recommendations Grouped by Stakeholder The committee’s recommendations in the main text of the report are presented here by stakeholder: scientists and researchers, the National Science Foundation, other funders, journals and conference organizers, educational institutions, professional societies, and journalists. Some recommendations appear more than once because they are addressed to more than one stakeholder. SCIENTISTS AND RESEARCHERS RECOMMENDATION 4-1: To help ensure the reproducibility of computational results, researchers should convey clear, specific, and complete information about any computational methods and data products that support their published results in order to enable other researchers to repeat the analysis, unless such information is restricted by non-public data policies. That information should include the data, study methods, and computational environment:  the input data used in the study either in extension (e.g., a text file or a binary) or in intension (e.g., a script to generate the data), as well as intermediate results and output data for steps that are nondeterministic and cannot be reproduced in principle;  a detailed description of the study methods (ideally in executable form) together with its computational steps and associated parameters; and  information about the computational environment where the study was originally executed, such as operating system, hardware architecture, and library dependencies (Library dependency, in the context of research software as used here, is the relationship of pieces of software that are needed for another software to run. Problems often occur when installed software has dependencies on specific versions of other software.).7 RECOMMENDATION 5-1: Researchers should, as applicable to the specific study, provide an accurate and appropriate characterization of relevant uncertainties when they report or publish their research. Researchers should thoughtfully communicate all recognized uncertainties and estimate or acknowledge other potential sources of uncertainty that bear on their results, including stochastic uncertainties and uncertainties in measurement, computation, knowledge, modeling, and methods of analysis. RECOMMENDATION 6-1: All researchers should include a clear, specific, and complete description of how the reported result was reached. Different areas of study or types of inquiry may require different kinds of information. Reports should include details appropriate for the type of research, including:  a clear description of all methods, instruments, materials, procedures, measurements, and other variables involved in the study;  a clear description of the analysis of data and decisions for exclusion of some data and inclusion of other;  for results that depend on statistical inference, a description of the analytic decisions and when these decisions were made and whether the study is exploratory or confirmatory; 7 The definition of “library dependency” was corrected during copy editing of the prepublication version of the report. 175

Prepublication copy, uncorrected proofs.  a discussion of the expected constraints on generality, such as which methodological features the authors think could be varied without affecting the result and which must remain constant;  reporting of precision or statistical power; and  discussion of the uncertainty of the measurements, results, and inferences. RECOMMENDATION 6-2: Academic institutions and institutions managing scientific work such as industry and the national laboratories should include training in the proper use of statistical analysis and inference. Researchers who use statistical inference analyses should learn to use them properly. RECOMMENDATION 6-6: Many stakeholders have a role to play in improving computational reproducibility, including educational institutions, professional societies, researchers, and funders.  Educational institutions should educate and train students and faculty about computational methods and tools to improve the quality of data and code and to produce reproducible research.  Professional societies should take responsibility for educating the public and their professional members about the importance and limitations of computational research. Societies have an important role in educating the public about the evolving nature of science and the tools and methods that are used.  Researchers should collaborate with expert colleagues when their education and training are not adequate to meet the computational requirements of their research.  In line with its priority for “harnessing the data revolution,” the National Science Foundation (and other funders) should consider funding of activities to promote computational reproducibility. RECOMMENDATION 6-10: When funders, researchers, and other stakeholders are considering whether and where to direct resources for replication studies, they should consider the following criteria:  The scientific results are important for individual decision-making or for policy decisions.  The results have the potential to make a large contribution to basic scientific knowledge.  The original result is particularly surprising, that is, it is unexpected in light of previous evidence and knowledge.  There is controversy about the topic.  There was potential bias in the original investigation, due, for example, to the source of funding.  There was a weakness or flaw in the design, methods, or analysis of the original study.  The cost of a replication is offset by the potential value in reaffirming the original results.  Future expensive and important studies will build on the original scientific results. RECOMMENDATION 7-1: Scientists should take care to avoid overstating the implications of their research and also exercise caution in their review of press releases, especially when the results bear directly on matters of keen public interest and possible action. 176

Prepublication copy, uncorrected proofs. THE NATIONAL SCIENCE FOUNDATION RECOMMENDATION 4-1: To help ensure the reproducibility of computational results, researchers should convey clear, specific, and complete information about any computational methods and data products that support their published results in order to enable other researchers to repeat the analysis, unless such information is restricted by non-public data policies. That information should include the data, study methods, and computational environment:  the input data used in the study either in extension (e.g., a text file or a binary) or in intension (e.g., a script to generate the data), as well as intermediate results and output data for steps that are nondeterministic and cannot be reproduced in principle;  a detailed description of the study methods (ideally in executable form) together with its computational steps and associated parameters; and  information about the computational environment where the study was originally executed, such as operating system, hardware architecture, and library dependencies (Library dependency, in the context of research software as used here, is the relationship of pieces of software that are needed for another software to run. Problems often occur when installed software has dependencies on specific versions of other software.).8 RECOMMENDATION 4-2: The National Science Foundation should consider investing in research that explores the limits of computational reproducibility in instances in which bitwise reproducibility is not reasonable in order to ensure that the meaning of consistent computational results remains in step with the development of new computational hardware, tools, and methods. RECOMMENDATION 6-3: Funding agencies and organizations should consider investing in research and development of open-source, usable tools and infrastructure that support reproducibility for a broad range of studies across different domains in a seamless fashion. Concurrently, investments would be helpful in outreach to inform and train researchers on best practices and how to use these tools. RECOMMENDATION 6-5: In order to facilitate the transparent sharing and availability of digital artifacts, such as data and code, for its studies, the National Science Foundation (NSF) should:  Develop a set of criteria for trusted open repositories to be used by the scientific community for objects of the scholarly record.  Seek to harmonize with other funding agencies the repository criteria and data- management plans for scholarly objects.  Endorse or consider creating code and data repositories for long-term archiving and preservation of digital artifacts that support claims made in the scholarly record based on NSF-funded research. These archives could be based at the institutional level or be part of, and harmonized with, the NSF-funded Public Access Repository. 8 The definition of “library dependency” was corrected during copy editing of the prepublication version of the report. 177

Prepublication copy, uncorrected proofs.  Consider extending NSF’s current data-management plan to include other digital artifacts, such as software.  Work with communities reliant on non-public data or code to develop alternative mechanisms for demonstrating reproducibility Through these repository criteria, NSF would enable discoverability and standards for digital scholarly objects and discourage an undue proliferation of repositories, perhaps through endorsing or providing one go-to website that could access NSF-approved repositories. RECOMMENDATION 6-6: Many stakeholders have a role to play in improving computational reproducibility, including educational institutions, professional societies, researchers, and funders.  Educational institutions should educate and train students and faculty about computational methods and tools to improve the quality of data and code and to produce reproducible research.  Professional societies should take responsibility for educating the public and their professional members about the importance and limitations of computational research. Societies have an important role in educating the public about the evolving nature of science and the tools and methods that are used.  Researchers should collaborate with expert colleagues when their education and training are not adequate to meet the computational requirements of their research.  In line with its priority for “harnessing the data revolution,” the National Science Foundation (and other funders) should consider funding of activities to promote computational reproducibility. RECOMMENDATION 6-8: Many considerations enter into decisions about what types of scientific studies to fund, including striking a balance between exploratory and confirmatory research. If private or public funders choose to invest in initiatives on reproducibility and replication, two areas may benefit from additional funding:  education and training initiatives to ensure that researchers have the knowledge, skills, and tools needed to conduct research in ways that adhere to the highest scientific standards; that describe methods clearly, specifically, and completely; and that express accurately and appropriately the uncertainty involved in the research; and  reviews of published work, such as testing the reproducibility of published research, conducting rigorous replication studies, and publishing sound critical commentaries. RECOMMENDATION 6-9: Funders should require a thoughtful discussion in grant applications of how uncertainties will be evaluated, along with any relevant issues regarding replicability and computational reproducibility. Funders, and they should also introduce review of reproducibility and replicability guidelines and activities into their merit-review criteria, as a low-cost way to enhance both. RECOMMENDATION 6-10: When funders, researchers, and other stakeholders are considering whether and where to direct resources for replication studies, they should consider the following criteria:  The scientific results are important for individual decision-making or for policy decisions. 178

Prepublication copy, uncorrected proofs.  The results have the potential to make a large contribution to basic scientific knowledge.  The original result is particularly surprising, that is, it is unexpected in light of previous evidence and knowledge.  There is controversy about the topic.  There was potential bias in the original investigation, due, for example, to the source of funding.  There was a weakness or flaw in the design, methods, or analysis of the original study.  The cost of a replication is offset by the potential value in reaffirming the original results.  Future expensive and important studies will build on the original scientific results. OTHER FUNDERS RECOMMENDATION 6-3: Funding agencies and organizations should consider investing in research and development of open-source, usable tools and infrastructure that support reproducibility for a broad range of studies across different domains in a seamless fashion. Concurrently, investments would be helpful in outreach to inform and train researchers on best practices and how to use these tools. RECOMMENDATION 6-6: Many stakeholders have a role to play in improving computational reproducibility, including educational institutions, professional societies, researchers, and funders.  Educational institutions should educate and train students and faculty about computational methods and tools to improve the quality of data and code and to produce reproducible research.  Professional societies should take responsibility for educating the public and their professional members about the importance and limitations of computational research. Societies have an important role in educating the public about the evolving nature of science and the tools and methods that are used.  Researchers should collaborate with expert colleagues when their education and training are not adequate to meet the computational requirements of their research.  In line with its priority for “harnessing the data revolution,” the National Science Foundation (and other funders) should consider funding of activities to promote computational reproducibility. RECOMMENDATION 6-8: Many considerations enter into decisions about what types of scientific studies to fund, including striking a balance between exploratory and confirmatory research. If private or public funders choose to invest in initiatives on reproducibility and replication, two areas may benefit from additional funding:  education and training initiatives to ensure that researchers have the knowledge, skills, and tools needed to conduct research in ways that adhere to the highest scientific standards; that describe methods clearly, specifically, and completely; and that express accurately and appropriately the uncertainty involved in the research; and  reviews of published work, such as testing the reproducibility of published research, conducting rigorous replication studies, and publishing sound critical commentaries. 179

Prepublication copy, uncorrected proofs. RECOMMENDATION 6-9: Funders should require a thoughtful discussion in grant applications of how uncertainties will be evaluated, along with any relevant issues regarding replicability and computational reproducibility. Funders, and they should also introduce review of reproducibility and replicability guidelines and activities into their merit-review criteria, as a low-cost way to enhance both. RECOMMENDATION 6-10: When funders, researchers, and other stakeholders are considering whether and where to direct resources for replication studies, they should consider the following criteria:  The scientific results are important for individual decision-making or for policy decisions.  The results have the potential to make a large contribution to basic scientific knowledge.  The original result is particularly surprising, that is, it is unexpected in light of previous evidence and knowledge.  There is controversy about the topic.  There was potential bias in the original investigation, due, for example, to the source of funding.  There was a weakness or flaw in the design, methods, or analysis of the original study.  The cost of a replication is offset by the potential value in reaffirming the original results.  Future expensive and important studies will build on the original scientific results. JOURNALS AND CONFERENCE ORGANIZERS RECOMMENDATION 6-4: Journals should consider ways to ensure computational reproducibility for publications that make claims based on computations, to the extent ethically and legally possible. Although ensuring such reproducibility prior to publication presents technological and practical challenges for researchers and journals, new tools might make this goal more realistic. Journals should make every reasonable effort to use these tools, make clear and enforce their transparency requirements, and increase the reproducibility of their published articles. RECOMMENDATION 6-7: Journals and scientific societies requesting submissions for conferences should disclose their policies relevant to achieving reproducibility and replicability. The strength of the claims made in a journal article or conference submission should reflect the reproducibility and replicability standards to which an article is held, with stronger claims reserved for higher expected levels of reproducibility and replicability. Journals and conference organizers are encouraged to:  set and implement desired standards of reproducibility and replicability and make this one of their priorities, such as deciding which level they wish to achieve for each Transparency and Openness Promotion guideline and working towards that goal;  adopt policies to reduce the likelihood of non-replicability, such as considering incentives or requirements for research materials transparency, design, and analysis plan transparency, enhanced review of statistical methods, study or analysis plan preregistration, and replication studies; and 180

Prepublication copy, uncorrected proofs.  require as a review criterion that all research reports include a thoughtful discussion of the uncertainty in measurements and conclusions. EDUCATIONAL INSTITUTIONS RECOMMENDATION 6-6: Many stakeholders have a role to play in improving computational reproducibility, including educational institutions, professional societies, researchers, and funders.  Educational institutions should educate and train students and faculty about computational methods and tools to improve the quality of data and code and to produce reproducible research.  Professional societies should take responsibility for educating the public and their professional members about the importance and limitations of computational research. Societies have an important role in educating the public about the evolving nature of science and the tools and methods that are used.  Researchers should collaborate with expert colleagues when their education and training are not adequate to meet the computational requirements of their research.  In line with its priority for “harnessing the data revolution,” the National Science Foundation (and other funders) should consider funding of activities to promote computational reproducibility. RECOMMENDATION 6-2: Academic institutions and institutions managing scientific work such as industry and the national laboratories should include training in the proper use of statistical analysis and inference. Researchers who use statistical inference analyses should learn to use them properly. PROFESSIONAL SOCIETIES RECOMMENDATION 6-6: Many stakeholders have a role to play in improving computational reproducibility, including educational institutions, professional societies, researchers, and funders.  Educational institutions should educate and train students and faculty about computational methods and tools to improve the quality of data and code and to produce reproducible research.  Professional societies should take responsibility for educating the public and their professional members about the importance and limitations of computational research. Societies have an important role in educating the public about the evolving nature of science and the tools and methods that are used.  Researchers should collaborate with expert colleagues when their education and training are not adequate to meet the computational requirements of their research.  In line with its priority for “harnessing the data revolution,” the National Science Foundation (and other funders) should consider funding of activities to promote computational reproducibility. RECOMMENDATION 6-7: Journals and scientific societies requesting submissions for conferences should disclose their policies relevant to achieving reproducibility and replicability. 181

Prepublication copy, uncorrected proofs. The strength of the claims made in a journal article or conference submission should reflect the reproducibility and replicability standards to which an article is held, with stronger claims reserved for higher expected levels of reproducibility and replicability. Journals and conference organizers are encouraged to:  set and implement desired standards of reproducibility and replicability and make this one of their priorities, such as deciding which level they wish to achieve for each Transparency and Openness Promotion guideline and working towards that goal;  adopt policies to reduce the likelihood of non-replicability, such as considering incentives or requirements for research materials transparency, design, and analysis plan transparency, enhanced review of statistical methods, study or analysis plan preregistration, and replication studies; and  require as a review criterion that all research reports include a thoughtful discussion of the uncertainty in measurements and conclusions. JOURNALISTS RECOMMENDATION 7-2: Journalists should report on scientific results with as much context and nuance as the medium allows. In covering issues related to replicability and reproducibility, journalists should help their audiences understand the differences between non-reproducibility and non-replicability due to fraudulent conduct of science and instances in which the failure to reproduce or replicate may be due to evolving best practices in methods or inherent uncertainty in science. Particular care in reporting on scientific results is warranted when:  the scientific system under study is complex and with limited control over alternative explanations or confounding influences;  a result is particularly surprising or at odds with existing bodies of research;  the study deals with an emerging area of science that is characterized by significant disagreement or contradictory results within the scientific community; and  research involves potential conflicts of interest, such as work funded by advocacy groups, affected industry, or others with a stake in the outcomes. MEMBERS OF THE PUBLIC AND POLICY MAKERS RECOMMENDATION 7-3: Anyone making personal or policy decisions based on scientific evidence should be wary of making a serious decision based on the results, no matter how promising, of a single study. Similarly, no one should take a new, single contrary study as refutation of scientific conclusions supported by multiple lines of previous evidence. 182

Next: Appendix D: Using Bayes Analysis for Hypothesis Testing »
Reproducibility and Replicability in Science Get This Book
×
Buy Prepub | $69.00 Buy Paperback | $65.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

One of the pathways by which the scientific community confirms the validity of a new scientific discovery is by repeating the research that produced it. When a scientific effort fails to independently confirm the computations or results of a previous study, some fear that it may be a symptom of a lack of rigor in science, while others argue that such an observed inconsistency can be an important precursor to new discovery.

Concerns about reproducibility and replicability have been expressed in both scientific and popular media. As these concerns came to light, Congress requested that the National Academies of Sciences, Engineering, and Medicine conduct a study to assess the extent of issues related to reproducibility and replicability and to offer recommendations for improving rigor and transparency in scientific research.

Reproducibility and Replicability in Science defines reproducibility and replicability and examines the factors that may lead to non-reproducibility and non-replicability in research. Unlike the typical expectation of reproducibility between two computations, expectations about replicability are more nuanced, and in some cases a lack of replicability can aid the process of scientific discovery. This report provides recommendations to researchers, academic institutions, journals, and funders on steps they can take to improve reproducibility and replicability in science.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!