National Academies Press: OpenBook

Reproducibility and Replicability in Science (2019)

Chapter: Appendix B: Agendas for Committee Meetings

« Previous: Appendix A: Biographical Sketches of Committee Members and Staff
Suggested Citation:"Appendix B: Agendas for Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 167
Suggested Citation:"Appendix B: Agendas for Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 168
Suggested Citation:"Appendix B: Agendas for Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 169
Suggested Citation:"Appendix B: Agendas for Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 170
Suggested Citation:"Appendix B: Agendas for Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 171
Suggested Citation:"Appendix B: Agendas for Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 172
Suggested Citation:"Appendix B: Agendas for Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 173
Suggested Citation:"Appendix B: Agendas for Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 174

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Prepublication copy, uncorrected proofs. APPENDIX B: Agendas for Committee Meetings First Meeting December 12–13, 2017 TUESDAY, DECEMBER 12, 2017 Welcome and Introductions Mary Ellen O’Connell, Executive Director, Division of Behavioral and Social Science and Education Harvey Fineberg, Committee Chair; President, Gordon and Betty Moore Foundation The Scientific Enterprise Edward (Ned) Hall, Committee Member; Chair, Department of Philosophy, Harvard University National Science Foundation’s Interests and Goals for the Study Joan Ferrini-Mundy, Chief Operating Officer, National Science Foundation Perspectives on Reproducibility and Replication: Scientific Societies, Part I Panelists, primarily leaders from U.S. scientific societies and organizations, have been asked to focus on the following topics: • Within your field of science, what is the level of awareness, interest, concern, and involvement in reproducibility and replicability (R&R) of research results? • Are there specific areas within your field of science that are more likely to have issues with reproducing scientific results? • What reproducibility challenges does your field of science face with cross disciplinary research? Behavioral and Social Sciences William G. Jacoby, Department of Political Science, Michigan State University; Editor, American Journal of Political Science Howard S. Kurtzman, Acting Executive Director, Science Directorate, American Psychological Association Felice J. Levine, Executive Director and Ethics Officer, American Educational Research Association Physical Sciences Kate Kirby, Chief Executive Officer, American Physical Society David Sholl, John F. Brock III School Chair, School of Chemical & Biomolecular Engineering, Georgia Institute of Technology Statistics Ron Wasserstein, Executive Director, American Statistical Association Earth Sciences 167

Prepublication copy, uncorrected proofs. Brooks Hanson, Senior Vice President Publications, American Geophysical Union Engineering Philip DiVietro, Managing Director of Publishing, American Society of Mechanical Engineers John Baillieul, Distinguished Professor, Department of Mechanical Engineering, Boston University Public Comments WEDNESDAY, DECEMBER 13, 2017 Welcome, Day One and Day Two Overviews Harvey Fineberg, Committee Chair Perspectives on Reproducibility and Replication: Scientific Societies and Agencies, Part II (1) Panelists, primarily leaders from U.S. scientific societies and organizations, have been asked to focus on the following topics:  Within your field of science, what is the level of awareness, interest, concern, and involvement in reproducibility and replicability (R&R) of research results?  Are there specific areas within your field of science that are more likely to have issues with reproducing scientific results?  What reproducibility challenges does your field of science face with cross disciplinary research? Life Sciences Yvette Seger, Director of Science Policy, Federation of American Societies for Experimental Biology Reproducibility of Scientific Research within the Agencies: Patricia Valdez, Extramural Research Integrity Officer, National Institutes of Health Anne Plant, Biosystems and Biomaterials Division, National Institutes of Standards and Technology (2) International panelists have been asked to focus on the following topics:  What is the level of awareness, interest, concern, and involvement in reproducibility and replicability (R&R) of research results within your national scientific societies?  Are there specific areas of science that are more likely to have issues with reproducing scientific results?  What R&R issues exist for cross disciplinary research? Eric-Jan Wagenmakers, Professor of Psychology, University of Amsterdam Jean Phillipe de Jong, The Dutch Royal Society of Sciences 168

Prepublication copy, uncorrected proofs. (3) The editor of major a cross-disciplinary journal was asked to focus on the following questions:  Can journals assess levels of R&R across science?  What R&R challenges does cross-disciplinary research pose that can be addressed by journals?  Are cross-disciplinary papers handled differently from “pure” science papers in terms of peer review or publishing decisions?  What R&R challenges does cross-disciplinary research pose that can be addressed by journals? Veronique Kiermer, Executive Editor, Public Library of Science Reporting of Reproducibility Issues in Science Richard Harris, Science Correspondent, National Public Radio Public Comments 169

Prepublication copy, uncorrected proofs. Second Meeting February 22–23, 2018 THURSDAY, FEBRUARY 22, 2018 Welcome and Introductions Harvey Fineberg, Committee Chair; President, Gordon and Betty Moore Foundation Perspectives on Reproducibility and Replication: American Economic Association The speaker has been asked to focus on the following questions:  Within economics, what is the level of awareness, interest, concern, and involvement in reproducibility and replicability (R&R) of research results?  Are there specific areas within economics that are more likely to have issues with reproducing scientific results?  What reproducibility challenges does economics face with cross disciplinary research? Margaret Levenstein, Professor of Economics and Director, Inter-university Consortium for Political and Social Research, University of Michigan, Panel 1: Overview of Extent of Reproducibility Issues in Scientific and Engineering Research The panelists have been asked to focus on the following session questions:  How extensive is the lack of reproducibility in research results in science and engineering, in general?  At what level does a lack of reproducibility become a problem for the wellbeing of science or engineering?  Does the lack to reproduce scientific results impact the public perception of specific scientific fields and/or science and engineering in general? John Ioannidis, C.F. Rehnborg Chair in Disease Prevention and Co-Director, Meta-Research Innovation Center, Stanford University Brian Nosek, director, Center for Open Science and Professor of Psychology, University of Virginia Daniel Sarewitz, Co-Director, Consortium for Science, Policy & Outcomes, and Professor of Science and Society, Arizona State University Panel 2, Part 1: Reproducibility Issues in Computational Sciences and Statistics The panelists in this session have been asked to address the session questions (above) with a focus on the management of computational code and data. David Madigan, Executive Vice President and Dean of the Faculty of Arts and Sciences, Columbia University Arjun Kumar Manrai, Department of Biomedical Informatics, Harvard University 170

Prepublication copy, uncorrected proofs. Panel 2, Part 2: Reproducibility Issues in Computational Sciences and Statistics The panelists in this session have been asked to address the session questions (above) with a focus on the impact of the misuse of statistics in research. Giovanni Parmigiani, Harvard T.H. Chan School of Public Health and Dana Farber Cancer Institute Steven Goodman, Professor of Medicine and Epidemiology, Associate Dean for Clinical and Translational Research, and Co-Director, Meta-research Innovation Center at Stanford (METRICS), Stanford University Panel 2, Part 3: Reproducibility Issues in Economics and Social Science The panelists in this session have been asked to address the session questions (above) as they relate to economics, social sciences, and psychology. Paul L. Joskow, Elizabeth and James Killian Professor of Economics, Emeritus Department of Economics, Massachusetts Institute of Technology Arthur “Skip” Lupia, Hal R Varian Collegiate Professor of Political Science, University of Michigan Joseph Simmons, Professor of Operations, Information, and Decisions, Wharton School, University of Pennsylvania Public Comments FRIDAY, FEBRUARY 23, 2018 Welcome and Introductions Harvey Fineberg, Committee Chair Panel 3: Reproducibility Issues in Engineering Gianluca Setti, Politecnico di Torino, Italy, and IEEE editor Panel 4: Reporting of Reproducibility Issues in Science Christie Aschwanden, Lead Science Editor, FiveThirtyEight Laura Helmuth, Science Editor, Washington Post Public Comments 171

Prepublication copy, uncorrected proofs. Third Meeting WEDNESDAY, APRIL 18, 2018 Welcome and Introductions Harvey Fineberg, Committee Chair; President, Gordon and Betty Moore Foundation Perspectives on Scientific Progress and Irreproducibility Richard Shiffrin, Department of Psychological and Brain Sciences, Indiana University Panel 1: Reproducibility in the Physical and Earth Sciences Joan Brennecke, Cockrell Family Chair in Engineering #16, McKetta Department of Chemical Engineering, University of Texas at Austin Peter Mohr, Atomic Physics Division, National Institute of Standards and Technology Panel 2: Reproducibility in Industry and Industrial Engineering Carl Ascoli, Chief Science Officer, Rockland Immunochemicals William Lyons, Director, Global Research & Development Strategy for the Global Technology Organization, Boeing Research & Technology Introduction to Economics and Reproducibility Daniel L. Goroff, Vice President and Program Director, The Alfred P. Sloan Foundation Panel 3: The Economics of Addressing Reproducibility Issues in Science Heidi Williams, Department of Economics, Massachusetts Institute of Technology Myron P. Gutmann, Professor of History and Director of the Institute of Behavioral Science, University of Colorado in Boulder Richard Freeman, Herbert Ascherman Chair in Economics, Harvard University Brent Goldfarb, Management & Organizations Department and Dingman Center for Entrepreneurship, University of Maryland Public Comments 172

Prepublication copy, uncorrected proofs. Fourth Meeting MAY 9, 2018 Welcome and Introductions Harvey Fineberg, Committee Chair; President, Gordon and Betty Moore Foundation Panel: Perspectives on Reproducibility and Replication of Results in Climate Science  How has the awareness and understanding about reproducibility and replication in climate science evolved over recent years?  Are there specific challenges regarding reproducibility that you have encountered or are aware of? Identify specific steps that are being taken, either by you or by others, to ameliorate these issues.  Highlight historical and potential new approaches to reproducing and replicating climate science research using examples such as paleoclimate data to test models and estimate uncertainties. Michael Evans, Department of Geology, University of Maryland Gavin Schmidt, Director, Goddard Institute for Space Studies, National Aeronautics and Space Administration Rich Loft, Director, Technology Development Division, National Center for Atmospheric Research Andrea Dutton, Department of Geological Sciences, University of Florida Wrap-up Harvey Fineberg, Committee chair 173

Prepublication copy, uncorrected proofs. Fifth Meeting MAY 31, 2018 Welcome and Call to Order Harvey Fineberg, Committee Chair; President, Gordon and Betty Moore Foundation Panel: International Perspectives on Reproducibility and Replication in Science and Engineering  Are there specific examples in your country/region where a lack of reproducibility and replicability in research results has led to doubt about reported results more broadly? Are reproducibility and replication of research results a global concern or is it a concern focused within specific countries?  Are there particular scientific fields in which lack of reproducibility and replicability is more/less of a concern?  Are there any concrete actions that organizations (e.g. funders, publishers, societies) in your country or region have taken to address concerns about reproducibility and replicability? What actions should they take?  Should the research community work regionally and/or globally to address concerns about reproducibility and replicability? If so, what should be the priorities? Laura Fierce, Environmental and Climate Sciences Department, Brookhaven National Laboratory, and member, Executive Committee, Global Young Academy [in person] Koen Vermeir, French National Centre for Scientific Research, former co-chair, Scientific Excellence and Open Science Programs, and member, Executive Committee, Global Young Academy [via Zoom] Harry Xia, President, Alliance for Scientific Editing in China [in person] Suman Chakraborty, Department of Mechanical Engineering, Indian Institute of Technology Kharagpur, India [via Zoom] 174

Next: Appendix C: Recommendations Grouped by Stakeholder »
Reproducibility and Replicability in Science Get This Book
×
Buy Prepub | $69.00 Buy Paperback | $65.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

One of the pathways by which the scientific community confirms the validity of a new scientific discovery is by repeating the research that produced it. When a scientific effort fails to independently confirm the computations or results of a previous study, some fear that it may be a symptom of a lack of rigor in science, while others argue that such an observed inconsistency can be an important precursor to new discovery.

Concerns about reproducibility and replicability have been expressed in both scientific and popular media. As these concerns came to light, Congress requested that the National Academies of Sciences, Engineering, and Medicine conduct a study to assess the extent of issues related to reproducibility and replicability and to offer recommendations for improving rigor and transparency in scientific research.

Reproducibility and Replicability in Science defines reproducibility and replicability and examines the factors that may lead to non-reproducibility and non-replicability in research. Unlike the typical expectation of reproducibility between two computations, expectations about replicability are more nuanced, and in some cases a lack of replicability can aid the process of scientific discovery. This report provides recommendations to researchers, academic institutions, journals, and funders on steps they can take to improve reproducibility and replicability in science.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!