National Academies Press: OpenBook

Reproducibility and Replicability in Science (2019)

Chapter: Appendix B: Agendas of Open Committee Meetings

« Previous: Appendix A: Biographical Sketches of Committee Members and Staff
Suggested Citation:"Appendix B: Agendas of Open Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

Appendix B

Agendas of Open Committee Meetings

First Meeting
DECEMBER 12-13, 2017

TUESDAY, DECEMBER 12, 2017

Welcome and Introductions

Mary Ellen O’Connell, Executive Director, Division of Behavioral and Social Science and Education

Harvey Fineberg, Committee Chair; President, Gordon and Betty Moore Foundation

The Scientific Enterprise

Edward (Ned) Hall, Committee Member; Chair, Department of Philosophy, Harvard University

National Science Foundation’s Interests and Goals for the Study

Joan Ferrini-Mundy, Chief Operating Officer, National Science Foundation

Perspectives on Reproducibility and Replication: Scientific Societies, Part I

Panelists, primarily leaders from U.S. scientific societies and organizations, have been asked to focus on the following topics:

Suggested Citation:"Appendix B: Agendas of Open Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
  • Within your field of science, what is the level of awareness, interest, concern, and involvement in reproducibility and replicability of research results?
  • Are there specific areas within your field of science that are more likely to have issues with reproducing scientific results?
  • What reproducibility challenges does your field of science face with cross disciplinary research?

Behavioral and Social Sciences

William G. Jacoby, Department of Political Science, Michigan State University; Editor, American Journal of Political Science

Howard S. Kurtzman, Acting Executive Director, Science Directorate, American Psychological Association

Felice J. Levine, Executive Director and Ethics Officer, American Educational Research Association

Physical Sciences

Kate Kirby, Chief Executive Officer, American Physical Society

David Sholl, John F. Brock III School Chair, School of Chemical and Bio-molecular Engineering, Georgia Institute of Technology

Statistics

Ron Wasserstein, Executive Director, American Statistical Association

Earth Sciences

Brooks Hanson, Senior Vice President Publications, American Geophysical Union

Engineering

Philip DiVietro, Managing Director of Publishing, American Society of Mechanical Engineers

John Baillieul, Distinguished Professor, Department of Mechanical Engineering, Boston University

Public Comments

Suggested Citation:"Appendix B: Agendas of Open Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

WEDNESDAY, DECEMBER 13, 2017

Welcome, Day One and Day Two Overviews

Harvey Fineberg, Committee Chair

Perspectives on Reproducibility and Replication: Scientific Societies and Agencies, Part II

  1. Panelists, primarily leaders from U.S. scientific societies and organizations, have been asked to focus on the following topics:
    • What reproducibility challenges does your field of science face with cross-disciplinary research?
    • Within your field of science, what is the level of awareness, interest, concern, and involvement in reproducibility and replicability of research results?
    • Are there specific areas within your field of science that are more likely to have issues with reproducing scientific results?

Life Sciences

Yvette Seger, Director of Science Policy, Federation of American Societies for Experimental Biology

Reproducibility of Scientific Research within the Agencies

Patricia Valdez, Extramural Research Integrity Officer, National Institutes of Health

Anne Plant, Biosystems and Biomaterials Division, National Institute of Standards and Technology

  1. International panelists have been asked to focus on the following topics:
    • What is the level of awareness, interest, concern, and involvement in reproducibility and replicability of research results within your national scientific societies?
    • Are there specific areas of science that are more likely to have issues with reproducing scientific results?
    • What reproducibility and replicability issues exist for cross-disciplinary research?
Suggested Citation:"Appendix B: Agendas of Open Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

Eric-Jan Wagenmakers, Professor of Psychology, University of Amsterdam

Jean Phillipe de Jong, The Dutch Royal Society of Sciences

  1. The editor of a major cross-disciplinary journal was asked to focus on the following questions:
    • Can journals assess levels of R&R across science?
    • What R&R challenges does cross-disciplinary research pose that can be addressed by journals?
    • Are cross-disciplinary papers handled differently from “pure” science papers in terms of peer review or publishing decisions?
    • What R&R challenges does cross-disciplinary research pose that can be addressed by journals?

Veronique Kiermer, Executive Editor, Public Library of Science

Reporting of Reproducibility Issues in Science

Richard Harris, Science Correspondent, National Public Radio

Public Comments

Suggested Citation:"Appendix B: Agendas of Open Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

Second Meeting
FEBRUARY 22-23, 2018

THURSDAY, FEBRUARY 22, 2018

Welcome and Introductions

Harvey Fineberg , Committee Chair; President, Gordon and Betty Moore Foundation

Perspectives on Reproducibility and Replication: American Economic Association

The speaker has been asked to focus on the following questions:

  • Within economics, what is the level of awareness, interest, concern, and involvement in reproducibility and replicability of research results?
  • Are there specific areas within economics that are more likely to have issues with reproducing scientific results?
  • What reproducibility challenges does economics face with cross-disciplinary research?

Margaret Levenstein, Professor of Economics and Director, Inter-university Consortium for Political and Social Research, University of Michigan

Panel 1: Overview of Extent of Reproducibility Issues in Scientific and Engineering Research

The panelists have been asked to focus on the following session questions:

  • How extensive is the lack of reproducibility in research results in science and engineering, in general?
  • At what level does a lack of reproducibility become a problem for the wellbeing of science or engineering?
  • Does the lack to reproduce scientific results impact the public perception of specific scientific fields and/or science and engineering in general?

John Ioannidis, C.F. Rehnborg Chair in Disease Prevention and Co-Director, Meta-Research Innovation Center, Stanford University

Brian Nosek, Director, Center for Open Science and Professor of Psychology, University of Virginia

Suggested Citation:"Appendix B: Agendas of Open Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

Daniel Sarewitz, Co-Director, Consortium for Science, Policy & Outcomes, and Professor of Science and Society, Arizona State University

Panel 2, Part 1: Reproducibility Issues in Computational Sciences and Statistics

The panelists in this session have been asked to address the session questions (above) with a focus on the management of computational code and data.

David Madigan, Executive Vice President and Dean of the Faculty of Arts and Sciences, Columbia University

Arjun Kumar Manrai, Department of Biomedical Informatics, Harvard University

Panel 2, Part 2: Reproducibility Issues in Computational Sciences and Statistics

The panelists in this session have been asked to address the session questions (above) with a focus on the impact of the misuse of statistics in research.

Giovanni Parmigiani, Harvard T.H. Chan School of Public Health and Dana Farber Cancer Institute

Steven Goodman, Professor of Medicine and Epidemiology, Associate Dean for Clinical and Translational Research, and Co-Director, Meta-Research Innovation Center at Stanford (METRICS), Stanford University

Panel 2, Part 3: Reproducibility Issues in Economics and Social Science

The panelists in this session have been asked to address the session questions (above) as they relate to economics, social sciences, and psychology.

Paul L. Joskow, Elizabeth and James Killian Professor of Economics, Emeritus Department of Economics, Massachusetts Institute of Technology

Arthur (Skip) Lupia, Hal R. Varian Collegiate Professor of Political Science, University of Michigan

Suggested Citation:"Appendix B: Agendas of Open Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

Joseph Simmons, Professor of Operations, Information, and Decisions, Wharton School, University of Pennsylvania

Public Comments

FRIDAY, FEBRUARY 23, 2018

Welcome and Introductions

Harvey Fineberg, Committee Chair

Panel 3: Reproducibility Issues in Engineering

Gianluca Setti, Politecnico di Torino, Italy, and IEEE editor

Panel 4: Reporting of Reproducibility Issues in Science

Christie Aschwanden, Lead Science Editor, FiveThirtyEight

Laura Helmuth, Science Editor, The Washington Post

Public Comments

Suggested Citation:"Appendix B: Agendas of Open Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

Third Meeting
WEDNESDAY, APRIL 18, 2018

Welcome and Introductions

Harvey Fineberg, Committee Chair; President, Gordon and Betty Moore Foundation

Perspectives on Scientific Progress and Irreproducibility

Richard Shiffrin, Department of Psychological and Brain Sciences, Indiana University Bloomington

Panel 1: Reproducibility in the Physical and Earth Sciences

Joan Brennecke, Cockrell Family Chair in Engineering #16, McKetta Department of Chemical Engineering, The University of Texas at Austin

Peter Mohr, Atomic Physics Division, National Institute of Standards and Technology

Panel 2: Reproducibility in Industry and Industrial Engineering

Carl Ascoli, Chief Science Officer, Rockland Immunochemicals

William Lyons, Director, Global Research and Development Strategy for the Global Technology Organization, Boeing Research and Technology

Introduction to Economics and Reproducibility

Daniel L. Goroff, Vice President and Program Director, Alfred P. Sloan Foundation

Panel 3: The Economics of Addressing Reproducibility Issues in Science

Heidi Williams, Department of Economics, Massachusetts Institute of Technology

Myron P. Gutmann, Professor of History and Director of the Institute of Behavioral Science, University of Colorado Boulder

Richard Freeman, Herbert Ascherman Chair in Economics, Harvard University

Brent Goldfarb, Management and Organizations Department and Dingman Center for Entrepreneurship, University of Maryland

Public Comments

Suggested Citation:"Appendix B: Agendas of Open Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

Fourth Meeting
MAY 9, 2018

Welcome and Introductions

Harvey Fineberg, Committee Chair; President, Gordon and Betty Moore Foundation

Panel: Perspectives on Reproducibility and Replication of Results in Climate Science

The panelists have been asked to focus on the following session questions:

  • How has the awareness and understanding about reproducibility and replication in climate science evolved over recent years?
  • Are there specific challenges regarding reproducibility that you have encountered or are aware of? Identify specific steps that are being taken, either by you or by others, to ameliorate these issues.
  • Highlight historical and potential new approaches to reproducing and replicating climate science research using examples such as paleoclimate data to test models and estimate uncertainties.

Michael Evans, Department of Geology, University of Maryland

Gavin Schmidt, Director, Goddard Institute for Space Studies, National Aeronautics and Space Administration

Rich Loft, Director, Technology Development Division, National Center for Atmospheric Research

Andrea Dutton, Department of Geological Sciences, University of Florida

Wrap-Up

Harvey Fineberg, Committee Chair

Suggested Citation:"Appendix B: Agendas of Open Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×

Fifth Meeting
MAY 31, 2018

Welcome and Call to Order

Harvey Fineberg, Committee Chair; President, Gordon and Betty Moore Foundation

Panel: International Perspectives on Reproducibility and Replication in Science and Engineering

The panelists have been asked to focus on the following session questions:

  • Are there specific examples in your country/region where a lack of reproducibility and replicability in research results has led to doubt about reported results more broadly? Are reproducibility and replication of research results a global concern or is it a concern focused within specific countries?
  • Are there particular scientific fields in which lack of reproducibility and replicability is more/less of a concern?
  • Are there any concrete actions that organizations (e.g., funders, publishers, societies) in your country or region have taken to address concerns about reproducibility and replicability? What actions should they take?
  • Should the research community work regionally and/or globally to address concerns about reproducibility and replicability? If so, what should be the priorities?

Laura Fierce, Environmental and Climate Sciences Department, Brookhaven National Laboratory, and member, Executive Committee, Global Young Academy [in person]

Koen Vermeir, French National Centre for Scientific Research, former Co-Chair, Scientific Excellence and Open Science Programs, and Member, Executive Committee, Global Young Academy [via Zoom]

Harry Xia, President, Alliance for Scientific Editing in China [in person]

Suman Chakraborty, Department of Mechanical Engineering, Indian Institute of Technology Kharagpur, India [via Zoom]

Suggested Citation:"Appendix B: Agendas of Open Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 199
Suggested Citation:"Appendix B: Agendas of Open Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 200
Suggested Citation:"Appendix B: Agendas of Open Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 201
Suggested Citation:"Appendix B: Agendas of Open Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 202
Suggested Citation:"Appendix B: Agendas of Open Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 203
Suggested Citation:"Appendix B: Agendas of Open Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 204
Suggested Citation:"Appendix B: Agendas of Open Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 205
Suggested Citation:"Appendix B: Agendas of Open Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 206
Suggested Citation:"Appendix B: Agendas of Open Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 207
Suggested Citation:"Appendix B: Agendas of Open Committee Meetings." National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: 10.17226/25303.
×
Page 208
Next: Appendix C: Recommendations Grouped by Stakeholder »
Reproducibility and Replicability in Science Get This Book
×
Buy Paperback | $65.00 Buy Ebook | $54.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

One of the pathways by which the scientific community confirms the validity of a new scientific discovery is by repeating the research that produced it. When a scientific effort fails to independently confirm the computations or results of a previous study, some fear that it may be a symptom of a lack of rigor in science, while others argue that such an observed inconsistency can be an important precursor to new discovery.

Concerns about reproducibility and replicability have been expressed in both scientific and popular media. As these concerns came to light, Congress requested that the National Academies of Sciences, Engineering, and Medicine conduct a study to assess the extent of issues related to reproducibility and replicability and to offer recommendations for improving rigor and transparency in scientific research.

Reproducibility and Replicability in Science defines reproducibility and replicability and examines the factors that may lead to non-reproducibility and non-replicability in research. Unlike the typical expectation of reproducibility between two computations, expectations about replicability are more nuanced, and in some cases a lack of replicability can aid the process of scientific discovery. This report provides recommendations to researchers, academic institutions, journals, and funders on steps they can take to improve reproducibility and replicability in science.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!