Appendix C
Workshop Agenda
Enhancing Scientific Reproducibility in Biomedical Research Through
Transparent Reporting—A Workshop
September 25–26, 2019
National Academy of Sciences Building, Lecture Room
2101 Constitution Avenue, NW, Washington, DC 20418
DAY 1: September 25, 2019
8:00 a.m. | Breakfast available outside the Lecture Room |
8:30 a.m. | Welcome and opening remarks |
HARVEY FINEBERG, Workshop Chair | |
President |
|
Gordon and Betty Moore Foundation |
|
Highlights and related recommendations from the National Academies report on Reproducibility and Replicability in Science | |
9:15 a.m. | Q&A with audience |
SESSION I | CULTIVATING TRANSPARENT REPORTING IN BIOMEDICAL RESEARCH |
Session Objectives:
- Highlight current efforts by researchers, institutions, funders, and journals to increase transparency in proposing and reporting preclinical biomedical research.
- Discuss the incentives, disincentives, challenges, and opportunities for researchers when it comes to transparent reporting of preclinical biomedical research (e.g., pressure to publish, institutional resources, training, funding).
- Discuss experience with implementation of policies to encourage transparent reporting across the biomedical research life cycle.
- Consider the role of stakeholders in supporting a cultural shift toward transparent reporting in preclinical biomedical research. For more information on cultural barriers as sources of non-reproducibility, see p. 58, p. 97, and p. 104 of the National Academies’ Reproducibility and Replicability in Science consensus study report.
9:30 a.m. | Opening remarks by session moderator |
ALEXA MCCRAY | |
Professor of Medicine |
|
Harvard Medical School |
|
9:40 a.m. | A researcher (early career) perspective |
YARIMAR CARRASQUILLO | |
Investigator |
|
National Center for Complementary and Integrative Health, National Institutes of Health (NIH) |
|
9:55 a.m. | A researcher/researcher support perspective |
BRIAN NOSEK | |
Co-founder |
|
Center for Open Science |
|
10:10 a.m. | A researcher (later career)/society publisher perspective |
ARTURO CASADEVALL | |
Professor, Molecular Microbiology and Immunology, Johns Hopkins University |
|
Editor-in-Chief, mBio |
|
10:25 a.m. | An NIH perspective |
CARRIE WOLINETZ | |
Acting Chief of Staff and Associate Director for Science Policy |
|
Office of the Director, NIH |
|
10:40 a.m. | Audience Q&A with the panel |
Discussion questions:
|
|
11:10 a.m. | BREAK |
SESSION II | ANSWERING THE CALL FOR CHANGE: LESSONS LEARNED AND BEST PRACTICES |
Session Objectives:
- Consider lessons learned from institutional and/or field-specific best practices for increased transparency in reporting rigor elements (i.e., research design, methodology, analysis, interpretation, and reporting of results) that are generalizable across biomedical research domains.
- Consider available tools and best practices for increased transparent reporting that support researchers and are generalizable across biomedical research domains.
- Discuss the roles of educational institutions, professional societies, researchers, and funders in improving computational reproducibility (Reproducibility and Replicability in Science Recommendation 6-6).
- Discuss how funding agencies and organizations could invest in research and development of open source, usable tools, and infrastructure that support reproducibility for a broad range of studies across different domains in a seamless fashion, as well as in outreach to inform and train researchers on best practices (Reproducibility and Replicability in Science Recommendation 6-1).
11:30 a.m. | Opening remarks by session moderator |
VERONIQUE KIERMER | |
Executive Editor |
|
Public Library of Science (PLOS) |
11:40 a.m. | A clinical researcher perspective: Lessons from the SPIRIT initiative |
AN-WEN CHAN | |
Phelan Scientist, Women’s College Research Institute |
|
Associate Professor, University of Toronto |
|
11:50 a.m. | An institution perspective |
GEETA SWAMY | |
Vice Dean for Scientific Integrity |
|
Associate Vice President for Research |
|
Duke University |
|
12:00 p.m. | A funder perspective |
MAGALI HAAS | |
Chief Executive Officer and President |
|
Cohen Veterans Bioscience |
|
12:10 p.m. | Moderated panel discussion among speakers |
12:30 p.m. | Audience Q&A with the panel |
Discussion questions: | |
|
|
1:00 p.m. | BREAK (Lunch available outside the Lecture Room) |
SESSION III | STAKEHOLDER PERSPECTIVES ON CHECKLISTS AND GUIDELINES |
Session Objectives:
- Discuss journal and funder assessments of researchers’ adherence to transparent reporting guidelines, including discussion of the effectiveness of checklists.
- Highlight empirical assessments of checklist application from funders, journals, and researchers; and
- Consider practical application and effectiveness of checklists and guidelines to encourage or require transparent reporting of preclinical biomedical research.
- Discuss how funders could require thoughtful discussion in grant applications of how uncertainties will be evaluated, along with any relevant issues regarding replicability and computational reproducibility (Reproducibility and Replicability in Science Recommendation 6-9).
- Discuss how journals and scientific societies could disclose their policies relevant to achieving reproducibility and replicability, and how journals could be encouraged to set and implement desired standards of reproducibility and replicability and adopt policies to reduce the likelihood of non-replicability (Reproducibility and Replicability in Science Recommendation 6-7).
2:00 p.m. | Opening remarks by session moderator |
BARRY COLLER | |
Physician-in-Chief, Vice President for Medical Affairs, and David Rockefeller Professor |
|
The Rockefeller University |
|
2:10 p.m. | The checklist approach at life science journals—challenges and opportunities |
SOWMYA SWAMINATHAN | |
Head of Editorial Policy |
|
Nature Research |
|
MALCOLM MACLEOD | |
Professor |
|
University of Edinburgh |
|
2:30 p.m. | An NIH funder perspective |
SHAI SILBERBERG | |
Director, Research Quality |
|
National Institute of Neurological Disorders and Stroke, NIH |
|
2:40 p.m. | Moderated panel discussion among speakers |
3:10 p.m. | Audience Q&A with the panel |
Discussion questions: | |
|
|
SESSION IV PART 1 | TOWARD MINIMAL REPORTING STANDARDS FOR PRECLINICAL BIOMEDICAL RESEARCH |
Session Objective:
- Discuss opportunities for improving the consistency of reporting guidelines and requirements for rigor and transparency by journals, funders, and institutions across the biomedical research life cycle.
4:00 p.m. | Discussion with audience on potential steps stakeholders could take to support harmonizing reporting guidelines |
HARVEY FINEBERG, Workshop Chair and session moderator | |
President |
|
Gordon and Betty Moore Foundation |
|
BENEDICT KOLBER | |
Associate Professor |
|
Duquesne University |
|
RICHARD NAKAMURA | |
Former Director (Retired) |
|
Center for Scientific Review, NIH |
|
FRANKLIN SAYRE | |
STEM Librarian |
|
Thompson Rivers University |
|
VALDA VINSON | |
Editor, Research |
|
Science |
|
5:00 p.m. | ADJOURN WORKSHOP DAY 1 |
DAY 2: September 26, 2019
8:00 a.m. | Breakfast available outside the Lecture Room |
8:30 a.m. | Welcome and overview of day 1 |
HARVEY FINEBERG, Workshop Chair | |
President |
|
Gordon and Betty Moore Foundation |
|
9:00 a.m. | Keynote address |
MARCIA MCNUTT | |
President |
|
National Academy of Sciences |
|
9:20 a.m. | Q&A session |
9:30 a.m. | BREAK |
SESSION IV PART 2 | TOWARD MINIMAL REPORTING STANDARDS FOR PRECLINICAL BIOMEDICAL RESEARCH |
Session Objectives:
- Consider approaches to compare reporting of rigor elements proposed in grant applications to those included in publications.
- Suggest stakeholder actions to encourage transparent reporting and practical next steps toward establishing minimal reporting standards for preclinical biomedical research.
10:00 a.m. | Opening remarks by session moderator |
HARVEY FINEBERG, Workshop Chair | |
President |
|
Gordon and Betty Moore Foundation |
|
10:10 a.m. | An early career researcher perspective |
MICHAEL KEISER | |
Assistant Professor |
|
University of California, San Francisco |
|
10:20 a.m. | An institution perspective |
MELISSA RETHLEFSEN | |
Associate Dean, George A. Smathers Libraries |
|
Fackler Director, Health Science Center Libraries |
|
University of Florida |
|
10:30 a.m. | A research educator perspective |
STEVEN GOODMAN | |
Professor of Medicine and Health Research and Policy |
|
Co-director, Meta-Research Innovation Center at Stanford |
|
Stanford University |
|
10:40 a.m. | Moderated panel discussion among speakers |
11:10 a.m. | Small group table discussion and reporting |
Discussion questions: | |
|
|
|
12:25 p.m. | Workshop wrap-up and concluding discussion with audience |
12:30 p.m. | ADJOURN WORKSHOP DAY 2 |