National Academies Press: OpenBook

Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop (2020)

Chapter: 7 Stakeholder Opportunities for Promoting Transparent Reporting

« Previous: 6 Toward Minimal Reporting Standards for Preclinical Biomedical Research
Suggested Citation:"7 Stakeholder Opportunities for Promoting Transparent Reporting." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
Page 91
Suggested Citation:"7 Stakeholder Opportunities for Promoting Transparent Reporting." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
Page 92
Suggested Citation:"7 Stakeholder Opportunities for Promoting Transparent Reporting." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
Page 93
Suggested Citation:"7 Stakeholder Opportunities for Promoting Transparent Reporting." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
Page 94
Suggested Citation:"7 Stakeholder Opportunities for Promoting Transparent Reporting." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
Page 95
Suggested Citation:"7 Stakeholder Opportunities for Promoting Transparent Reporting." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
Page 96
Suggested Citation:"7 Stakeholder Opportunities for Promoting Transparent Reporting." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
Page 97
Suggested Citation:"7 Stakeholder Opportunities for Promoting Transparent Reporting." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
Page 98
Suggested Citation:"7 Stakeholder Opportunities for Promoting Transparent Reporting." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
Page 99
Suggested Citation:"7 Stakeholder Opportunities for Promoting Transparent Reporting." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
Page 100

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

7 Stakeholder Opportunities for Promoting Transparent Reporting For the final session of the workshop, participants broke into small groups to consider the roles and responsibilities of researchers, publish- ers, institutions, and funders in improving transparent reporting of bio- medical research. In preparation for the small group activity, Harvey Fineberg shared his impressions from the workshop presentations and discussions thus far. There is simultaneously readiness and reluctance to act on improv- ing transparency, he observed. He noted that ideas and attitudes about transparency seem to be converging in the right direction, and resources are being mobilized. However, he said, there remain “cultural barriers to introducing, implementing, and fulfilling the aspirations of open science through transparency.” Tools are available to researchers, publishers, and funders to promote rigorous research and transparent reporting. The task now is to determine what may be missing, what needs to be improved, and how stakeholder needs can be addressed, Fineberg said. During the small group discussions, he asked participants to identify what they con- sider the most important opportunities for moving forward. Over the course of the workshop, perspectives were shared by researchers—from early career scientists to senior investigators and research support staff, as well as representatives of publishers, institu- tions, and funders. Fineberg asked participants to consider how stake- holder interests could be harmonized, and operations aligned, to realize the shared objective of increased transparency and rigor in research. He also asked participants to consider what leadership role each stakeholder plays in driving transparent reporting in biomedical research. 91 PREPUBLICATION COPY—Uncorrected Proofs

92 ENHANCING SCIENTIFIC REPRODUCIBILITY SMALL GROUP DISCUSSION REPORTS1 Upon reconvening in plenary session, a facilitator from each small group summarized the individual participants’ discussion (a summary of points made by small group participants is provided in Box 7-1). Group discussions were guided by the following prompts: • What actions should funders, researchers, institutions, and journals take to drive widespread adoption of minimal reporting standards? • Are reporting categories in guidelines for publishing (e.g., mate- rials, design, analysis, and reporting) relevant for funders? For institutions? For small publishers/professional societies? • What other information or reporting categories would be relevant? • How should funders instruct reviewers of grant applications to reinforce transparent reporting? How much information should funders request (i.e., to what level of detail) in grant applications? Is it possible to obtain sufficient information about transparent reporting in grant applications without dramatic expansion of the application? BOX 7-1 Suggestions by Individual Workshop Participants on Stakeholder Opportunities to Promote Transparent Reporting in Preclinical Biomedical Research Researchers • Researchers may help promote adoption of minimal standards for trans- parency and rigor through a number of actions, including ° Considering ways to include information regarding the reasoning and decision points that go into a research project, in addition to using a minimal checklist, when submitting research applications to funders or manuscripts to journals; and °  Considering ways to leverage the expertise of university librarians and create resources for investigators starting a new laboratory (e.g., look to concepts such as the University of California’s QB3 “startup in a box” framework—a combination of hands-on support, services, and mentorship—which is designed to lower the barriers to innovation for university entrepreneurs interested in starting a company). 1 This small group exercise was intended to engage participants in using what they had learned thus far in the workshop. This section of the proceedings summarizes the small group discussions based on the report by each group’s designated rapporteur and should not be construed as reflecting any consensus of the group. All group responses and propos- als reported are for discussion purposes only. PREPUBLICATION COPY—Uncorrected Proofs

STAKEHOLDER OPPORTUNITIES FOR PROMOTING TRANSPARENT REPORTING 93 •  Researchers across all career stages have opportunities to take action to promote reproducibility through transparent reporting: °  Considering ways to facilitate more substantive feedback on propos- als, particularly for early-stage research, through peer review—inter- nally within institutions as well as through external grant applications; °  Incorporating reproducibility and/or replicability studies into graduate training programs (e.g., coursework, qualifying exams); and °  Promoting a culture of openness and establishing data-sharing best practices within their laboratories. NOTE: Points made by individual workshop participants discussing the role of research- ers in promoting transparent reporting, as reported by Michael Keiser. Institutions •  Institutions have opportunities to address impediments to transparency and reproducibility, including °  Clarifying the criteria for tenure and promotion decisions for early- career researchers; °  Finding ways to incentivize “transparency, openness, contribution to the research corpus, datasets, [and] even being honest about … negative results” (e.g., awards, performance bonuses, incorporation into training throughout researchers’ careers); °  Reframing practices that promote transparency and reproducibility (e.g., checklists, data sharing) as a routine part of the scientific re- search process rather than added administrative burden; °  Identifying and sharing existing research practices that enhance data sharing and transparent reporting; °  Considering better mechanisms for connecting researchers with in- stitutional resources; and °  Engaging leadership and on-the-ground researchers to help build momentum at all levels of the institution. NOTE: Points made by individual workshop participants discussing the role of institu- tions in promoting transparent reporting, as reported by Franklin Sayre and Deborah Sweet. Funders • Funding organizations have opportunities to enhance transparency and reproducibility, including °  Considering a priority set of core standards based on what is most im- portant for enhancing transparent reporting in biomedical research; °  Harmonizing a set of minimal reporting standards across funding groups to help reduce the burden on researchers; °  Supporting training opportunities and the development of tools and platforms that are interoperable across systems; °  Promoting culture change through leadership at all levels of an orga- nization; and continued PREPUBLICATION COPY—Uncorrected Proofs

94 ENHANCING SCIENTIFIC REPRODUCIBILITY BOX 7-1 Continued °  efining metrics for determining whether interventions lead to the D intended outcome. NOTE: Points made by individual workshop participants discussing the role of funders in promoting transparent reporting, as reported by Richard Nakamura. Publishers •  ublishers (and funders) can help “bookend the process” of promoting P transparent reporting through a number of activities, including: °  Working with other stakeholders from across sectors (e.g., funders, researchers, institutions) to harmonize a set of minimal reporting stan- dards to facilitate better science; °  Aligning reporting requirements with transparent reporting practices so expectations are clear for researchers throughout the biomedical research life cycle; °  Considering pragmatic and inclusive approaches to implementing interventions, such as checklists; and °  Helping to coordinate action to address areas of highest need (e.g., study design, statistics, data archiving, sourcing of biological materials). NOTE: Points made by individual workshop participants discussing the role of publish- ers in promoting transparent reporting, as reported by Valda Vinson and Veronique Kiermer. SOURCE: Rapporteurs’ summary of the small group facilitators’ reports. This summary is based on individual comments made by individual workshop participants and should not be construed as reflecting any consensus of workshop attendees. All suggestions and proposals are reported for discussion purposes only. Researchers Michael Keiser reported for a small group that discussed the roles and responsibilities of researchers in improving transparent reporting of biomedical research. The small group discussions focused on potential actions that various stakeholders could take to help promote widespread adoption of minimal standards for transparency and rigor. The first proposal, Keiser described, would be for researchers to report, in a standardized way, the reasoning and decision points that went into a given research project as well as relevant gaps in informa- tion. He acknowledged that few tools exist to facilitate this type of trans- parent reporting, so the development of new, more dynamic tools may PREPUBLICATION COPY—Uncorrected Proofs

STAKEHOLDER OPPORTUNITIES FOR PROMOTING TRANSPARENT REPORTING 95 be needed. Keiser suggested that this information should be included instead of or in addition to minimal reporting standards when submit- ting research applications to funders or manuscripts to journals. The second proposal, Keiser outlined, would be to leverage the exper- tise of university librarians and create a resource for investigators starting a new laboratory. This resource might be similar in concept to the Uni- versity of California’s QB3 “startup in a box” framework—a combination of hands-on support, services, and mentorship—which is designed to lower the barriers to innovation for university entrepreneurs interested in starting a company.2 This type of resource could provide biomedical researchers with a basic set of tools along with guidance on data access and sharing, standardization, compliance, and other information that early stage investigators may need to establish a new lab. Such a resource could also be useful for more senior investigators seeking to implement new tools or approaches. Keiser shared additional points raised during the small group dis- cussion regarding actions specific to trainees, early career researchers, and senior investigators that could help promote reproducibility and replication. For trainees, a few participants suggested that reproducibility and replicability could be incorporated into graduate training programs. For example, graduate students, in consultation with faculty mentors, could be required to reproduce or replicate a study as part of their training grant or qualifying exam. Keiser said the feasibility of reproducing a study might vary by discipline, so perhaps a compilation of existing studies for this type of exercise could be a useful resource. Participants in the small group discussions pointed out that aligning training with a desired change in research practice would help support a culture of reproduc- ibility rather than add administrative burden. Additionally, Keiser noted that participants in the small group discussed the value of implementing formal coursework in experimental research design for undergraduates across institutions. Participants in the small group said that early career researchers may be more directly connected to the research and the data and more comfort- able adopting new tools and practices than senior investigators, Keiser reported. Participants suggested that more substantive feedback, particu- larly for early stage research proposals, through peer review—internally within institutions as well as through external grant applications—would benefit investigators at all stages of their careers. Finally, Keiser relayed that small group participants discussed the influential role of senior investigators in promoting a culture of openness 2 Available at https://qb3.org/support-for-launch (accessed January 10, 2020). PREPUBLICATION COPY—Uncorrected Proofs

96 ENHANCING SCIENTIFIC REPRODUCIBILITY (e.g., code review within the lab, experimental reproducibility and repli- cability by others within the group) and data-sharing practices (e.g., open source tools, preprints, repositories for sharing data with the broader research community). Institutions Franklin Sayre and Deborah Sweet reported for the two small groups that discussed ways institutions can help promote research practices that improve transparency and reproducibility. Sayre laid out a few institutional impediments to transparency and reproducibility, which were raised by small group participants. In par- ticular, he noted that the real and perceived incentives for tenure and promotion decisions do not necessarily align with incentives for trans- parent reporting. The tenure process can be a source of great anxiety for early career researchers. Participants suggested that this anxiety and focus on particular metrics, such as publication in high-impact journals, may promote a culture of poor research practices. Participants from both small groups discussed the need for more clarity for early career researchers on how tenure and promotion decisions are made at institutions. Sweet added that when funders show they value transparency, institutions will follow, which is why cooperation across stakeholders is critical. Sweet reported that small group participants discussed the possibility of assign- ing value to transparent reporting and team science. For example, she suggested that institutions could request tenure support letters to address a nominee’s “quality of contribution” to the research environment. Small group participants also discussed how standards for awarding doctoral degrees should not be based on publication of a given number of papers, but should instead reward “transparency, openness, contribution to the research corpus, datasets, [and] even being honest about your nega- tive results,” Sweet said. She added that positive incentives for applying best practices in research transparency might include awards (e.g., the National Institute of Neurological Disorders and Stroke Landis Award3) or performance bonuses. She highlighted the importance of training doc- toral students from the start and providing training to the broader scien- tific community as well. Training should make researchers aware of the availability of statistical advice and the value of consulting statisticians at the start of the research process. Steven Goodman added that institu- tions have a responsibility to provide adequate capacity for statistical and computational science support for researchers. 3 Available at https://www.ninds.nih.gov/Funding/About-Funding/landis-award-for- outstanding-mentorship (accessed January 10, 2020). PREPUBLICATION COPY—Uncorrected Proofs

STAKEHOLDER OPPORTUNITIES FOR PROMOTING TRANSPARENT REPORTING 97 Sayre described another impediment discussed by small group par- ticipants, which was the perception that activities to support transparent reporting constitute an administrative burden and that reproducibility studies are separate from the general “conduct of science.” Participants highlighted the need to reframe practices that promote transparency and reproducibility (e.g., checklists, data sharing) as a routine part of the sci- entific research process. Sweet shared a suggestion by small group partici- pants that institutions might consider developing an inventory of existing research practices from across different laboratories to help identify and share approaches for enabling data sharing and transparent reporting. Institutions could then consider ways to incentivize the uptake of best practices, Sweet said, and perhaps create disincentives for not meeting particular research practice standards. Sayre relayed a third impediment discussed by small group partici- pants, which was the gap between researchers and institution-level policy, incentives, infrastructure, and research support. Participants suggested there is a need for guidance and better mechanisms to connect researchers with institutional resources. Sweet highlighted one main message raised during the small group discussion: the research culture set by leader- ship matters, and permeates throughout the organization. She added that institutions should create a “culture of openness” in which there are mechanisms for researchers to raise concerns about reproducibility and consequences for noncompliance with transparent reporting. Sayre also acknowledged that institutional leadership is needed to support widespread adoption of transparent reporting. However, he relayed that “[early career] researchers are often more interested in reproducibility, in transparency, and in sharing their work openly.” Small group participants discussed how small laboratories or investigators are often the driving force for grassroots change, so engaging these individuals could help build momentum for reproducibility and transparency within institu- tions. Sweet shared that small group participants raised the importance of approaching issues with transparent reporting from the perspective that most researchers want to “do the right thing and perform science honestly,” rather than taking a punitive approach. Funders Richard Nakamura reported for a small group that discussed the roles and responsibilities of funders in improving transparent reporting of bio- medical research. Participants in the small group considered the benefit of harmonizing a set of minimal reporting standards across organizations so that investigators are not put in a position of having to meet multiple requirements when applying to different funders. Nakamura relayed the PREPUBLICATION COPY—Uncorrected Proofs

98 ENHANCING SCIENTIFIC REPRODUCIBILITY consideration that the application of a common set of minimal reporting standards would need to be flexible across organizations. As discussed throughout the workshop, participants in this small group underscored the need to prioritize a core set of standards based on what is most impor- tant for enhancing transparent reporting in biomedical research. Having tools available will be the key to helping facilitate interoper- ability across systems and reduce the burden on researchers, Nakamura reported. Small group participants pointed out that funders should adapt to support the development of tools and platforms for transparent report- ing as well as replication studies. The small group discussed how funding organizations can play a significant role in promoting culture change. Small group participants emphasized the need for training, Nakamura said. He added that National Institutes of Health training requirements on this topic should be forth- coming. Similar to other small group discussions, participants highlighted the role of leadership, including those at funding organizations, scientific societies, and academia. Department chairs and senior scientists as well as early career scientists will need to speak up about the problems related to reproducibility and replicability and push for culture change. Finally, Nakamura said the small group discussed the importance of having metrics to measure whether or not a given approach leads to improvement. He added that there should be a feedback loop to evaluate whether a given intervention leads to the intended outcome. Publishers Valda Vinson and Veronique Kiermer reported for two small groups, which considered the roles and responsibilities of publishers in improv- ing transparent reporting of biomedical research. Vinson summarized the takeaway message of one small group discussion by saying, “If you want science better, make better science easier.” Kiermer summarized that institutions can help facilitate transparent reporting practices and make them normative, while journals and funders may “bookend the process” by aligning reporting requirements so that expectations are clear for researchers from the beginning. One key topic discussed by small group participants was the need for coordinated action across sectors to address areas of highest need. One small group focused on study design and statistics, which might benefit from training modules developed for researchers and institutions as well as for journals and funders, Vinson said. Kiermer shared two areas of need discussed by the other small group: data archiving and sourcing biological materials. Data archiving is essential for transparent report- ing, Kiermer said, but efforts are not coordinated. Publishers find that PREPUBLICATION COPY—Uncorrected Proofs

STAKEHOLDER OPPORTUNITIES FOR PROMOTING TRANSPARENT REPORTING 99 researchers reach the publication stage without having considered from the beginning how they intend to store datasets, including images. As reported by Kiermer, small group participants suggested that the research flow could be improved (1) if funders set expectations from the begin- ning (e.g., requiring a data management plan as a condition of funding; providing guidance); (2) if institutions provided guidance to researchers on developing a data management plan and use of data repositories; and (3) if journals required reporting of raw data, which could help facilitate review or reproducibility studies following publication. Biological mate- rials are also a source of variability in preclinical studies that contribute to the lack of reproducibility, said Kiermer. Small group participants sug- gested that clear identification of reagents, authentication of biological material, and archiving of reagents in a repository are all ways to help address this issue, Kiermer reported. For example, participants said the MD Anderson Cancer Center requires authentication of cell lines every year and has established a core facility to help carry out this work. The small groups both discussed practical approaches for harmoniz- ing a set of minimal reporting standards to facilitate better science. Vin- son and Kiermer said that as discussed throughout the workshop, small group participants acknowledged that checklists on their own are not a solution—there would need to be a pragmatic and inclusive approach to implementation. Participants suggested that establishing a coordinated framework, which clearly outlines stakeholder expectations (e.g., editors, authors, reviewers, and the public), would help to drive the adoption of interventions, such as checklists, Vinson said. “Less is more” when pro- moting compliance with checklists, Kiermer added. Participants acknowl- edged the challenges of tracking compliance and noted the importance of metadata and persistent identifiers for digital objects “so that there is a clear provenance” from grant application to final publication, Kiermer said. Finally, small group participants emphasized the need for cross-sector leadership and broad sector inclusion, Kiermer relayed. Participants noted the important role of institutional libraries, and the need to include the perspective of researchers. In this regard, Kiermer said that several small group participants supported the establishment of an ongoing National Academies forum as discussed by Marcia McNutt (see Chapter 2). Vinson suggested the concept of a “sea change” initiative to improve reproduc- ibility in biomedical research that could be modeled after the American Association for the Advancement of Science STEM Equity Achievement (SEA)4 Change initiative on diversity, equity, and inclusion in science, or 4 Available at https://seachange.aaas.org (accessed January 10, 2020). PREPUBLICATION COPY—Uncorrected Proofs

100 ENHANCING SCIENTIFIC REPRODUCIBILITY the Athena SWAN Charter5 promoting gender equity in science in the United Kingdom. In essence, Vinson explained, institutions could self- assess their status on the issue, chart a path toward improvement, and be rewarded for taking transformative action. CLOSING STATEMENT In closing the workshop, Fineberg observed that “there is a movement toward higher levels of rigor and transparency in science.” He encour- aged participants to continue to foster this movement and accelerate progress toward improving reproducibility and replicability across the biomedical research life cycle through transparent reporting. 5 Available at https://www.ecu.ac.uk/equality-charters/athena-swan (accessed January 10, 2020). PREPUBLICATION COPY—Uncorrected Proofs

Next: Appendix A: References »
Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop Get This Book
×
Buy Paperback | $50.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Sharing knowledge is what drives scientific progress - each new advance or innovation in biomedical research builds on previous observations. However, for experimental findings to be broadly accepted as credible by the scientific community, they must be verified by other researchers. An essential step is for researchers to report their findings in a manner that is understandable to others in the scientific community and provide sufficient information for others to validate the original results and build on them. In recent years, concern has been growing over a number of studies that have failed to replicate previous results and evidence from larger meta-analyses, which have pointed to the lack of reproducibility in biomedical research.

On September 25 and 26, 2019, the National Academies of Science, Engineering, and Medicine hosted a public workshop in Washington, DC, to discuss the current state of transparency in the reporting of preclinical biomedical research and to explore opportunities for harmonizing reporting guidelines across journals and funding agencies. Convened jointly by the Forum on Drug Discovery, Development, and Translation; the Forum on Neuroscience and Nervous System Disorders; the National Cancer Policy Forum; and the Roundtable on Genomics and Precision Health, the workshop primarily focused on transparent reporting in preclinical research, but also considered lessons learned and best practices from clinical research reporting. This publication summarizes the presentation and discussion of the workshop.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!