Encouraging Compliance with and Continuing the Development of Standards
COMPLIANCE AND ENFORCEMENT
When the principles of publication are not upheld, the scientific enterprise suffers. For example, a recent survey of academic geneticists reported numerous adverse consequences of their colleagues’ withholding publication-related information, data, or materials (Campbell et al., 2002). Among the consequences cited were the inability to replicate published research, substantial publication delays, and abandonment of promising lines of research. In addition, a majority of geneticists felt that withholding detracted from communication in science, slowed scientific progress, harmed peer relationships, and adversely affected the education of students and postdoctoral fellows. Some reported that they had ended collaborations as a result of a colleague’s withholding materials and data.
Other adverse effects of noncompliance with sharing requests were noted in discussions at the workshop. Researchers whose requests for publication-related materials or data are not granted may be unable to do the best science and thus may not remain competitive in their field. In addition, they may spend time and effort in unnecessarily repeating work done by others. Ultimately, some workshop participants felt that if problems of noncompliance are not resolved, the result will be a scientific culture that is rife with conflict.
The obligation for fulfilling community standards for sharing publication-related data and materials lies first with
the authors of a publication. As a practical matter, the author designated in the publication as the corresponding author should be responsible for identifying which coauthor has the materials and other information requested by a third party and should confirm that they are provided when requests are made.
Workshop participants expressed a variety of views of how the scientific community can encourage authors to comply with their obligation to share. Several panelists emphasized that researchers should first take an informal, one-on-one approach to resolve compliance issues, because an initial lack of a response of an author to a request does not necessarily indicate ill intent. Requestors should first consider simple measures, such as telephoning an author to determine why he or she did not respond to a request, and seek resolution before expanding a dispute to involve other parties.
Institutions involved in the scientific enterprise—including journal editorial offices, universities, and funding organizations—should also assume some responsibility for ensuring that authors make available the resources that will enable other researchers to replicate, verify or refute, and build on reported results. If it becomes necessary for a requestor to move beyond straightforward overtures to a paper’s authors, most of the workshop participants agreed that journals should assume primary responsibility for enforcement.
It is not known how many instances of noncompliance are ever brought to the attention of journal editors or other external authorities. However, with respect to cases that are reported to journals, journal editors at the workshop reported a high rate of success in getting authors to share published materials and data, noting that a telephone call or letter from the editor-in-chief or managing editor to an author is often sufficient to resolve problems. Many journal editors stated their willingness to enforce standards of sharing, but one editor expressed concern about adjudicating complicated disputes over the sharing of data and materials, particularly those involving legal wrangling over intellectual property issues.
Sometimes an author might have the responsibility not to honor a request for published materials or data—for example, if bioterrorism is
suspected—until the motivations and credentials of the requestor are validated. Although that subject has not been well explored, it seems appropriate to have journal editors mediate disagreements in such situations.
In addition to helping to resolve cases of noncompliance after publication, scientific journals can play an important role in encouraging authors to comply with standards and the principles of publication before a paper is published by incorporating transparent standards into their official policies. As noted in Chapter 2, a recent examination of the instructions for authors of 56 life-sciences and clinical-medicine journals showed that the specifics of these policies vary considerably (Table 2–1). Furthermore, 25 of the 56 journals do not have any stated policy on sharing of data or materials in their instructions for authors, and clinical journals are more likely than life-sciences journals to lack such a policy.
Another factor that may contribute to noncompliance is that few journals—even among journals that have an official policy for sharing materials or data—provide any statement or policy guidelines as to the consequences for authors who do not comply. Furthermore, most journals do not provide a procedure for registering or publicizing complaints about noncompliance. Of journals that specify sanctions, most say that they would consider denying a noncomplying author further rights to publish in their journals; several participants considered this insufficient to engender compliance. Some workshop participants noted that peer pressure and opinion can be influential in bringing about compliance. A journal might choose to publicly declare an author’s noncompliance (after all honest attempts are exhausted) in a specific section dedicated to this purpose.
Nicholas Cozzarelli, editor-in-chief of the Proceedings of the National Academy of Sciences (PNAS), provided commentary at the workshop about an “escalator policy” for handling complaints about noncompliance. If an author does not provide requested material after receiving a letter from PNAS, it will threaten not to publish future papers by that author. Finally, the PNAS will threaten not to let that author’s coauthors publish again in the journal. Further workshop comment by Laurie
Goodman, executive editor of Genome Research, indicated that the journal would remove a paper from its on-line version and insert a note that the paper had been removed if an author failed to comply with the journal’s policy for sharing materials or data.
Workshop participants discussed the role of funding organizations, universities, and other research institutions in enforcing compliance with standards of publication. According to Philip Campbell, editor-in-chief of Nature, “editors at the moment are acting in isolation in trying to impose sanctions.” He said that knowing what sanctions would be vigorously supported if he contacted a noncomplying author’s research institution or funder might make it easier to enforce compliance.
It is unclear to most scientists what procedures and policies universities have in place for ensuring that their investigators and staff comply with the community’s expectations of authors. It is difficult to find current published policies laid out by universities or organizations that provide research funding regarding formal procedures for resolving problems of noncompliance by their employees or grantees. Although a telephone call or letter to an author from a program director or other representative of an organization can be effective in achieving compliance, funding organizations and universities, like journals, can encourage compliance earlier in the process by developing and enforcing transparent policies that encourage sharing of research resources. For example, the National Institutes of Health (NIH), National Science Foundation (NSF), and other funding organizations, such as the Howard Hughes Medical Institute (HHMI), have policies that reinforce and in some cases extend the standards set by the research community for depositing data in public databases. NIH has also issued a set of principles and guidelines on obtaining and disseminating biomedical research resources (NIH, 1999), although these are not tied specifically to publication. In addition, beginning in October 2003, researchers who receive funding from NIH will have to meet broad standards for data sharing. According to a statement issued by NIH in February 2003, “The NIH expects and supports the timely release and sharing of final research data from NIH-supported studies for use by other researchers. Consequently, investiga-
tors submitting an NIH application seeking $500,000 or more…are expected to include a plan for data sharing or state why data sharing is not possible” (NIH, 2003). To facilitate such sharing, NIH will allow researchers to request, in their original grant application or as a supplement to an existing grant, funds for sharing or archiving data.
One reason that researchers have cited for not sharing published materials is the time, effort, and cost involved in doing so. In some cases, that is a legitimate concern. NIH policy allows grantee organizations and investigators to charge requestors for the reasonable cost of production of biologic materials and for packaging and shipping, although income from these charges must be recorded as program income.
Other means exist to facilitate and minimize the costs of sharing publication-related research resources, including the deposition of research materials in existing public repositories, such as the American Type Culture Collection, and establishment of new repositories to facilitate sharing. NIH has established repositories to meet the needs of some specific research communities, such as the NIH AIDS Research and Reference Reagent Program (www.aidsreagent.org) and the Cancer Genome Anatomy Project (cgap.nci.nih.gov) And some researchers have established their own “cottage industries” for producing and distributing commonly requested materials.
A number of workshop participants advocated encouraging compliance with community standards by rewarding authors for sharing rather than by focusing on penalties for not sharing. They noted that whereas publication provides benefits for an author, the benefits of sharing publication-related resources are not always clear. For example, authors often fail to acknowledge those who have provided materials, data, or other information that helped in obtaining the findings they are publishing. Sharing should be recognized by citing a relevant publication of the donor of the material, and in the acknowledgement section of a paper. Another idea is to create a public database for acknowledgments. Such approaches would make it easier to recognize and reward those researchers who have been generous in sharing publication-related materials, data, software, or other information.
The existence of technical standards that allow data, materials, and other scientific information to be easily shared is critically important to the research communities that use those resources. Because they are so important, scientists organize circles to discuss standards in national and international workshops, at scientific-society meetings, and on electronic bulletin boards. For example, the Microarray Gene Expression Data Group calls itself a “grass-roots movement” to develop standards for DNA-array experiments and data representation so that investigators can compare and validate results from different arrays. Those kinds of efforts might be expanded to develop consensus on what data should be included in a publication and how data, materials, and information related to the publication should be shared, considering the needs of investigators in the particular field. Professional scientific societies, many of whom publish journals, also provide a natural forum for the discussion of community standards for sharing publication-related materials and data. Societies can help to identify repositories for specific types of data and materials that should be used by community members to facilitate sharing in different disciplines.
Crystallography: A Case Study
The development of standards for sharing publication-related data in crystallography provides a useful illustration of how one community of researchers has worked to establish norms that maximize the scientific value of information produced by individual investigators. It also shows how the research community, in concert with journals and funding organizations, continues to adjust standards and seek their enforcement as the field evolves.
In 1971, a group of crystallographers established the Protein Data Bank (PDB) as a worldwide archive for three-dimensional structure data on biologic macromolecules. At first, only a handful of structures were deposited in the archive each year. However, deposition of structures in
PDB began to rise dramatically in the 1980s because of technological advances in crystallography and changes in community views on data sharing. In 1989, the International Union of Crystallography (IUCr), which represents the crystallography research community, adopted a resolution calling on crystallographers to deposit atomic coordinates and related data in the appropriate structural database where a research article drawing conclusions from the data was submitted for publication. Because substantial time and effort are needed to solve a protein structure, the IUCr policy allowed authors to request a 1-year hold before public release of the data by the database so that authors could reap the intellectual benefits of their efforts.
By the early 1990s, as a result of the efforts of members of the crystallography community, many journals, NSF, and NIH had adopted the IUCr policy, including the 1-year hold allowance. The decision by HHMI to require deposition of coordinates by all its investigators was one of the critical steps in making the standards universal. Adoption of the standards by all key funding organizations and the most prestigious journals was pivotal in encouraging compliance.
In recent years, the time needed to determine the structure of most proteins has decreased dramatically, and standards for deposition and public release of structural data have continued to evolve. Interest in gaining access to macromolecular-structure data has grown as a result of the continuing development of new methods for analyzing the data and new experimental uses for them, including studies on protein folding, protein family organization, structure prediction, and drug design. In response to that interest, the PDB is evolving from a simple repository of data to one that provides mechanisms for researchers to understand biological function through investigation of sequence and molecular structure. As part of this evolution, the PDB is engaged in effort to achieve uniformity of format among all the records deposited in the archive since its inception.
As a result of the changes in the field, and with the vigorous support of scientists in the crystallography community, in the late 1990s some scientific journals and several special NIH research programs adopted an
immediate-release policy for macromolecular-structure data. In January 1999, NIH announced a new policy requiring its grant recipients to arrange for immediate release on publication of data deposited in the PDB. Community standards for sharing these data are still in flux. Although a number of journals now require immediate release of structure data, some still allow authors to request a 1-year hold. Others have adopted the latest IUCr recommendations, which urge authors to release their data immediately after the publication date but still give them the leeway to request a delay of up to 6 months. In addition, standards and policies still vary with regard to the kinds of data from structure studies, other than atomic coordinates, that should be deposited in PDB.
There are analogies between the evolution of community standards in the crystallography community and the current debate about sharing genome-sequence data on publication. When NIH and some journals first considered requiring immediate release of protein-structure data, various arguments were made against changing the standard. Opponents of the change argued that authors needed the protection conferred by a 1-year delay and that companies might defer publication without such protection. They also cited the need to protect the interests of graduate students and postdoctoral fellows. In addition, some argued that the new standard would fail because journals would not be able to enforce it. Despite those arguments, several journals did adopt the standard and are enforcing it. Community standards in the field will no doubt continue to evolve.
Community standards are not federal regulations; rather, they are self-imposed by the community and sometimes incorporated in journal policies. The responsibility for developing and updating community standards lies with all members of the community who participate in the publication process and have an interest in the progress of science— academic, government, and industrial scientists; publishers and editors of scientific journals; and institutions and organizations that conduct and fund scientific research.
In many fields of the life sciences, uniform standards for reporting data are still being developed. However, it is the committee’s view that once such standards have been established, journals should enforce them. Community standards, like the principles articulated in this report, are really only valuable to the extent that they are upheld by the scientific journals and honored by the community. The data generated by modern science may be increasingly diverse and complex and present novel challenges, but the power of the principles first established by Henry Oldenburg and the Philosophical Transactions of the Royal Society in 1665 remain undiminished: The rewards of publication counterbalance inclinations to secrecy. Oldenburg’s simple idea created an ethic of open disclosure of scientific results that has lasted for centuries and served to move science forward. We hope this report, which reaffirms that ethic, will be a useful contribution to the community’s discussions of standards for sharing publication-related data and materials.
With this philosophy in mind, the committee puts forward several recommendations for consideration and discussion by the community:
Recommendation 6. Scientific journals should clearly and prominently state (in their instructions for authors and on their Web sites) their policies for distribution of publication-related materials, data, and other information. Policies for sharing materials should include requirements for depositing materials in an appropriate repository. Policies for data sharing should include requirements for deposition of complex datasets in appropriate databases and for the sharing of software and algorithms integral to the findings being reported. The policies should also clearly state the consequences for authors who do not adhere to the policies and the procedure for registering complaints about noncompliance.
Recommendation 7. Sponsors of research should clearly and prominently state their policies for distribution of publication-related materials and data by their grant or contract recipients or employees.
Recommendation 8. If an author does not comply with a request for data or materials in a reasonable time period (60 days), and the requestor has contacted the author to determine if extenuating circumstances (travel, sabbatical, or other reasons) may have caused the delay, it is acceptable for the requestor to contact the journal in which the paper was published. If that course of action is not successful in due course (another 30 days), the requestor may reasonably contact the author’s university or other institution or the funder of the research in question for assistance. Those entities should have a policy and process in place for responding to such requests for assistance in obtaining publication-related data or materials.
Recommendation 9. Funding organizations should provide the recipients of research grants and contracts with the financial resources needed to support dissemination of publication-related data and materials.
Recommendation 10. Authors who have received data or materials from other investigators that have contributed to the work published should appropriately and publicly acknowledge such contributions.