National Academies Press: OpenBook

Fostering Integrity in Research (2017)

Chapter: 11 Findings and Recommendations

« Previous: 10 Education for the Responsible Conduct of Research
Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×

11

Findings and Recommendations

The committee’s findings and recommendations reported in this chapter are based on its examinations of changes in the research environment since the 1992 Responsible Science report and on the committee’s consensus on the means by which the U.S. research enterprise and its participants might best foster scientific integrity in the changing environment (NAS-NAE-IOM, 1992). Despite the intensification and acceleration of forces originally discussed in Responsible Science, and the emergence of some trends that were not apparent then, the core values of the responsible conduct of science have not changed and should not change. These core values include objectivity, honesty, openness, accountability, fairness, and effective stewardship. The committee has structured its recommendations around these values and an understanding that research is conducted as part of a larger social enterprise. The resources produced by the research enterprise—including knowledge and highly trained people—are intended to benefit the public. Scientists are provided with opportunities and freedom to pursue new knowledge and train future scientists with the implicit understanding that they are responsible for the conduct of their research and the reliability of the knowledge they produce and that they must conduct their research responsibly as a duty to the public.

UNDERSTANDING THE ISSUES

Changing Environment

A number of changes in the research environment that were identified in the early 1990s as problematic for maintaining principles of research integrity and good scientific practices have generally continued along their long-term trend

Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×

lines. These include growth in the size and scope of the research enterprise, the increasing need for and complexity of collaboration, the expansion of regulatory requirements, and an increased focus on industry-sponsored research.

Several important new trends that were not examined in Responsible Science have also emerged, including the pervasive and growing importance of information technology in research, the globalization of research, and the increasing relevance of knowledge generated in certain fields to policy issues and political debates. These changes—the growing importance of information technology in particular—have led to important shifts in the institutions that support and underlie the research enterprise, such as science, engineering, technology, and medical publishing. The understanding of how colleagues, incentives, and environments influence ethical decision making has also advanced significantly. New challenges must be straightforwardly addressed to support researchers, research institutions, journals, and sponsors in their efforts to foster integrity, prevent and discourage research misconduct and detrimental research practices, and respond to these problems when they occur.

Updating Concepts: The Role of Detrimental Research Practices

Much of the discussion, thinking, and actions aimed at fostering research integrity has revolved around the actions of miscreant individuals in committing acts of research misconduct and its components—fabrication, falsification, and plagiarism. Actions that Responsible Science characterized as questionable research practices have received less attention. The accumulation of knowledge has brought the critical need to address these elements to the fore. Actions such as failing to retain or share data and code supporting published work in accordance with disciplinary standards, practices such as honorary or ghost authorship, and using inappropriate statistical or other methods of measurement and data presentation to enhance the significance of research findings are clearly detrimental to the research process and may impose comparable or even greater costs on the research enterprise than those arising from research misconduct. The committee believes that identifying these actions as detrimental research practices (DRPs) will be helpful in focusing attention and developing approaches to discourage and minimize them.

At the same time, based on better insight and understanding of the importance of environmental influences on individual choices, the environments in which research is performed need to be thoughtfully assessed and shaped. In addition to DRPs committed by individual researchers, organizations such as research institutions, research sponsors, and journals may also take actions that constitute detrimental research practices, often by failing to acknowledge or act upon implicit or explicit incentives and reward systems that can undermine the integrity of the research enterprise.

Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×

Incidence and Costs

The incidence of discovered research misconduct is tracked by official statistics, survey results, and analysis of retractions, and all of these indicators have shown increases in recent years. However, it is difficult to estimate precisely the incidence of misconduct in relation to an established baseline and to determine trends. It is possible to say that while research misconduct is unusual, it is not rare. High-profile cases continue to appear regularly from around the world at the same time that the overall size of the research enterprise has vastly expanded. A variety of DRPs appear to be unfortunately fairly common, at least in the fields and disciplines that have been studied. Examining specific cases of misconduct shows that tolerance for DRPs enables misconduct and leads to delays in uncovering it.

Both research misconduct and DRPs impose significant costs on the research enterprise, including careers that are destroyed or sidetracked, the financial costs to taxpayers of fraudulent or otherwise irreproducible research and work done to extend it, reputational costs to institutions, and the costs of investigations. Particular cases of misconduct and DRPs have also negatively affected society at large, such as a purported finding of a causal link between a widely used childhood vaccine and autism that has played a role in discouraging vaccinations. Such cases cause direct harm and also damage societal trust in the research endeavor. Beyond questions of needless human suffering, the total scale of monetary costs from research misconduct and detrimental research practices may run from several hundred million dollars up to multiple billions of dollars per year in the United States alone.1

Some of the actions and approaches needed to foster integrity recommended below do not have major costs associated with them. Others do, whether at the lab, institutional, or disciplinary level, and these costs are difficult to estimate. The research enterprise and sponsors may need to confront the need to spend more per research output, with the end result being fewer or slower research outputs. But those outputs—and the research cultures and environments in which they arise—will be much more robust, especially in those disciplines that have seen major issues of lack of robustness and trust.

Understanding the Causes

Why people engage in criminal or other deviant behavior and the conditions that encourage or discourage such behavior are issues of perennial interest in the behavioral and social sciences. Recent work has contributed useful insights that

___________________

1Chapter 5 contains a detailed discussion of the current state of knowledge concerning the various costs and consequences of research misconduct and DRPs and how their scale and scope might be estimated.

Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×

are relevant to understanding why and under what conditions researchers commit misconduct and engage in DRPs.

Some past assumptions and assertions have held that the character of scientists as searchers for truth and strong traditions of mentorship would limit those who would commit research misconduct to a few “bad apples.” However, evidence from recent years makes it clear that scientists are not immune to the environmental forces that contribute to deviant behavior in all professions, nor are they exempt from a variety of cognitive biases that are a normal part of the human condition (Mazar and Ariely, 2015). The environments in which researchers are educated, socialized, and perform their work require significant attention.

Current patterns of U.S. research funding and organization have been identified by some leading scientists as creating hypercompetitive research environments that are damaging the long-term health of research in some of the largest fields and disciplines (Alberts et al., 2014). These hypercompetitive environments contain characteristics that behavioral and social sciences research suggests facilitate and encourage detrimental and deviant behavior. While addressing larger structural issues in U.S. research funding and organization is beyond the scope of this study, more research on the causes of research misconduct and detrimental research practices is needed to develop better strategies for prevention, as well as specific steps to assess the integrity of research environments and to act on the findings of such assessments to implement good practices and foster sound research environments.

The Need for More Robust Approaches

Research misconduct and DRPs need to be addressed in several ways. The primary means include (1) efforts to prevent them through responsible conduct of research (RCR) education and environmental assessment and improvement; (2) efforts to uncover research misconduct, investigate, and take corrective actions through the efforts of researchers, institutions, federal and private research sponsors, and journals; and (3) efforts to discourage and eliminate DRPs through the implementation of standards and best practices, such as effective mentoring, requirements for data and code sharing, and implementation of greater transparency in reporting results.

The committee examined the status of efforts in all three areas and concluded that improvements are needed across the board. The findings and recommendations that follow provide a roadmap for the actions that need to be taken.

FINDINGS AND RECOMMENDATIONS

The committee reaffirms the central recommendation from Responsible Science that formally places the primary responsibility for acting to define and strengthen basic principles and practices for the responsible conduct of research

Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×

on individual scientists and research institutions. At the same time, the committee based its recommendations on its understanding that the integrity of research depends on creating and maintaining a system and environment of research in which institutional arrangements, practices, policies, and incentive structures support responsible conduct. Fostering research integrity is an obligation shared not only by individual researchers but also by leaders and those involved with all organizations sponsoring, conducting, or disseminating research, including corporate and government research organizations.

The committee also endorses the definition of research misconduct recommended in Responsible Science while recommending refinements in its use. In particular, through its examination of current practices advancing research integrity and responses to deviations, the committee became aware of variations in federal approaches to the evaluation of plagiarism that need to be harmonized. The following findings and recommendations are intended to serve as a framework for actions that will improve knowledge of research misconduct, detrimental research practices, and contributing factors; strengthen approaches to addressing them; and ultimately lead to a significant reduction or even elimination of these behaviors and the risks that they pose to the research enterprise.

FINDING A: Developing and implementing improved approaches to fostering research integrity and meeting the current threats to integrity posed by research misconduct and detrimental research practices are urgent tasks. These improved approaches should reflect an understanding of the complex interactions among the many components of the research enterprise and its multiple stakeholders.

The research enterprise is a large, diverse, and complex system, and society invests considerable resources in it. Research misconduct and detrimental research practices constitute long-term threats to the research enterprise’s ability to deliver the benefits expected by society. While the values and ideals of science should remain unchanged, the experience of the past several decades drives home the lesson that significant changes in the practices and institutional arrangements of the research enterprise are necessary to strengthen the self-correcting mechanisms of science. Developing and implementing these practices and approaches will require us to better understand how research environments and the incentives created by structural relationships among the institutions of science can support or undermine the efforts of individual researchers to behave responsibly.

The first set of recommendations targets the broad, long-term need for sustained, cooperative efforts by the major components of the research enterprise: individual researchers; research institutions; research sponsors; science, engineering, technology, and medical journal and book publishers; and scientific societies. They also cover the shorter-term need to improve research environments by assessing and then addressing identified weaknesses.

Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×

RECOMMENDATION ONE: In order to better align the realities of research with its values and ideals, all stakeholders in the research enterprise—researchers, research institutions, research sponsors, journals, and societies—should significantly improve and update their practices and policies to respond to the threats to research integrity identified in this report.

Lack of attention to or tolerance of detrimental research practices by stakeholders makes it more difficult to expose misconduct, wastes human and financial resources, impairs the overall quality of research, and diminishes public trust in science. In addition, weaknesses in the system for identifying, investigating, and sanctioning research misconduct—most notably unevenness in the policies and capabilities of research institutions and journals—create barriers to uncovering misconduct and taking corrective action. Changes in the funding and organization of research are affecting institutional and laboratory environments in ways that can undermine incentives to behave responsibly. For example, Alberts et al. (2014) noted,

As competition for jobs and promotions increases, the inflated value given to publishing in a small number of so-called ‘high impact’ journals has put pressure on authors to rush into print, cut corners, exaggerate their findings, and overstate the significance of their work. Such publication practices, abetted by the hypercompetitive grant system and job market, are changing the atmosphere in many laboratories in disturbing ways.

Similarly, in industrial R&D organizations, pressures associated with regulatory approvals or commercial release may create disincentives for full data transparency or biases that promote conclusions of safety and efficacy. Finally, changes in the research environment such as technological advances and globalization are making it more difficult and complex for all stakeholders in the enterprise to update and ensure adherence to best practices.

The checklists presented in Chapter 9 should form the basis of strategies to refine and implement best practices by researchers, research institutions, research sponsors, journals, and societies.

RECOMMENDATION TWO: Since research institutions play a central role in fostering research integrity and addressing current threats, they should maintain the highest standards for research conduct, going beyond simple compliance with federal regulations in undertaking research misconduct investigations and in other areas.

In order to maintain the highest standards for research conduct, research institutions need to exercise vigilance in several distinct areas:

Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×
  • Creating and sustaining a research culture that fosters integrity and encourages adherence to best practices through effective education and training and other mechanisms;
  • Monitoring the integrity of research environments through internal assessments and multi-institution benchmarking exercises, and acting on the results;
  • Ensuring that institutional policies and processes to investigate and address allegations of research misconduct are robust and generate just and timely outcomes; and
  • Ensuring that senior institutional leaders such as the president, other senior executives, administrators, and faculty leaders are guiding and actively engaged in these efforts.

Because they are the facilitators and stewards of research activity, as well as the employers and educators of researchers, research institutions (including academic/nonprofit, industrial, and governmental organizations) will play a central role in determining how well the research enterprise as a whole fosters research integrity and addresses current threats. Institutions can undertake this important work collaboratively—through related organizations and associations—as well as in partnership with other stakeholders, such as federal and private research sponsors and science, engineering, technology, and medical journal and book publishers.

The key responsibilities for research institutions fall into four areas. The first is creating and sustaining a research culture that fosters integrity and encourages adherence to best practices. The leadership of universities and other research institutions, including presidents, other senior executives and administrators, and faculty leaders, has a central role to play in building and sustaining environments that promote responsible research conduct. This is not only important for fostering integrity in the research process but also will encourage science of the highest quality. This includes maintaining education and training efforts that support a culture of integrity, consistent with the current state of knowledge. Recommendation Ten describes in more detail how responsible conduct of research education and training programs should be developed and implemented.

A second task is monitoring the integrity of research environments. Such monitoring is critical to further advance understanding of how institutional structure, context, and incentives interact to buttress or detract from research integrity. The 2002 report Integrity in Scientific Research: Creating an Environment That Promotes Responsible Conduct explains that research organizations “engage in activities that help establish an internal climate and organizational culture that are either supportive of or ambivalent toward the responsible conduct of research” (IOM-NRC, 2002). Institutional assessment and benchmarking exercises can be important tools helping institutional leadership to fulfill this role; tools now exist that can be used to perform these assessments (Martinson et al., 2013). Research

Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×

institutions should regularly assess their research integrity climate and share information among departments and peer institutions, using approaches such as those used in the Project on Scholarly Integrity undertaken by the Council of Graduate Schools (CGS, 2012). Related organizations and associations (e.g., the Association of American Universities, Association of Public and Land-Grant Universities, Council of Graduate Schools) should contribute to this effort. Part of this activity should be devoted to further advance understanding of how institutional structures, context, and incentives interact to buttress or detract from research integrity.

Research institutions also have an obligation to implement improvements to their research environments based on the knowledge gained in these assessments. Recent efforts involving assessment of research integrity climates and sharing of information among departments and peer institutions have yielded important insights. Where institution-wide assessments identify units with particularly strong integrity environments, they should be examined and their practices should be disseminated and emulated.

The third institutional responsibility is ensuring that research institutions sustain the capacity needed to effectively investigate and address allegations of research misconduct. No institution can be expected to prevent all lapses in research integrity, but all should ensure that when problems in the conduct of research are alleged or identified, there is a prompt, effective, and documented response to the allegation. Currently, there is limited evidence to assess how institutions are performing, including specific cases that have been reported in the media, the results of surveys of institutional officials undertaken by the Office of Research Integrity, and the presentations made to the committee by federal agency officials. This evidence is discussed in Chapter 7 and indicates that, while some institutions are performing at an outstanding level in this area, others are not. This report describes several highly publicized cases from recent years in which institutional responses to alleged research misconduct or to credible questions about reported results were deficient. Given the critical role that institutions play in fostering research integrity, substantial damage can be done in cases where they fall short. The best practices for research institutions described in Chapter 9 comprise a starting point for institutional efforts.

The committee appreciates that sustaining appropriate institutional capacity can be challenging. Because of the relative infrequency of inquiries and investigations responding to allegations of research misconduct, particularly at smaller institutions, it may be difficult to maintain institutional memory in some areas. The specific examples examined by the committee also show that it is often difficult for organizations to work impartially when powerful individuals have been accused of misconduct or when the institution’s own financial or reputational interests are involved. The Research Integrity Advisory Board proposed below can serve as a resource as institutions seek to maintain the highest standards in how they address lapses in integrity.

Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×

Institutions are organized differently, so they should have the flexibility to develop and implement their policies in ways that make sense for them. In some institutions, formal responsibility for responding to research misconduct allegations lies with the graduate dean or vice president for research. In others, a designated research integrity officer or compliance officer who reports to the vice president for research might have this responsibility. In all cases, this official—whose responsibilities might also include oversight of education, training, and assessment as discussed below—should have direct access to the president and other institutional leaders.

One insight from specific cases is that the existence of multiple channels for raising concerns even prior to making allegations can be very helpful. For example, some institutions have had success with an ombuds system independent of those formally responsible for responding to allegations.

A fourth responsibility is ensuring that senior institutional leaders are guiding and actively engaged in the preceding three tasks. For example, when institutional leaders are accessible and knowledgeable about institutional capacity to address allegations of misconduct, they are in a position to be helpful in keeping people and processes on track when specific allegations arise. Should later events call into question the rigor of an institutional response to allegations of misconduct in research, top institutional leadership should be expected, as a matter of course, to examine the shortcomings of the process and share lessons learned with the larger community of scholars as a contribution to improvement of research integrity across the community. Institutional leaders must be accountable for the quality of responses to questions about research integrity.

Institutional leaders are also in the best position to implement changes based on the results of research integrity climate assessments, and they can communicate directly and regularly institutional standards and expectations as well as the importance of the quality of research conducted under institutional auspices.

RECOMMENDATION THREE: Research institutions and federal agencies should work to ensure that good-faith whistleblowers are protected and that their concerns are assessed and addressed in a fair, thorough, and timely manner.

Those who raise concerns about the integrity of research, often referred to as whistleblowers, can play a critical role in supporting best practices in research and in uncovering research misconduct, as described in Chapter 7. Individuals closest to the research are in the best position to identify and correct problems as early as possible and can be expected to play this role for the foreseeable future. Inadequate responses to expressed concerns have constituted a critical point of failure in many cases of misconduct where investigations were delayed or sidetracked. Those who raise concerns are typically the most vulnerable participants in the system, holding little institutional power or status. Research institutions

Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×

and federal agencies should understand the implicit bias that exists against those who in good faith raise fact-based concerns about the integrity of research. The report discusses several alternative approaches to strengthening whistleblower protections that should be considered and implemented.

RECOMMENDATION FOUR: To provide a continuing organizational focus for fostering research integrity that cuts across disciplines and sectors, a Research Integrity Advisory Board (RIAB) should be established as an independent nonprofit organization. The RIAB will work with all stakeholders in the research enterprise—researchers, research institutions, research sponsors and regulators, journals, and scientific societies—to share expertise and approaches for addressing and minimizing research misconduct and detrimental research practices. The RIAB will also foster research integrity by stimulating efforts to assess research environments and to improve practices and standards.

While various groups, institutions, and individuals are doing valuable work to foster and promote research integrity in the United States, no permanent organizational focus for efforts to foster research integrity at a national level currently exists. The RIAB would provide a continuing organizational focus for fostering research integrity that cuts across disciplines and sectors. It should be established independent of government.

The RIAB would perform several functions, including:

  • Working with public and private research sponsors to develop improved practices and approaches to addressing research misconduct and fostering integrity. For example, the RIAB could serve as a forum for the discussion of issues where community consensus currently does not exist (such as what the appropriate penalties for research misconduct should be) or where current disparate approaches should be harmonized (such as the implementation of the federal research misconduct policy in areas such as plagiarism).
  • Working with science, engineering, technology, and medical journal and book publishers to develop improved practices and approaches. The biannual Journal Summit organized by the National Academy of Sciences generates a number of useful ideas that could be explored further by the RIAB.
  • Identifying important topics and questions related to research misconduct and research integrity, including pathways to improve research environments and RCR education, where research could produce valuable insights, and perhaps serve as a mechanism for commissioning such research.
Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×
  • Working with research institutions, institutional officials, and groups such as the new Association of Research Integrity Officers to identify and develop resources aimed at improving institutional capability to respond to research misconduct allegations and sustain environments that encourage responsible conduct. These resources could include just-in-time training materials, referrals to experts with relevant scientific and/or legal knowledge who could be consulted on specific cases, and help with organizing external review of investigation committee task statements and reports.

The RIAB will have no direct role in investigations, regulation, or accreditation. Rather, it will serve as a neutral resource based in the research enterprise that helps the research enterprise foster integrity in a changing environment. It will work best as an independent, nonprofit organization with a small permanent staff of three or four people, supplemented by fellows and consultants. An annual budget of about $3 million would be adequate. The RIAB would be governed by its members, with a rotating executive committee selected to develop strategy and oversee operations. Funding would come in the form of regular contributions from members such as the major public and private sponsors of research, universities and other research institutions, industrial members, scientific societies, and science, engineering, technology, and medical journal and book publishers. Further discussion of the RIAB and consideration of alternatives is contained in Chapter 8.

FINDING B: Ensuring greater openness and accountability in science is essential to fostering research integrity and improving research quality. Establishing and agreeing on new standards and building the infrastructure needed to implement those standards will require collaborative, focused efforts on the part of the research enterprise and its stakeholders.

The values of openness and accountability make transparency and striving for reproducibility of scientific findings central to the responsible conduct and dissemination of research. As technological advances and other shifts continue to transform scientific work, responsive changes to standards and practices are needed. Examples from recent years show that some cases of fabrication and falsification have been uncovered relatively quickly by researchers seeking to replicate the work when data were available. In other examples, failure to require that researchers provide access to data and code has been associated with delays in uncovering lapses in integrity. Clarifying and updating authorship standards, implementing data- and code-sharing requirements, securing adherence to existing requirements, and heightened attention to appropriate use of sound statistical methods will help to foster integrity by facilitating the processes by which research results are confirmed or refuted.

Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×

Facilitating broader access to data and code can also help to accelerate the advance of knowledge. In recent years, the problem of irreproducibility of research results has attracted increasing attention and concern. It is important to note that some research results will not be reproducible even where there are no mistakes or lapses in integrity. Researchers can ensure openness, honesty, accountability, and transparency, but cannot completely ensure the reproducibility of their work. The baseline responsibility of researchers, institutions, journals, and sponsors is to ensure that published research provides enough information about methods and tools that other researchers attempting to replicate the work could succeed or, if not, could provide compelling evidence that the work could not be reproduced.

All of the actions outlined in this set of recommendations are aimed at ensuring higher levels of openness and accountability, which are essential to strengthening the operation of the scientific process and for ameliorating many of the weaknesses that are apparent in current systems and practices. As pointed out in various parts of the report, several of these areas of weakness have seen positive movement in recent years, but efforts on the part of one or more research enterprise stakeholders could bring practices into better alignment with scientific ideals.

RECOMMENDATION FIVE: Societies and journals should develop clear disciplinary authorship standards. Standards should be based on the principle that those who have made a significant intellectual contribution are authors. Significant intellectual contributions can be made in the design or conceptualization of a study, the conduct of research, the analysis or interpretation of data, or the drafting or revising of a manuscript for intellectual content. Those who engage in these activities should be designated as authors of the reported work, and all authors should approve the final manuscript. In addition to specifying all authors, standards should (1) provide for the identification of one or more authors who assume responsibility for the entire work, (2) require disclosure of all author roles and contributions, and (3) specify that gift or honorary authorship, coercive authorship, ghost authorship, and omitting authors who have met the articulated standards are always unacceptable. Societies and journals should work expeditiously to develop such standards in disciplines that do not already have them.

Authorship practices are a fundamental component of the research enterprise’s operation, and observance of good practices is a key factor in ensuring research integrity. Authorship crucially designates who bears responsibility for the work. By communicating the assumptions made and methods used in conducting experiments, researchers allow others to replicate, extend, and where necessary,

Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×

correct their work. As a result, science is typically a cumulative exercise that produces a growing body of reliable knowledge. Clarifying authorship responsibility is also critical in case of error or allegations of misconduct.

Detrimental practices such as coercive authorship, gift authorship, and unacknowledged ghost authorship impair the usefulness and reliability of authorship as the central institution for assigning credit for reported work, fixing responsibility for that work’s quality and integrity, and communicating critical information that allows other researchers to replicate, extend, and where necessary, correct that work.

Although some disciplines have developed clear authorship guidelines, authorship practices and conventions are largely left to individual institutions and journals. Greater clarity at the disciplinary level about the significant intellectual contributions that merit authorship, the roles that do not merit authorship, the significance of author order, and the responsibilities of a primary or corresponding author would be very helpful in facilitating appropriate decisions and practices in labs and collaborations. Universal condemnation (i.e., by all disciplines) of gift or honorary authorship, coercive authorship, and ghost authorship would also contribute to changing the culture of research environments where these practices are still accepted. Universal adoption of the requirement that all authorship roles be disclosed, as is the case for a growing number of journals, and commitment to the principle that all contributors who merit authorship should be listed would also be positive steps.

The committee favors an approach that authorship should be established through a significant intellectual contribution to the work in at least one area, such as planning, performing, analyzing, or writing. All authors should have the opportunity to approve the final manuscript.

The committee recognizes that flexibility in the development and implementation of authorship guidelines is needed due to significant differences between disciplines.

RECOMMENDATION SIX: Through their policies and through the development of supporting infrastructure, research sponsors and science, engineering, technology, and medical journal and book publishers should ensure that information sufficient for a person knowledgeable about the field and its techniques to reproduce reported results is made available at the time of publication or as soon as possible after that.

The information needed to verify and build upon published results can vary by field and discipline. Examples of such information include the specification of agents, materials and reagents, digital data, and software code and scripts used for analysis and production of results. With new advances in technology, such as the wide availability of image manipulation software as well as the pervasive ap-

Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×

plication of statistical and computational methods, continued adaptation and development of reporting standards and scientific practices are essential. In almost every area of scientific research, researchers have been quick to adopt advances in technology to accelerate the progress of research activities. In some specific cases discussed in the report, the absence of standards and lack of adherence to best practices have enabled fabricated work to go undetected or uncorrected for long periods of time. As information technology advances continue to transform scientific methods, the development and wide implementation of best dissemination practices needs to keep pace.

A research process that uses computational tools and digital data introduces myriad new potential sources of error: Were the methods described in the paper transcribed correctly into computer code? What were the parameter settings, input data, and function invocation sequences? How were the raw data filtered and prepared for analysis? Can the figures and tables reported in the published article be replicated by the associated data and code? Access to the data and code that produced the results is paramount, both for replication and validation purposes and for reconciling any differences in independent implementations.

Computation has facilitated vastly greater complexity in research. For example, the number of computational steps in deriving a scientific finding can be enormous, and these steps may not be completely captured in the traditional methods section of a scientific publication. All details of the specific computations that generated results, encapsulated in the code and data, must be made available to others for the findings to be reproducible.

The experimental and computational protocols and detailed methodology relevant to reproducibility should be made available by researchers. These include digital objects such as raw data—in fields where raw data are digital—and software, including source codes, scripts, and code books, sufficient to enable replication of computational research findings by one skilled in the discipline. These should be made openly available and reusable at the time of publication and persistently linked to or embedded in research articles.

Responding to recent attention to the problem of reproducibility, the research enterprise is beginning to take important steps. Some journals have begun to implement requirements that authors make the data and computer code required to regenerate the published results available upon request (Science, 2011). Many universities and funding agencies have created online repositories to support the dissemination of digital data, and best practices promulgating the routine sharing of digital scholarly objects that support verification of published findings must continue across computational research. Current digital data practices vary significantly by field and discipline, and making certain types of data broadly accessible presents special challenges. For example, the need to ensure privacy and anonymization of personal and clinical data that are to be shared requires technical ingenuity and imposes real costs. The successful development and implementation of new standards and requirements will depend upon sufficient

Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×

investments in necessary human and physical infrastructure, as described below in Recommendation Seven. Some fields and disciplines, such as astronomy, provide positive examples in which large amounts of digital data are being made widely available, while the infrastructure needs of other fields remain significant (NAS-NAE-IOM, 2009a).

The Transparency and Openness Promotion Guidelines developed by the Center for Open Science constitute an important contribution that can be studied and adapted by various fields. As is the case with other tasks identified in this report, building a more accountable and transparent research enterprise is a long-term, multistakeholder challenge.

There are notable exceptions to the presumption that all data and code should be shared, such as human subject privacy protections. When these safeguards are not at issue, the scientific community has an opportunity to act on the Office of Science and Technology Policy’s memorandum of February 2013 aimed at expanding access to federally funded research (Holdren, 2013).

Previous reports have made similar recommendations about access to data and code in broad and specific contexts (Fienberg et al., 1985; IOM, 2015; NAS-NAE-IOM, 2009a; NRC, 2003). While many of the steps necessary for implementing the recommendations contained in these previous reports remain to be taken, there is reason to hope that the importance of access to data and code is becoming sufficiently well recognized to enable significant progress in the next few years.

Massive national investments are being made in digital data collection, which is typically not hypothesis driven but is undertaken because it is possible. This opens potential new research questions, but it also heightens the imperative for data availability to enable the production of reliable scientific findings.

Finally, with automatic plagiarism detection software increasingly being used on published articles and research, it has become apparent that some misconduct can be caught prior to publication. The committee encourages publishers to coordinate knowledge and efforts to adopt new technologies as they become available to detect and reduce plagiarism prior to publication.

RECOMMENDATION SEVEN: Federal funding agencies and other research sponsors should allocate sufficient funds to enable the long-term storage, archiving, and access of datasets and code necessary for the replication of published findings.

Preparing data and code for release can be expensive and time-consuming. Researchers are currently rewarded for manuscript publication, but the professional rewards for preparing data and code for publication are minimal. The resources to support the endeavor are also often limited, and the feasibility and time required depend very much on the type of research data and how they were collected. This has the effect of penalizing those who spend the necessary time and resources to prepare data and code for publication. One way to address this

Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×

problem is for the community to adopt new practices, and recent changes in federal policy provide such an opportunity (Holdren, 2013).

In addition, journals should update their publication requirements to include access to data and codes needed to replicate results. These data and codes can be deposited at any repository that can reasonably guarantee a persistent URL, which should be provided in the text of the published paper. Even when complex computational architectures have been used that make independent execution of the software difficult, sharing the code openly allows others to inspect, assess, and perhaps adapt the methods.

The barriers to data sharing at the scale recommended here are significant, and this recommendation will take some time to implement. Setting priorities and achieving the necessary funding levels will require time. Efforts to make data available and encourage reproducibility should catalyze the development of new data tools that ultimately reduce costs over time. The key is to ensure that data are in a long-term repository with metadata that allow third parties to reuse them.

To facilitate the reuse of scientific code and data, these objects should be shared in such a way as to maximize access while respecting scientific norms such as attribution (Stodden, 2009). Permissive open licensing, such as the MIT License or Modified BSD License for software or the Creative Commons Public Domain certification for data, should be used.

RECOMMENDATION EIGHT: To avoid unproductive duplication of research and to permit effective judgments on the statistical significance of findings, researchers should routinely disclose all statistical tests carried out, including negative findings. Research sponsors, research institutions, and journals should support and encourage this level of transparency.

Available evidence indicates that scientific publications are biased against presenting negative results and that the publication of negative results is on the decline (Fanelli, 2010, 2012). In extreme cases, where nearly identical experiments are run a number of times with one positive result being reported and multiple negative results discarded, the failure to report negative results constitutes a detrimental research practice (Couzin-Frankel, 2013). Yet, in recent years, several analyses and opinion pieces have pointed to the value of publishing negative results. For example, dissemination of negative results has prompted a questioning of established paradigms, leading ultimately to groundbreaking new discoveries (Anderson et al., 2013). Publication of negative results can also lead to the uncovering of flaws and the subsequent development of improved research methods. For example, a number of papers reported the negative results of work seeking to replicate research on vaccines and autism discussed in Chapter 5 and Appendix D.

Changing the culture of research and publication so that negative results

Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×

reporting is expected and replication efforts are valued will require a persistent effort on the part of disciplines, sponsors, and journals. The attention received by several recent replication efforts, one of which involved publication in Science, is encouraging (OSC, 2015). As routine reporting of negative results and statistical tests becomes the standard for all fields, research spending will become more productive and more knowledge will be generated per dollar of research investment.

FINDING C: Improved strategies for fostering research integrity and for addressing threats to integrity posed by research misconduct and detrimental research practices need to be based on knowledge and evidence that does not currently exist. Investments are needed in research that improves understanding of key issues such as the relationship between structural conditions in science and the tendency for individuals to practice research according to the values and norms of integrity or to deviate from those values and norms. Improving knowledge in this area is essential to the long-term health of the research enterprise itself.

Upholding the values of objectivity, honesty, and openness in the contemporary context requires that research institutions, the federal government, science, engineering, technology, and medical journal and book publishers, and scientific societies should, individually and in collaboration, examine these systemic conditions and their impacts on incentives. This research needs to bring to bear the best of what is known about influences on human decision making from a range of social science fields to guide actions to improve research climates so that they reflect and reinforce the core values of science. This research should complement the research environment assessment activities at institutions discussed above in Recommendation Two. These investments should be directed to helping all participants in the research enterprise, from the local to the international, act upon the findings to reinforce the values and norms underlying integrity. National and potentially international benchmarks are needed so that assessments can be understood in light of disciplinary differences, much as national medical cost benchmarking is shedding light on where some unnecessary procedures are being undertaken in some regions, with corrective actions being taken.

RECOMMENDATION NINE: Government agencies and private foundations that support science, engineering, and medical research in the United States should fund research to quantify, and develop responses to, conditions in the research environment that may be linked to research misconduct and detrimental research practices. These research sponsors should use the data accumulated to monitor and modify existing policies and regulations.

Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×

Material presented in Chapters 5 and 6 illustrates that understanding of the causes and incidence of research misconduct and detrimental research practices has increased but that critical knowledge gaps remain. For example, official statistics on findings of research misconduct may represent a lower bound on incidence, with survey data pointing to a significantly higher incidence of misconduct, but no reliable estimate of incidence or trends exists. Also, detrimental research practices are more widespread and may ultimately be more damaging to the research enterprise than research misconduct, which points to the need to address challenges to research integrity more broadly. In addition, trends in some indicators—such as declining success rates for grant applications, and an increasing ratio of PhD production to available faculty positions—raise the possibility that both local organizational environments and the broader structural arrangements of research are moving in directions that might threaten research integrity. Additional theoretically grounded research with subsequent testing in practice is warranted to more completely inform efforts to improve research environments and incentive structures.

Data generated through regular institutional research integrity assessments (discussed under Recommendation Two), research on the factors contributing to research misconduct and how to address them, and information on effective educational approaches could provide valuable input to the policies and practices of research sponsors and federal agencies charged with overseeing institutional research misconduct investigations. For example, the RCR education policies of the National Science Foundation and National Institutes of Health should be modified over time as knowledge improves. Also, a better understanding of the linkages between hypercompetitive research environments and misconduct and detrimental practices could help to support and inform changes. Where elements are identified that support particularly robust integrity environments, they should be broadly shared.

Evidence could inform better policy in a number of other areas. For example, although the Office of Research Integrity and the National Science Foundation’s Office of the Inspector General both follow the 2000 federal research misconduct policy, there are a number of clear differences between the agencies in how they implement the policy, as discussed in Chapter 7. These include differences in how those found to have committed research misconduct are publicly identified, the scope of action available to the agencies outside formal investigations, and regulatory relationships between the agencies and research institutions. Greater understanding of the impacts of such differences could be helpful in determining whether and how to harmonize approaches across the federal government. Research bearing on other issues, such as what sort of corrective actions are appropriate for those who have committed research misconduct, how offenders should be rehabilitated, and the possible positive impacts on research integrity associated with data and code access mandates, would also be very useful to policy makers.

Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×

Finally, current practices in research funding and organization may contribute to a higher incidence of research misconduct and detrimental research practices, at least in some disciplines and institutions. Addressing the underlying structural problems of funding and organization would require significant policy changes that go beyond the scope of this study. However, a better understanding of the linkages between hypercompetitive research environments and misconduct and detrimental practices could help to support and inform such changes.

RECOMMENDATION TEN: Researchers, research sponsors, and research institutions should continue to develop and assess more effective education and other programs that support the integrity of research. These improved programs should be widely adopted across disciplines and across national borders.

Formal responsible conduct of research education and training efforts can play an important role in fostering integrity and strengthening research environments. Evidence developed to date indicates that much remains to be learned about the approaches that are most effective. RCR education should be looked to as a key element in strategies to promote integrity, but perhaps not as a primary means of addressing research misconduct and detrimental research practices in the short term. Evidence-based assessment and improvement of RCR education programs is needed, with the focus expanded to include the social and institutional environment for research. RCR education should engage not only junior scientists but also senior research scientists and industrial researchers.

FINDING D: Working to ensure research integrity at the global level is essential to strengthening science both in the United States and internationally.

The research enterprise is increasingly global in nature, and an international focus is imperative when seeking improvements in systems for safeguarding research integrity. As illustrated in recent media reports, all countries that perform a significant amount of research have experienced challenges in the area of research integrity, including high-profile cases of misconduct and, often, deficiencies in institutional and governmental responses. The World Conferences on Research Integrity have helped build a global community of experts. These conferences and other activities, such as Organisation for Economic Co-operation and Development workshops held to develop standard contract language for use in international collaborations, have made clear the value of cross-border exchange and learning. At the same time, varying policy contexts related to research support and institutional oversight may make thorough global harmonization of policies and practices difficult or impossible, at least in the immediate future.

Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×

RECOMMENDATION ELEVEN: Researchers, research institutions, and research sponsors that participate in and support international collaborations should leverage these partnerships to foster research integrity through mutual learning and sharing of best practices, including collaborative international research on research integrity.

While the committee has put its primary focus on how to better foster research integrity in the United States, the study was informed by a changing global context. The problems of research misconduct and detrimental research practices, and their resulting negative impacts, are global. Lack of training or ineffective training of students abroad has an impact in the United States when students or faculty move to U.S. institutions or collaborate with U.S.-based researchers. Many of the most visible and publicized cases of research misconduct have involved international coauthorship. Several recent research misconduct investigations undertaken by research institutions outside the United States have been exemplary and illustrate that U.S. researchers, institutions, and sponsors can learn a great deal from international colleagues (Ishii et al., 2014; Levelt et al., 2012).

In addition, disciplinary differences in research practices often vary to a greater degree than do differences between countries. Disciplines are for the most part global in scope, and disciplinary efforts to examine and upgrade practices will tend to be global as well. The Levelt report of the investigation of Diederik Stapel’s research misconduct by the three institutions that employed him over a period of several decades identified a number of weaknesses in practices that have been widely tolerated in social psychology, performing a long-term service for all researchers in this field.

Also, researchers and institutions have many opportunities to learn from each other. Just as U.S. institutions can learn from other U.S. institutions that are more effective at education or at addressing allegations of misconduct, mutual learning between U.S. and overseas institutions can encourage improvement and diffusion of best practices.

Given that research misconduct, detrimental research practices, and the need to foster research integrity are challenges facing all countries that fund and perform research, the global research enterprise will benefit from the knowledge gained from the research agenda outlined under Finding C, above. Expanding this research agenda to a global scale would be beneficial to all. For example, exploration of cross-national or cross-cultural differences in attitudes and norms related to research behaviors (e.g., plagiarism) would be useful input to the development of targeted educational interventions. Development of a global evidence base on research integrity could accelerate the diffusion of effective approaches to addressing specific problems or issues. The global interacademy organizations are playing a role in this process with their publications Responsible Conduct in the Global Research Enterprise: A Policy Report (IAC-IAP, 2012) and Doing Global Science: A Guide to Responsible Conduct in the Global Research Enterprise (IAP, 2016).

Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×
Page 205
Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×
Page 206
Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×
Page 207
Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×
Page 208
Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×
Page 209
Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×
Page 210
Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×
Page 211
Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×
Page 212
Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×
Page 213
Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×
Page 214
Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×
Page 215
Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×
Page 216
Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×
Page 217
Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×
Page 218
Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×
Page 219
Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×
Page 220
Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×
Page 221
Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×
Page 222
Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×
Page 223
Suggested Citation:"11 Findings and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press. doi: 10.17226/21896.
×
Page 224
Next: Appendixes »
Fostering Integrity in Research Get This Book
×
Buy Paperback | $54.00 Buy Ebook | $43.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The integrity of knowledge that emerges from research is based on individual and collective adherence to core values of objectivity, honesty, openness, fairness, accountability, and stewardship. Integrity in science means that the organizations in which research is conducted encourage those involved to exemplify these values in every step of the research process. Understanding the dynamics that support – or distort – practices that uphold the integrity of research by all participants ensures that the research enterprise advances knowledge.

The 1992 report Responsible Science: Ensuring the Integrity of the Research Process evaluated issues related to scientific responsibility and the conduct of research. It provided a valuable service in describing and analyzing a very complicated set of issues, and has served as a crucial basis for thinking about research integrity for more than two decades. However, as experience has accumulated with various forms of research misconduct, detrimental research practices, and other forms of misconduct, as subsequent empirical research has revealed more about the nature of scientific misconduct, and because technological and social changes have altered the environment in which science is conducted, it is clear that the framework established more than two decades ago needs to be updated.

Responsible Science served as a valuable benchmark to set the context for this most recent analysis and to help guide the committee's thought process. Fostering Integrity in Research identifies best practices in research and recommends practical options for discouraging and addressing research misconduct and detrimental research practices.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!