Practical Guidance on Science and Engineering Ethics Education for Instructors and Administrators: Papers and Summary from a Workshop December 12, 2012 (2013)

Why Teach Research Ethics?

MICHAEL KALICHMAN
Center for Ethics in Science and Technology
University of California at San Diego

This analysis begins with an assumption: As teachers of research ethics we care about the outcomes of our teaching. As is true for other types of teaching, for experimental science, for the practice of engineering, and for Alice in Wonderland (Carroll 1865), it is paramount to know where we want to go (i.e., our outcomes) before deciding how to get there. To offer some provisional guidance on where we want to go, both empirical findings and theoretical arguments will be reviewed.

Goals Based on Federal Requirements

Analysis of research and engineering ethics education goals in the United States should begin with a recognition that before the announcement of federal requirements for training in responsible conduct of research (RCR), research ethics courses, workshops, and seminars were rare. Since the announcement of the first National Institutes of Health (NIH) requirement (NIH 1989), NIH requirements have expanded (NIH 1992, 2009), the National Science Foundation (NSF) introduced an RCR requirement (NSF 2009), and increasing numbers of graduate training programs and even entire institutions are requiring RCR training independent of sources of funding.

Given that much of existing ethics education in science and engineering is in response to these federal requirements, the question of goals might be first addressed by reviewing the goals of the requirements. However, while the various announced federal requirements (NIH 1989, 1992, 2000, 2009; NSF 2009) speak to varying degrees about the means (e.g., topics to be covered, how many hours, who is responsible for complying with the requirement), none explicitly ties those means to the intended ends. NIH (1989) opens with the statement that “A fundamental aspect of research is that it be conducted in an ethical and scientifically responsible manner,” leaving the implicit assumption that the required RCR instruction will achieve this goal. NIH (1992) seeks only “that all NRSA [National Research Service Award] supported trainees are provided an opportunity for training in the responsible conduct of research.” The Public Health Service (NIH 2000) articulated clear and ambitious goals (“promoting the responsible conduct of research and discouraging research misconduct and questionable research practices through education and awareness”), but also only implies that RCR education should, can, and will make this all possible. Furthermore, while this requirement was the most explicit of all, it

was suspended just two months after being announced (NIH 2001). The updated NIH requirement (NIH 2009) is highly prescriptive about what should be done for RCR training, but only implies the intended outcomes by defining RCR as

the practice of scientific investigation with integrity. It involves the awareness and application of established professional norms and ethical principles in the performance of all activities related to scientific research.

Finally, NSF (2009) does little more than call for “appropriate training and oversight in the responsible and ethical conduct of research.”

In short, the published requirements for RCR education provide more insight into the intended means than the ends (goals) to be met or aspired to. This is to be contrasted with the criteria for accrediting engineering programs, which have emphasized outcomes rather than methods since adoption in 2000. Three of the current accreditation criteria (Engineering Accreditation Commission 2012, p. 3) are directly relevant to the ethical practice of engineering:

(c) an ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability…
(f) an understanding of professional and ethical responsibility…
(h) the broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and societal context

Decreased Research Misconduct

Although the federal RCR requirements provide limited insight into intended goals or outcomes, there is good reason to believe that they were put in place as a response to research misconduct.

In the 1980s, allegations of research misconduct involving top US scientists—for example, David Baltimore (Kevles 1998) and Robert Gallo (Crewdson 2003)—caused considerable concern not only in the scientific community but even in the public domain. Principals in both of the named cases were called to testify before a subcommittee of the House Committee on Energy and Commerce chaired by Representative John Dingell (Kevles 1998; Crewdson 2003). In 1989, Walter Schaffer, NIH’s newly assigned research training officer, was charged with coordinating the development of an educational requirement for the NRSA program. Schaffer’s recollection (personal communication, 2012) is that

a few high visibility misconduct cases…may have precipitated discussions related to training and education in the responsible conduct of research. These discussions and the fact that several institutions had already instituted apparently successful training modules led to the issuance of the notice in the NIH Guide to Grants and Contracts [NIH 1989].

With the 2009 update, it is clear that the NIH continues to believe that RCR education is an important part of graduate and postdoctoral research training. Similarly, the NSF requirement was plausibly promoted in part as a reaction to a perceived increase in research misconduct cases.

Increased Responsible Conduct of Research

Although a specific focus on decreased research misconduct may have been a motivating factor in crafting federal requirements for RCR education, this isn’t explicitly clear in the text of the requirements. Yet both the NIH and the NSF requirement clearly speak to the need to teach RCR. Consistent with this goal, Claudia Blair (personal communication, approximately 1996), former director of the NIH Institutional Affairs Office, noted that an important factor in developing the RCR education requirement was that past NIH trainees reported that they weren’t taught key aspects of their responsibilities (e.g., authorship, use of animal and human subjects) and that this was not likely to change unless such education was required of research faculty.

What, then, should be the goals for our teaching? One possibility is to simply meet the requirements to provide RCR education. But how clear are those requirements about what is expected? At present, the two prominent federal requirements for RCR education are those of the NIH and NSF. The NIH requirements, as summarized recently (NIH 2009), appear to be far more explicit than those of the NSF (2009), which simply states that “appropriate training” should be provided for “undergraduates, graduate students, and postdoctoral researchers who will be supported by NSF to conduct research.” In contrast, the NIH requirement gives much more detail about how teaching should be done, while not explicitly stating what the teaching is intended to do (Table 1).

A superficial review of the nine topics shown in Table 1 might suggest that NIH has in fact been very specific about what should be taught. But a list of topics is not a curriculum. For example, what does it mean to teach about “collaborative research including collaborations with industry”? Is it important to explain the risks of collaboration? Highlight published guidelines on collaborations? Address regulatory controls, including material transfer and intellectual property agreements? Explain possible conflicts in academic-industry collaborations? Provide advice on how to negotiate a collaboration? Review examples of great successes in collaboration? Or great failures? Identify possible causes of failed collaborations? Teach negotiation skills? Change attitudes about the advantages or disadvantages of collaborating with others? Produce students who will engage in more collaborations? Or fewer collaborations? Clearly it is not possible to cover all of this if, for example, the NIH guidelines nominally call for 9 topics to be covered in just 8 hours, leaving less than an hour for each topic.

TABLE 1 Topics recommended to be included as proposed in iterations of NIH requirements for RCR instructions.

n.a., not addressed

Improved Ethical Decision Making

In addition to the goals of decreasing research misconduct and increasing responsible conduct, there is a third possibility favored by many. The requirements for research ethics courses do not cite ethical decision-making skills as the goal of such teaching, but this is an objective often highlighted (e.g., many of the questions for this workshop focus on ethical decision making). The word “ethics” does not appear in the body of the original NIH requirements (NIH 1989, 1992), but the 1992 keywords include “Ethics/Values in Science & Technol[ogy].” The more recent update (NIH 2009) includes the words “ethics” or “ethical” eight times, in the context of “ethical behavior,” “ethical issues,” “ethical principles,” and “professional ethics.” And the 2009 NSF requirement mentions “ethical conduct of research.” Arguably both requirements are referencing something more than research misconduct, but it isn’t made clear whether the goal of promoting ethics means that the trainees are presumed to have deficits in knowledge, skills, attitudes, or perhaps all three.

Success in Meeting Goals

Either directly or by inference, a case can be made that the goals of federal RCR requirements include decreased research misconduct, increased responsible conduct of research, and improvements in ethical decision making. These are admirable goals, but it is important to determine whether they are in fact realistic outcomes for adult training programs that last nominally 8 hours (the floor set by the NIH requirement), and only in rare cases may be as much as 30 hours (e.g., the UC San Diego course on Ethics and Survival Skills in Academia). Decreased Research Misconduct

Unfortunately, if decreased research misconduct is the goal, there is little evidence of success. And if research misconduct is assumed to be intentional, knowing, or reckless misrepresentation (e.g., fabrication, falsification, or plagiarism), then it is fair to ask whether it is realistic to think that a typical research ethics course (of 8 hours or even 30 hours), much less a workshop (of as

little as 1 hour) could somehow change someone who would lie, cheat, or steal into someone who would not.

The proposition that RCR courses might dissuade students from committing research misconduct is problematic in at least two ways. First, statistically speaking it would be an overwhelming task to find evidence of an impact for an individual course because the rate at which cases of research misconduct are reported, investigated, and publicly announced is so low. It is therefore necessary to rely on national averages across diverse, isolated, and ad hoc courses. Second, despite over 20 years of RCR courses, there is no evidence that the courses have had any impact nationally on findings of research misconduct (Kalichman 2009). Although, as shown in Figure 1, the number of allegations (which includes all cases whether or not there was a finding of research misconduct) decreased during the period 1994–2010 (p < 0.05), the trend for research misconduct findings is not statistically significant (p > 0.05), notwithstanding a spike among Office of Research Integrity closed cases in 1995. This should not be surprising. As noted above changing someone’s character certainly seems an overly ambitious goal for a short-term RCR course.

FIGURE 1 Annual closed cases for allegations of research misconduct as reported by the Office of Research Integrity (ORI 2012) for the period 1994–2010. By linear regression, the total number of allegations per year fell during this period (p < 0.05), but findings of research misconduct show no trend either up or down (p > 0.05).

Improved Ethical Decision Making

One view of the NIH requirement was that its goal was to produce more ethical scientists (Elliott and Stern 1996, p. 347):

When the National Institutes of Health mandated training in ethics, the expectation was that funded organizations do what they could to produce ethical scientists: — that is, scientists who were committed to performing their work in a responsible manner. But, of course, no training grant can produce “ethical scientists” any more than it can produce “successful scientists.” What ethics training can do is to identify a narrow scope of skills and knowledge that students should learn. Through evaluation, ethics professors can then decide if these skills and knowledge have been learned.

To the extent that the goal is to improve the skill of ethical decision making, as opposed to moral development, Elliott and Stern (1996) make the case that this measure is amenable to teaching and evaluation. That said, ethical decision making is an interesting goal in that bad decision making might result from one of two circumstances: (1) a failure of character (e.g., someone who is readily willing to lie, cheat, or steal to achieve desired ends) or (2) someone of good character who lacks the necessary knowledge or skills to make good choices. Few people would argue that even the most remarkable of typically brief training programs could correct a character deficit in children (Kohn 1997), much less adults. And even if it could, one would hope that the number of people of such poor character is a very small fraction of researchers. On the other hand, if the assumption is that most researchers are of fundamentally good character but lacking in the skill or knowledge to make good choices, then this might be an ideal target for RCR training programs.

Many studies have successfully demonstrated that teaching students ethical decision making skills can result in statistically significant, positive results (e.g., Bebeau 2002; Bebeau et al. 1995; Mumford et al. 2008), but this is not always the case (e.g., Drake et al. 2005; Heitman et al. 2001; Schmaling and Blume 2009). A meta-analysis of published studies (Antes et al. 2009) showed frequent examples of improvement, but unfortunately the improvement was modest. Antes and colleagues (2010) reviewed much of the literature on effectiveness of RCR education and found that the results overall have been equivocal at best (p. 519):

These findings indicated that RCR instruction may not be as effective as intended and, in fact, may even be harmful. Harmful effects might result if instruction leads students to overstress avoidance of ethical problems, be overconfident in their ability to handle ethical problems, or overemphasize their ethical nature.

In fairness, it is worth asking whether these modest findings are an effect of averaging results from both courses taught with best practice pedagogies and those based, for example, on lecture alone. While it is reasonable to expect that the type of pedagogy does matter, the studies that have been conducted are almost invariably based on case-based discussions or other approaches that promote active learning and engagement. In short, the modest success of these courses is plausibly not the result of pedagogical failures alone.

In any case, if modest, but positive, changes are possible, this may be sufficient reason to teach RCR courses. However, that view might be tempered by knowing that most people, including most of those who now teach RCR courses, never had such training. Does this absence

of training mean they lack the capacity to make good ethical decisions? That is certainly possible, but seems unlikely. And if it is unlikely, then it is fair to ask why scores improve if adult students already have the ability to make good decisions? One simple possibility is that the artificial nature of asking students to address fictitious or real cases in an “ethics” class results in improvement largely because before the class the students didn’t know what the instructor was looking for (e.g., writing a detailed explanation of who has what at stake and considering ethical principles in choosing among competing possible actions). That doesn’t mean they would have made bad decisions on their own, it simply means they don’t yet know how their answer will be scored. This is an important distinction and could easily be tested by giving a new group of students a case for analysis and randomly assigning them to receive either no further instruction or guidance on what should be included in their analyses. It is worth considering the possibility that the second group would show a modest improvement not unlike that found for RCR courses (Antes et al. 2009).

Increased Responsible Conduct

If decreased research misconduct and improved ethical decision making are not realistic goals for RCR courses, then what remains? Both the NIH and NSF have labeled their requirements as RCR requirements. Therefore it is reasonable to assume that an intended outcome is an increase in the responsible conduct of research. What that means for NSF is not clear, but by inference from the list of topics for the NIH requirement (Table 1), the intention appears to be that researchers, engineers, and others should learn some set of facts about those topics. This begs many questions about which facts should be covered and whether all topics listed are equally appropriate to all kinds of scientists and engineers. However, it is reasonable to ask whether there is evidence that RCR courses can increase such knowledge.

Unfortunately, once RCR knowledge is reduced to testable facts, there are few things that might be asked, and they might not be terribly interesting (e.g., “What does IRB stand for?” or “Is plagiarism an example of research misconduct?”). Given these limitations, studies of the impact of RCR teaching on factual knowledge and on self-reported behaviors and perspectives have shown only modest effects (Antes et al. 2009; Elliott and Stern 1996; Powell et al. 2007; Schmaling and Blume 2009), no effects (Drake et al. 2005; Kalichman and Friedman 1992), or even negative effects (Anderson et al. 2007; Eastwood et al. 1996).

Current Goals for RCR Education

RCR requirements were likely created in the hope of decreasing research misconduct, but there is no evidence that simply meeting the requirements has had or will have an impact on research misconduct. Whether the requirements are interpreted as calls to decrease research misconduct, increase RCR, or increase ethical decision making, it is clear that the texts of the requirements do little to link particular methods or approaches to achieving those goals. NSF (2009) has given no guidance other than a mandate to provide “appropriate training and oversight in the responsible and ethical conduct of research,” and the NIH (2009) emphasizes expectations of what should be done (e.g., topics to be covered, hours of instruction, faculty involvement) rather than any particular outcome. In the absence of compelling evidence linking training programs to these outcomes, it is fair to conclude that the actual goals remain undefined.

Instead of relying only on federal requirements to inform goals for research ethics education, it might be useful to review how teachers of research ethics view the purposes of their teaching. The corresponding array of “goals” for teaching research ethics is vast (Kalichman and

Plemmons 2007); just a few examples are summarized in Table 2. By inference, probably the only consistent goal is to meet the requirements to provide RCR education. Ironically, as noted above, those requirements are rooted in a goal (decreased research misconduct) that is probably out of reach of most, if not all, courses.

TABLE 2 Selected examples of the wide variety of goals expressed by RCR instructors surveyed in 2003 and 2004 (Kalichman and Plemmons 2007).

Criteria for Goals

It may, therefore, be of value to think differently about this. Historically, in the absence of requirements for RCR education, relatively little was done formally. Then, in the face of increasing requirements, RCR education programs, approaches, and resources grew dramatically from 1989 to the present. Although the resulting options vary widely in terms of what is done (Mastroianni and Kahn 1998; Kalichman and Plemmons 2007) and what goals are pursued (Kalichman and Plemmons 2007), they typically meet the requirement to do something. From a cynical perspective, that might be seen as enough. However, from a more practical perspective, perhaps the requirements should not be seen as an end in themselves (i.e., to comply) but as a means to do something that has real value to the scientific enterprise. Specifically, perhaps there are goals that are particularly worth pursuing and that would also meet the federal requirements. If so, what features would be desirable for those goals?

A provisional list of criteria for goals worth pursuing was one of the challenges addressed at an NSF-funded meeting at Asilomar (in Pacific Grove, CA) in March 2012. The conference convened 18 leaders in RCR education, a program evaluator, and 5 graduate students. While agreement was not always unanimous, the following summarizes a consensus about criteria

worth considering for optimal goals for RCR education (listed in order of importance and priority):

1.   Important: The goal should address something that is particularly relevant (important) to the ethical or responsible conduct of science.

2.   Deficient: Some things that are important may not in fact be lacking. The goal should address something that needs improvement or correction because it is deficient.

3.   Independent: Even if something is important and deficient, it could be secondary to some other goal. Meeting the goal should be independent of first needing to meet other goals.

4.   Amenable to Intervention: Even if something is important and deficient, we may have no realistic way to repair that deficit. The goal should be something for which we have, or we could reasonably produce or acquire, an intervention that would enable us to make a change.

5.   Measurable: It is possible that there is something that we can change by intervention that is both important and deficient, but we have no means to assess our impact. The goal should be something for which we have the tools for defining measurable outcomes. [NOTE: Measurable outcomes can also include qualitative findings. The key is to have something credible to convince ourselves and others that there is some value added because of our efforts.]

6.   Magnitude: It is possible that there is something that we can change by intervention that is important, deficient, and measurable, but the magnitude of our impact might be too small to be considered cost effective. The goal should be something for which we can produce a change of sufficiently large magnitude.

7.   Feasible: Even if something reasonably meets all of the above criteria, it may not in fact be practical or feasible in the research environment because of the amount, type and availability of resources required or because of the characteristics of the research environment. The goal should be something that is feasible.

Suggested Principles, Goals, and Outcomes

Mindful of the above criteria for goals worth pursuing, taking into consideration past studies of research ethics education outcomes, and reflecting on anecdotal experiences of research ethics teaching, the following are suggested principles, goals, and outcome measures for research and engineering ethics teaching.

Principles

1.   The entire community of scientists and engineers benefits from diverse, ongoing options to engage in conversations about the ethical dimensions of research and engineering.

2.   Specific topics or materials aren’t important. The number of possibilities is endless. The point is to do something, not everything.

3.   Different members of the community of scientists and engineers and different disciplines are likely to require different goals, content, and approaches to ethics education.

Goals and Outcomes

A logical extension of the principles noted above is that the ultimate goals of research ethics courses should focus on the community and not on individuals. More specifically, the point is not that individuals should meet some nominal standard (i.e., leaving open the possibility that

someone could “test out” of the “ethics requirement”). Instead, the question is what sorts of outcomes might one look for with individual students that logically might serve the larger community goal. Table 3 illustrates four such goals and proposed corresponding outcome measures.

TABLE 3 Goals and outcome measures for trainees in responsible conduct of research (RCR) or responsible conduct of engineering (RCE) courses.

^*Including at least some reflection on each of four domains: (1) conduct of research (e.g., truth in reporting, mitigating bias); (2) protections for subjects of research (e.g., animal and human subjects); (3) interactions with colleagues (e.g., collaboration, peer review, mentoring); and (4) social responsibility (e.g., considering public interest in choice of projects, communicating with the public).

Conclusion

A prerequisite for effectiveness in ethics education for scientists and engineers is to be clear about the goals of that education. Based on data and experience, a case can be made that those goals should not reside solely in attempting to decrease misconduct, in an isolated course or program, nor in “one size fits all” approaches to topics to be covered and approaches to be used. Instead, a suggested beginning point is goals that emphasize increased conversations about responsible conduct; increased awareness of rules, issues, options, and resources; a clear understanding of the meaning of “ethical decision making”; and an acceptance of the importance of lifelong learning about responsible conduct.

Acknowledgments

Criteria for goals were developed jointly by the participants in an NSF-funded conference at Asilomar in March 2012. The participants are thanked for their roles in developing these criteria; however, the final version and the contents of this manuscript represent the views of the author and are not necessarily endorsed by the Asilomar participants: John Ahearne (Sigma Xi), Melissa Anderson (University of Minnesota), Mark Appelbaum (UC San Diego), Yuchen Cao (UC San Diego), Michael Davis (Illinois Institute of Technology), Chris DeBoever (UC San Diego), Mark Frankel (AAAS), C.K. Gunsalus (University of Illinois at Urbana-Champaign), Elizabeth Heitman (Vanderbilt), Joseph Herkert (Arizona State University), Rachelle Hollander (National Academy of Engineering), Crane Huang (UC San Diego), Deborah Johnson (University of Virginia), Nancy Jones (National Institute of Allergy and Infectious Diseases, NIH, DHHS), Nelson Kiang (Harvard Medical School), Philip Langlais (Old Dominion University), Francis Macrina (Virginia Commonwealth University), Brian Martinson (HealthPartners Research Foundation), Michael Mumford (University of Oklahoma), Ken Pimple (Indiana University), Dena

Plemmons (UC San Diego), Patrick Wu (UC San Diego), and Guangming Zheng (UC San Diego). The writing of this manuscript was supported in part by NSF Grant #1135358 and NIH Grant #UL1RR031980.

References

Anderson MS, Horn AS, Risbey KR, Ronning EA, DeVries R, Martinson BC. 2007. What do mentoring and training in the responsible conduct of research have to do with scientists’ misbehavior? Findings from a national survey of NIH-funded scientists. Academic Medicine 82(9):853–860.

Antes AL, Murphy ST, Waples EP, Mumford MD, Brown RP, Connelly S, Devenport LD. 2009. A metaanalysis of ethics instruction effectiveness in the sciences. Ethics and Behavior 19(5):379–402.

Antes AL, Wang X, Mumford MD, Brown RP, Connelly S, Devenport LD. 2010. Evaluating the effects that existing instruction on responsible conduct of research has on ethical decision making. Academic Medicine 85(3):519–526.

Bebeau MJ. 2002. Influencing the moral dimensions of professional practice: Implications for teaching and assessing for research integrity. In: Steneck NA and Sheetz MH, eds. Proceedings of the First ORI Research Conference on Research Integrity. Rockville MD: Office of Research Integrity, Department of Health and Human Services. pp. 179–187. Available online at www.ori.hhs.gov/documents/proceedings_rri.pdf.

Bebeau MJ, Pimple KD, Muskavitch KMT, Borden SL, Smith DH. 1995. Moral Reasoning in Scientific Research: Cases for Teaching and Assessment. Bloomington: Indiana University Press.

Carroll L. 1865. Alice’s Adventures in Wonderland. London: MacMillan and Co.

Crewdson J. 2003. Science Fictions: A Scientific Mystery, a Massive Cover-up and the Dark Legacy of Robert Gallo. New York: Little, Brown & Co.

Drake M, Griffin P, Kirkman R, Swann J. 2005. Engineering ethical curricula: Assessment and comparison of two approaches. Journal of Engineering Education 94:223–231.

Eastwood S, Derish P, Leash E, Ordway S. 1996. Ethical issues in biomedical research: Perceptions and practices of postdoctoral research fellows responding to a survey. Science and Engineering Ethics 2:89–114.

Elliott D, Stern JE. 1996. Evaluating teaching and students’ learning of academic research ethics. Science and Engineering Ethics 2:345–366.

Engineering Accreditation Commission. 2012. Criteria for Accrediting Engineering Programs: Effective for Reviews During the 2012-2013 Accreditation Cycle. Baltimore: Accreditation Board for Engineering and Technology (ABET). Available online at www.abet.org/uploadedFiles/Accreditation/Accreditation_Process/Accreditation_Documents/Current/eac-criteria-2012-2013.pdf.

Heitman E, Salis PJ, Bulger RE. 2001. Teaching Ethics in Biomedical Science: Effects on Moral Reasoning Skills. Investigating Research Integrity. Proceedings of the First ORI Research Conference on Research Integrity. pp. 195–202.

Kalichman MW. 2009. Evidence-based research ethics. American Journal of Bioethics 9(6&7):85–87.

Kalichman MW, Friedman PJ. 1992. A pilot study of biomedical trainees’ perceptions concerning research ethics. Academic Medicine 67:769–775.

Kalichman MW, Plemmons D. 2007. Reported goals for responsible conduct of research courses. Academic Medicine 82(9):846–851.

Kevles DJ. 1998. The Baltimore Case: A Trial of Politics, Science and Character. New York: W.W. Norton.

Kohn A. 1997. How not to teach values: A critical look at character education. Phi Delta Kappan 78(6):428–439.

Mastroianni AC, Kahn JP. 1998. The importance of expanding current training in the responsible conduct of research. Academic Medicine 73(12):1249–1254.

Mumford MD, Connelly MS, Brown RP, Murphy ST, Hill JA, Antes AL, Waples EP, Devenport LR. 2008. A sensemaking approach to ethics training for scientists: Preliminary evidence of training effectiveness. Ethics and Behavior 18:315–346.

NIH [National Institutes of Health]. 1989. Requirement for Programs on the Responsible Conduct of Research in National Research Service Award Institutional Training Programs. NIH Guide 18(45):1. Available online at http://grants.nih.gov/grants/guide/historical/1989_12_22_Vol_18_No_45.pdf.

NIH. 1992. Reminder and Update: Requirement for Instruction in the Responsible Conduct of Research in National Research Service Award Institutional Training Grants. NIH Guide 21(43). Available online at http://grants.nih.gov/grants/guide/notice-files/not92-236.html.

NIH. 2000. PHS Policy on Instruction in the Responsible Conduct of Research (RCR) Announced December 5. Available online at http://grants.nih.gov/grants/guide/notice-files/NOT-OD-01-007.html.

NIH. 2001. Notice of suspension of “PHS Policy on Instruction in the Responsible Conduct of Research.” February 22. Available online at http://grants.nih.gov/grants/guide/notice-files/NOT-OD-01-020.html.

NIH. 2009. Update on the Requirement for Instruction in the Responsible Conduct of Research. November 24. Available online at http://grants.nih.gov/grants/guide/notice-files/NOT-OD-10-019.html.

NSF [National Science Foundation]. 2009. Responsible Conduct of Research. Proposal and Award Policies and Procedures Guide. Part II – Award and Administration Guidelines, p. IV-3. Available online at www.nsf.gov/pubs/policydocs/pappguide/nsf10_1/nsf10_1.pdf.

ORI [Office of Research Integrity]. 2012. Annual Reports. Rockville MD: Department of Health and Human Services. Available online at http://ori.hhs.gov/annual_reports.

Powell S, Allison MA, Kalichman MW. 2007. Effectiveness of a responsible conduct of research course: A preliminary study. Science and Engineering Ethics 13(2):249–264.

Schmaling KB, Blume AW. 2009. Ethics instruction increases graduate students’ responsible conduct of research knowledge but not moral reasoning. Accountability in Research 16:268–283.