Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
4 Institutional Approaches to Fostering Integrity in Research Research organizations currently rely on a variety of methods for promoting integrity in research. They establish organizational compo- nents to comply with regulations imposed by an external environment; they offer educational programs to teach the elements of the responsible conduct of research;1 they implement policies and procedures that delin- eate the normative practices of responsible research and establish criteria for rewards and recognition; and they develop processes to evaluate and enforce institutional behavior.2 In addition, organizations engage in ac- tivities that help establish an internal climate and organizational culture that are either supportive of or ambivalent toward the responsible con- duct of research.3 These various approaches are not mutually exclusive, however, nor should they be. A number of programs and activities, inte- grated across the various levels of an organization, should be in place in order to maximize the impact on the research environment and to sup- port the responsible conduct of research. 1Chapter 5 provides a detailed discussion of promoting integrity in research through education. 2Chapters 2 and 3 provide discussions of the types of policies and procedures that should be implemented in a research organization to foster integrity in research. 3Chapter 3 provides a discussion of the elements of an organizationâs structure and pro- cesses that have an impact on how members of an organization perceive the ethical climate and culture. 72
INSTITUTIONAL APPROACHES TO FOSTERING INTEGRITY 73 To establish a basis for organizational learning and continuous qual- ity improvement, organizations should simultaneously implement pro- cesses for evaluating their efforts to foster responsible conduct of research. Evaluation can be approached in a variety of ways. One way is to rely on external evaluators to determine compliance with regulatory controls. Another is to rely on a system of performance-based assessments that are initiated and implemented internally. Such assessments can also be used to meet the accountability requirements of outside funding or govern- ment sources. In addition, peer reviewers may be used in institutional self-assessment processes; assessments done by peer reviewers may or may not be associated with accreditation by external organizations. This chapter provides a discussion of the strengths and limitations of these various approaches and establishes a rationale for the use of the institutional self-assessment approach to evaluation that is recommended in this report (see Chapter 6). REGULATORY COMPLIANCE The U.S. Congress has consistently affirmed the importance of integ- rity in research, and the federal government has established a framework for regulating misconduct in science. As described in Chapter 1, regula- tory language includes a definition of research misconduct and spells out the requirements for institutional policies and practices to handle reports of misconduct or other types of wrongdoing in the research environment. A regulatory framework requires a rule-making process that may be governed by legislative or administrative actions. The framework may be decentralized (i.e., each institution may develop its own approach), or it may require compliance with a common set of criteria, policies, and prac- tices. The safety standards promulgated by the Occupational Safety and Health Administration (OSHA) are an example of the latter. Regulatory models commonly involve data collection and the generation of reports that document each institutionâs compliance with governmental policies. OSHA, for example, very clearly specifies requirements for recording and reporting on-the-job injuries and illnesses, and is authorized to conduct workplace inspections to check reporting and insure compliance. Assessment Strategy Common components of regulatory frameworks include the specifi- cation of certain procedures and reporting requirements, the collection of data, and the preparation of reports of compliance practices. The regula- tory approach also involves a governmental unit that maintains oversight of the compliance and reporting procedures, investigates complaints
74 INTEGRITY IN SCIENTIFIC RESEARCH about rule violations, and offers technical assistance in rule-making and implementation of regulations. In university settings, evaluations for mandatory compliance with government regulatory standards are com- mon. In this framework, full-time government employees frequently (but not always) conduct inspections to evaluate compliance with a well-de- fined set of standards, followed by some mechanism for feedback (usu- ally an exit interview or a written report). Individual inspectors do not necessarily require professional credentials in the field or activity being evaluated, although many individuals in such positions have such quali- fications (i.e., they are not necessarily âpeersâ of the individuals whose conduct is being evaluated). In some highly regulated research environ- ments, such as those in which new drugs are developed, iterative partner- ships can evolve between regulators (e.g., the Food and Drug Administra- tion) and the regulated entities (e.g., pharmaceutical companies). Strengths Several models of regulatory frameworks for research that could be adapted to the oversight of integrity in research already exist. These mod- els include the regulatory frameworks for the oversight of the protection of human research subjects (The Common Rule), the evaluation of mis- conduct in science, the use of animals in research, and the handling of toxic or radioactive research materials. A regulatory approach to fostering integrity in research is consistent with other governmental efforts to encourage the use of commonly ac- cepted practices and to discourage irresponsible behavior in the research environment. Researchers and institutional officials are familiar with com- pliance requirements and often participate in the preparation of rule- making procedures. Individual research centers frequently have some latitude and discretion in adapting government requirements to their own needs, and the centers are responsible for designating specific officials who will ensure that faculty, trainees, and administrative staff under- stand the importance of the regulations. A regulatory framework fosters a collective and consistent response to social concerns by a broad array of research institutions, and it highlights best practices. The framework also can reduce the opportunity for idiosyncratic and irresponsible policies or practices to be implemented. Limitations A regulatory approach fostering integrity in research also has some limitations (Nature, 2001; Burman, et al., 2001; Sugar, 2002). Such an ap- proach increases the bureaucratization of science and requires documen-
INSTITUTIONAL APPROACHES TO FOSTERING INTEGRITY 75 tation that institutions may find burdensome. A 1999 report commis- sioned by the National Institutes of Health (NIH) at the request of Con- gress reviewed the burden of regulations across five key areas, including research integrity. The report identified four problems that were present in all the areas: rigid regulations that dictate process often limit flexibility without enhancing results; rules imposed by multiple agencies have in- consistent requirements; regulation of science by nonscience agencies of- ten leads to additional and nonproductive regulatory burden; and poor communication exists among federal agencies and research institutions (NIH, 1999). The report also notes that regulatory systems tend to impose the same requirements on all research institutions regardless of their char- acteristics. That is, requirements are aimed at the lowest performer, and hence provide little incentive for superior performance. Regulations often emphasize the areas of common agreement and can reduce important concerns to rules and procedures. It is difficult or im- possible for regulations alone to foster an understanding of the critical issues involved, and the required procedures are not always related to the desired outcomes. The adoption of new regulations and the creation of institutional and governmental oversight offices increase the cost of do- ing science and add to the administrative costs of research centers with- out necessarily creating a commensurate benefit. In addition, once regula- tions are adopted, they are difficult to change. Finally, in some instances regulators are forced to focus more on process than on results. That is, the regulators seek to evaluate the extent to which observable processes that are hypothesized to promote some desired outcome are in place; examples of such processes might include protocols to protect human or animal research subjects, measures to ensure laboratory safety, or desirable data- handling practices. Although a focus on process can facilitate intended outcomes, it can also be directed to matters that are relatively unimpor- tant or that are poorly connected with the desired result. A PERFORMANCE-BASED APPROACH A performance-based model for the evaluation of organizational ef- forts to foster integrity in the research environment offers selected goals and benchmarks4 that can be used as criteria to assess the success of 4A benchmark is a standard or point of reference used in measuring and/or judging quality or value. Benchmarking is the process of continuously comparing and measuring an organizationâs performance, practices, policies, and philosophies against leading, high-per- forming organizations anywhere in the world to gain information that will help the organi- zation take action to improve its performance (NPR, 1997).
76 INTEGRITY IN SCIENTIFIC RESEARCH efforts. These goals and benchmarks are generally linked to rewards, in- centives, and, at times, penalties for specific types of behavior. A perfor- mance-based approach provides a direct role for research institutions and research team leaders in fostering norms for faculty, research staff, train- ees, and students within diverse research settings. The development of a performance-based model would require insti- tutions to formulate a coherent statement of goals that describes the prin- ciples of integrity in research that they wish to encourage. The model would also require institutions to implement these goals through a series of actions and assessment strategies. Such actions could include the fol- lowing: â¢ posting the statement (including selected criteria related to person- nel actions, such as recruitment offers and hiring and promotion policies and practices) in public places throughout the research institution; â¢ creating a bonus plan or award system to reward exceptional be- havior; â¢ providing mentorship opportunities for senior and junior faculty and investigators that emphasize the importance of learning about the responsible conduct of research; and â¢ publicizing and possibly sanctioning actions that are inconsistent with the institutionâs research mission. Performance-based systems require that benchmarks be set and that a database be generated in order to measure faculty and trainee compliance with the specified standards to identify areas that need improvement. Such benchmarks can be formal or informal, but they require a broad consensus that what is being measured is important and relevant to integ- rity in research. Performance benchmarks must be communicated clearly so that all members of a research institution understand what is being measured and why. Performance-based approaches should be imple- mented incrementally, because data must be collected and analyzed be- fore major institutional changes can be implemented. Such implementa- tion is in keeping with the spirit of continuous quality improvement. Research on organizational behavior found that institutions have ethi- cal climates that differ according to the values, standards, and interests of their members (Victor, 2001). Performance incentives are tools that insti- tutions use to align their membersâ behaviors and practices with the val- ues that are expressed as the institutionsâ missions and goals. These tools represent one part of a broader set of normative control systems for orga- nizations, and they are shaped not only by aspirational goals but also by societal norms, the form of the organization, the behavior of the leader- ship, and unique features of research groups.
INSTITUTIONAL APPROACHES TO FOSTERING INTEGRITY 77 Assessment Strategy Assessment strategies may focus on the development of mission state- ments and benchmark tools alone, or they may include an analysis of the ways that institutional officials use such tools to influence faculty adher- ence to responsible research practices. Assessment efforts can also be used to review compliance strategies (including the compliance of faculty and research staff), student surveys, and sponsor evaluations, as well as to analyze rewards, incentives, and penalties. In one application of a performance-based assessment, for example, Gilmer (1995) has the students in her research ethics course at Florida State University develop individual portfolios. The students must define their goals for learning and then present several pieces of their work that demonstrate that such learning has been accomplished. Portfolio assess- ment has been used primarily in educational settings to document the progress and achievements of individual students or teachers, but it has the potential to be a valuable tool for program assessment as well. Another example is the Quality Improvement Program instituted by the Office of Human Research Protections at the Department of Health and Human Services (DHHS) (OHRP, 2002). This voluntary program is intended to help institutions prepare for and successfully achieve accredi- tation of their human research protection programs by private accrediting agencies. One feature of this program is a Quality Assurance Self-Assess- ment Tool, which helps the institution determine its level of compliance with federal regulations. In a related effort, the Health Improvement In- stitute (a private foundation), under a contract with DHHS, has estab- lished a national Award for Excellence in Human Research Protection, and the institute currently is developing performance-based evaluation criteria for the award (HII, 2002). On a more general basis, a classic example of performance-based as- sessment is the Government Performance and Results Act of 1993 (Pub. L. No: 103â62), which requires federal agencies to develop strategic plans for delivering high-quality services or products to the public. Each agency has to (1) establish top-level agency goals and objectives, as well as an- nual program goals; (2) define how it intends to achieve those goals; and (3) demonstrate how it will measure agency and program performance in achieving those goals (NPR, 1997). As part of this initiative, the National Performance Review (NPR) was created to sponsor and organize bench- marking studies aimed at making government work better and cost less. Performance-based assessment and benchmarking also are the basis for the Malcolm Baldrige Quality Award. Established by the Malcolm Baldrige National Quality Improvement Act of 1987 (Pub. L. No. 100â 107), the award is administered by the National Institute of Standards and
78 INTEGRITY IN SCIENTIFIC RESEARCH Technology (NIST) and recognizes U.S. organizations for their achieve- ments in quality and performance. The awards are given to businesses (manufacturing, service, small business) and to education and health care organizations (Seymour, 1995). Judging is based on seven criteria: leader- ship, strategic planning, customer and market focus, information and analysis, human resource focus, process management, and business re- sults (NIST, 2001). The Baldrige criteria are used by thousands of diverse organizations for self-assessment and training and as a means to develop performance and business processes. Several Institute of Medicine (IOM) reports describe performance- based approaches to improving community health (IOM, 1996a, b, 1997). Strengths Performance-based systems are increasingly common in diverse in- stitutional settings, including health care (with the new emphasis on qual- ity), the transportation sector, and various sectors of the manufacturing and service industries. Conceptual frameworks, measurement tools, and institutional case studies exist that can provide the foundation for the development of such a system in the area of integrity in research (see Appendix B). The setting of benchmarks represents a visible and tangible public commitment to integrity in research, and at the same time recog- nizes the need for oversight and assessment of questionable conduct in the research environment. Such goals can be flexible and consistent with the diverse institutional cultures of different research centers. Rewards, incentives, and penalties offer valuable educational resources in that they can be used to demonstrate the types of conduct that research institutions do and do not encourage. Limitations Performance-based systems also have limitations. They require a con- siderable amount of institutional commitment and involvement. Institu- tional officers need to exercise leadership and authority in the develop- ment of a mission statement and performance goals, as well as in the selection of benchmarks that will be used to guide and evaluate behavior. The adoption of performance-based goals can be divisive and controver- sial if faculty do not share common norms and aspirations, or if such goals lead to restrictions on the types of research that can be conducted. Institution-directed performance goals may run counter to the autonomy and innovative spirit of the research environment in most academic cen- ters. Current limitations, which can be addressed through further re- search, include the need to identify the benchmarks and criteria most
INSTITUTIONAL APPROACHES TO FOSTERING INTEGRITY 79 relevant to integrity in research, and the need to develop and validate the instruments and measures for assessment. AN INSTITUTIONAL SELF-ASSESSMENT AND PEER REVIEW APPROACH The complexities of balancing formal and informal approaches to fos- tering integrity in research have led to efforts to have research institutions assess their own performances, including the performances of their man- agements, faculty and research staff, in terms of complying with stated standards, goals, and practices. Such self-assessments may include evalu- ations of aspects of certification or institutional assurance of compliance with professional standards within a broader organizational context; this practice is frequently used in the accreditation of professional schools and departments, as well as of educational institutions. This approach is de- scribed in detail in Chapter 6. It is presented here only briefly to provide a basis for comparison with the other approaches discussed above. Assessment Strategy The strategy used in the self-assessment and assurance framework has multiple distinct features. The self-assessment process may be volun- tary or mandatory. The process of institutional assurance of compliance with professional standards may be linked to an informal peer-review process or may be part of a formal accreditation procedure. Different institutions will draw on one or more of these features, according to their own goals and resources. One such approach is pre- sented for purposes of illustration. The Middle States Commission on Higher Education (MSCHE, 2002) has recently developed proposals for new accreditation processes for its member institutions. The proposals include several distinct features of the internal assessment and accreditation process, including the following: â¢ institutional self-study; â¢ a team visit; â¢ types of accreditation actions; â¢ periodic review reports; â¢ institutional profile (annual) reports; â¢ candidacy and initial accreditation procedures; â¢ public information; â¢ use of technology (e.g., electronic submission of report materials); and â¢ training of evaluators and the institutionsâ departmental chairs.
80 INTEGRITY IN SCIENTIFIC RESEARCH The accreditation proposals are intended to address the commissionâs desire to balance the need to be careful and thorough with competing concerns about the slow and time-consuming nature of self-assessment and accreditation procedures (personal communication, J. Morse, Middle State Commission on Higher Education, [January 4, 2002]; see also www. msache.org/chx02.pdf). Additional examples of assessment strategies are described in Appendix B. Strengths The mission statements of accreditation bodies frequently assert that their role is to promote academic quality through formal recognition of the importance of compliance with professional standards and to ad- vance the process of self-regulation (see, for example, the mission state- ment of the Council for Higher Education Accreditation [CHEA, 2001]). Institutional self-assessment and accreditation procedures can be powerful influences in the shaping of professional behavior within a self- regulatory system. In the mid-1990s, for example, the National Council of Examinations for Engineers and Surveyors introduced engineering ethics questions in its examination used for granting professional engineering licenses. Every state and U.S. territory now uses these examinations. The inclusion of ethics questions in the engineering examination has been a practical force for introducing these issues into the classroom as well as into the review sessions that are popular with applicants for professional engineering licenses (Rabins, 1998). If self-assessment and assurance requirements are implemented through standard practices used by faculty, then they can become impor- tant components in the overall climates of research organizations. In this way, they will have the ability to change institutional cultures over time. The process of self-regulation has the advantage of focusing on those areas that the members of the profession believe are essential to quality and integrity. The introduction of elements of integrity in research and the responsible conduct of research into ongoing processes of institutional self-assessment and assurance helps accustom researchers to addressing these issues as part of the normal accreditation process. It thus diminishes the growth of dispar- ate units within research organizations and strengthens the alignment of concerns about integrity in research with research practices. Limitations As noted above in the example of MSCHE practices, self-assessment and accreditation practices are frequently slow and time-consuming. Re-
INSTITUTIONAL APPROACHES TO FOSTERING INTEGRITY 81 sources for the training of faculty evaluators and the operation of review committees are required at multiple levels within research institutions, including individual departments, research centers, and general adminis- trations. Turnovers in personnel, a lack of familiarity with review and assessment procedures, and changes in the standards that are used as the basis for self-assessment and evaluation can further complicate the cum- bersome nature of the assessment process. At this time, an obvious limitation to the self-assessment approach as a means to foster integrity in research is the need to develop and validate evaluation instruments and measures. Further research on methods and measure and elements of the research environment (as described in Chap- ter 7) should eliminate this limitation. Examples of relevant assessment tools that might be adapted to the research environment are described in Appendix B. SUMMARY Evaluations of activities within research institutions occur in diverse forms and are influenced by different approaches that may consist of voluntary or mandatory elements and that may rely upon professional or volunteer reviewers. The committee has not found research evidence that suggests that any particular approach produces significant differences in measurable outcomes. Each approach has certain strengths as well as limitations. The regu- latory compliance approach may be more desirable when professional standards are clear and when measurable indicators can be developed to assess levels of compliance with selected processes and rules in different research settings. However, this approach often fosters attention to mini- mal standards (the lowest-common-denominator approach) rather than encouraging institutions to determine what is right for their situation and to invest in the efforts necessary to foster more desirable outcomes. Although benchmarks are not yet available to support performance monitoring in the area of research integrity, they could be developed through educational programs and consensus building efforts. The adop- tion of a performance-based approach enables institutions to move be- yond procedural compliance mechanisms as a self-regulatory device (e.g., certifying that they have adopted certain policies, procedures, and educa- tional training efforts) in favor of a framework that fosters greater indi- viduality but still adheres to certain performance standards that reflect basic peer and community values. The institutional self-assessment approach is frequently overlooked in policy debates about research oversight. This approach is commonly associated with highly specialized efforts, such as the accreditation of
82 INTEGRITY IN SCIENTIFIC RESEARCH academic institutions or procedures for the handling of laboratory ani- mals. Recent concerns about the treatment of human subjects in research studies have brought new attention to the strategies involved in institu- tional self-assessment and assurance procedures (IOM, 2001; NBAC, 2001). The committee believes that much of this analysis is relevant to discussions of integrity in research, and therefore the committee suggests that more work is needed to determine how self-assessment coupled with peer review, particularly in the context of institutional accreditation, can be adapted to efforts to foster integrity in research. Chapter 6 provides a fuller discussion of this method of assessment. REFERENCES Burman WJ, Reves RR, Cohn DL, Schooley RT. 2001. Breaking the camelâs back: Multicenter clinical trials and local institutional review boards. Annals of Internal Medicine 134:152â 157. CHEA (Council for Higher Education Accreditation). 2001. Directory of CHEA Participating and Recognized Organizations, 2000â2001. [Online]. Available: http://www.chea.org/ Directories/index.cfm [Accessed February 25, 2002]. Gilmer PJ. 1995. Teaching science at the university level: What about the ethics? Science and Engineering Ethics 1:173â180. HII (Health Improvement Institute). 2002. Award for Excellence in Human Research Protec- tions. [Online]. Available: http://www.hii.org/ [Accessed May 14, 2002]. IOM (Institute of Medicine). 1996a. Using Performance Monitoring to Improve Community Health: Conceptual Framework and Community Experience. Washington, DC: National Academy Press. IOM. 1996b. Using Performance Monitoring to Improve Community Health: Exploring the Issues. Washington, DC: National Academy Press. IOM. 1997. Improving Health in the Community: A Role for Performance Monitoring. Washing- ton, DC: National Academy Press. IOM. 2001. Preserving Public Trust. Washington, DC: National Academy Press. MSCHE (Middle States Commission on Higher Education). 2002. Characteristics of Excel- lence in Education: Eligibility Requirements and Standards for Accreditations. [Online]. Available www.msache.org/chx02.pdf [Accessed March 18, 2002]. NBAC (National Bioethics Advisory Commission). 2001. Ethical and Policy Issues in Research Involving Human Participants. Rockville, MD: U.S. Government Printing Office. NIH (National Institutes of Health). 1999. NIH Initiative to Reduce Regulatory Burden. [Online]. Available: http://grants1.nih.gov/grants/policy/regulatoryburden/ [Accessed May 6, 2002]. NIST (National Institute of Standards and Technology). 2001. Getting Started with the Baldrige National Quality Program: Criteria for Performance Excellence. [Online]. Available: http: //www.quality.nist.gov/ [Accessed May 15, 2002]. NPR (National Performance Review). 1997. Serving the American Public: Best Practices in Performance Management. [Online]. Available: http://govinfo.library.unt.edu/npr/li- brary/papers/benchmrk/nprbook.html [Accessed May 15, 2002]. Nature. 2001. Time to cut regulations that protect only regulators. Nature 414:379.
INSTITUTIONAL APPROACHES TO FOSTERING INTEGRITY 83 OHRP (Office of Human Research Protections). 2002. Quality Improvement Program. [Online]. Available: http://ohrp.osophs.dhhs.gov/humansubjects/qip/qip.htm [Accessed May 14, 2002]. Rabins MJ. 1998. Teaching engineering ethics to undergraduates: Why? what? how? Science and Engineering Ethics 4:291â302 Seymour D. 1995. The AQC Baldrige Report: Lessons Learned by Nine Colleges and Universities Undertaking Self-Study with the Malcolm Baldrige National Quality Award Criteria. Wash- ington, DC: American Association for Higher Education. Sugar AM. 2002. Letters: The crisis in local institutional review boards. Annals of Internal Medicine 136:410â411. Victor, B. 2001. Integrity in the business environment. Presentation at the June 28, 2001, meet- ing of the Institute of Medicine Committee on Assessing Integrity in Research Envi- ronments, Washington, DC.