HEATHER E. CANARY
Department of Communication
The University of Utah
JOSEPH R. HERKERT
School of Letters and Sciences
Arizona State University
Assessment is generally one of the most difficult and controversial aspects of research and educational programs and projects. In the case of science and engineering ethics research and education, it is all the more difficult because interest in assessment in this area is relatively recent and methods developed in other contexts are not always readily transferable.
This paper draws on some of the literature on assessment in science and engineering ethics as well as our own experiences in conducting two NSF-funded research projects: “Integrating Microethics and Macroethics in Graduate Science and Engineering Education: Development and Assessment of Instructional Models” (here called “MicroMacro”; Herkert et al. 2009; Canary et al. 2012) and “Developing and Assessing Macroethics Modules for the Collaborative Institutional Training Initiative (CITI) Responsible Conduct of Research Courses” (“MacroCITI”). Our experience with assessment in ethics centers is limited; we draw instead on our leadership and participation in the ethics-across-the-curriculum program sponsored by the Lincoln Ethics Teaching Fellows Program of the Arizona State University (ASU) Lincoln Center for Applied Ethics (Herkert 2011).
First we discuss challenges in assessing ethics education in programs and centers, and then strategies to address them. We end with practical guidance gained from our review and experiences.
Published accounts of assessment attempts identify several challenges inherent to assessing ethics education in academic centers and research programs (e.g., Antes et al. 2009; Davis and Feinerman 2012; Herkert et al. 2009). The difficulties arise from multiple content areas and goals represented in academic/research programs and centers as well as multiple modes of education used across contexts. However, research and experience also indicate methods for addressing these complications.
One challenge is that a collection of activities that constitute an entire academic/research program or center inevitably covers different content and represents different strategies for
content delivery. For instance, centers might take a multidisciplinary approach to ethics, with faculty weaving ethics education into their specific disciplinary courses. Although general ethical issues might be similar across disciplines, ways in which those issues surface likely are radically different. That is, a materials engineer working in a laboratory needs to be aware of conflict of interest issues just as a practicing civil engineer does, but such issues emerge differently in academic labs than they do in consulting engineering practice.
Similarly, ethics education programs typically draw on faculty, students, and other participants across time and interests, all of which constitute contextual differences. Any center or program assessment efforts must take these disciplinary and contextual differences into account. For example, Davis and Feinerman (2012) noted that incorporating ethics in different engineering courses at two universities required assessment that would be specific enough to tap what students learned and at the same time broad enough to apply across instructional contexts. Assessment strategies constructed too broadly do not effectively measure anything, or simply tap what would be considered “common knowledge.” On the flip side, those constructed too narrowly for specific courses or delivery modes cannot be compared across contexts.
A second assessment challenge is how to tap differing foci and goals represented by components of programs and centers. Antes and colleagues (2009), in their meta-analysis of ethics education in science, point to the necessity of matching assessment criteria to instructional goals. Although that might seem obvious, it is not always apparent or easy to find ways to assess the actual goals or outcomes of ethics education because of the many possible foci of such efforts. More specifically, entire programs or centers might include an array of goals and outcomes that are not all accomplished in any one context or endeavor. One program activity might focus on ethical decision making and another on understanding standards for ethical behavior.
In an attempt to discern the most relevant issues as well as the most effective strategies for teaching ethics in the sciences, the Antes et al. (2009) meta-analysis distilled potential moderating variables—such as instructional context, instructional activities, and student characteristics—that influence outcomes of such instruction. These elements influence what is assessed as well as best ways to assess learning and outcomes. The authors’ analysis points to the importance of carefully designing ethics educational contexts that are interactive, incorporate content about domains of ethical behavior (e.g., conflicts of interest, authorship), and include several types of instructional activities. Antes and colleagues also note that few published studies of ethics instruction constitute rigorous research projects that provide enough information to compare results across studies. Our experiences in our NSF-funded studies and in preparing this paper lead us to concur with that conclusion.
Our initial effort (the MicroMacro project) to conduct the type of rigorous study of ethics education recommended by Antes and colleagues (2009) took these challenges and issues into account. As we report in two papers (Canary et al. 2012; Herkert et al. 2009), we took great care to design alternative modes for teaching about social and ethical issues encountered at both the individual level (“micro ethics”) and the collective level (“macro ethics” or “social responsibility”). We sought input from experts from across the United States to identify realistic and relevant instructional goals.
The earlier paper (Herkert et al. 2009) discusses the rather lengthy process involved in developing goals and outcomes that would be both realistic and assessable. One step in that process involved a three-day workshop with about 20 people participating in several in-depth sessions about micro- and macroethical issues important for students to consider (see Herkert et al. for a complete account of issues, goals, and outcomes discussed). An example of a micro
issue identified during the workshop is “professional norms such as objectivity, transparency, accuracy, and efficiency” (Herkert et al., p. 7), and an example of a macro issue, “ways to envision the possible social implications of research” (Herkert et al., p. 7). An in-depth discussion of the relative merits of teaching micro- and macroethics is beyond the scope of this paper; we refer the reader to our earlier conference proceedings (Canary et al. 2012; Herkert et al. 2009).
One workshop session was devoted to assessment, and resulted in new ideas for matching assessment approaches to instructional goals. After the workshop, the research team met to refine workshop ideas into manageable instructional goals and realistic student outcomes that we could assess across both different instructional models and the entire research project. We worked as a team and with input from a panel of experts to develop multiple assessment strategies that would reliably and validly evaluate learning and achievement of the identified goals. We discuss those strategies in the next section.
Another set of assessment challenges arose during the MicroMacro project and again in our second project, MacroCITI. Results of the MicroMacro project pointed to clear benefits of integrating macro- and microethical issues in graduate ethics education. Students in instructional models in that program demonstrated gains in knowledge of ethical standards, ethical sensitivity, and ethical judgment (Canary et al. 2012). However, many institutions rely solely on CITI for ethics education for graduate students.
CITI, managed out of the University of Miami, provides online ethics education programs for multiple disciplines and multiple purposes (e.g., human subjects, responsible conduct of research). But no CITI module is dedicated to the social responsibility aspects of research. Accordingly, we used materials developed for instruction and assessment in the MicroMacro project as a springboard for developing new online instructional materials in social responsibility for CITI. This project presented us with an assessment challenge previously identified by Borenstein and colleagues (2010): the difficulty in assessing online instruction and learning. The online environment is very different from a typical interactive in-person class and response rates for online surveys are dramatically lower than in face-to-face contexts.
Additionally, brief educational experiences such as the new CITI module are not conducive to collecting pre- and posttest data, as was possible in the MicroMacro project. Other ethics programs or centers might have similar instructional contexts, such as field experiences, community engagements, or brief activities. As with the online CITI modules, such nontraditional educational contexts are associated with assessment challenges.
For ongoing research or educational programs, an important aspect of assessment is defining a clear set of objectives among participating instructors. As mentioned above, assessment needs to match the program; even if there are differences across segments of the program, common threads may be assessed uniformly.
For the MicroMacro research program, we determined that all four instructional modes (standalone course, hybrid face-to-face/online course, ethics material embedded in a science course, and lab engagement) would cover data management, conflict of interest, military research, and sustainability. These common threads represent two “micro” issues and two “macro” issues identified in early stages of our project. Content and approaches differed, but all of these areas could be assessed with a quantitative tool. Furthermore, we identified three of the four goals for ethics education outlined by Davis (1999) that would be realistic to achieve and
assess in our research program: (1) increased knowledge of relevant standards of conduct, (2) greater ethical sensitivity, and (3) improved ethical judgment.
With the four common content threads and three instructional goals we had a structure for our assessment strategies. We developed quantitative items (true/false, Likert-type scale, and multiple choice) to measure each of the three goals for each of the four content areas. Importantly, we also used existing measures of moral judgment (collaboratively agreed upon as appropriate measures of ethical judgment), a short form of the Engineering and Sciences Issues Test (ESIT; Borenstein et al. 2010), and the Moral Judgment Test (MJT; Lind 2009). Results indicated that our study-specific measure and the MJT did not capture changes in ethical judgment but the ESIT did (Canary et al. 2012).
Another approach to assess a common goal across instructional contexts is from the ASU Lincoln Ethics Teaching Fellows Program (Herkert 2011). At the end of the fellowship year, fellows were asked to consider how including ethics content in their courses related to fostering critical thinking skills in their students. Herkert, the program director, thematically analyzed responses and noted the following common themes: linking ethics and decision making, recognizing multiple views, questioning assumptions, and moving away from simple dichotomies. These answers comport with the findings of Antes and colleagues (2009) that the most effective ethics education focused on ethical decision making.
One strategy for ethics education assessment in centers and programs, then, is to include critical thinking or decision making components in the educational experiences and then evaluate the development of those skills. That could be accomplished with case studies, open-ended responses, or quantitative tools such as the ESIT.
Another way to assess changes in students’ understanding of ethics, while taking into account the specific content of different contexts in a program or center, is to tailor pre- and posttests to each context. Davis and Feinerman (2012) noted the importance of such tailoring. Pre- and posttest surveys can ask students to define, describe, or discuss the focus of the course. For instance, as an outcome of the ASU Lincoln Ethics Teaching Fellowship, Canary redesigned a Communication in Leadership course, using an ethics lens to cover issues throughout the semester rather than in discrete units. Students wrote their personal definitions of “leadership” at the beginning of the semester and again on the last day of class. A comparison of answers revealed more nuanced understandings of leadership, including its ethical dimensions, at the end of the course (Canary 2011).
We did a similar assessment in the MicroMacro project. At the beginning of each participating course and again during the last week of the semester, students were asked, “How do you view your role in society as a scientist or engineer?” Preliminary results did not identify dramatic differences between answers before and after instruction, with the exception of one new theme, “to be socially responsible in contributions,” that emerged in the posttest. However, we are conducting a more detailed analysis of language used in answers to determine whether there are more nuanced differences between time periods and instruction groups.
Another assessment strategy for centers is to conduct systematic analyses of project or annual reports from center personnel. For instance, the ASU Lincoln Ethics Teaching Fellowship required each fellow to provide a final report to the Lincoln Center. These reports included each fellow’s major activities as well as personal reflections and evaluations of future contributions to ethics education that might result from those activities. Such reports could be thematically analyzed across programs or centers to distinguish between effective and ineffective educational activities.
Simple metrics can also be useful in assessing the work of centers. The Lincoln Ethics Teaching Fellows Program, for example, uses the total number of students exposed to ethics education through new and revised courses created by the teaching fellows. Voluntary studies of ethics education commonly suffer from small numbers of student participants and low response rates (Canary et al. 2012; Davis and Feinerman 2012); this type of metric might be one way of describing potential impact of programs and centers.
There will be advantages and disadvantages to any assessment tool. No single tool can do everything, so the best idea is to incorporate a variety of assessment tools. Qualitative, open-ended questions as well as targeted, quantitative items will capture different aspects of educational experiences and development.
Additionally, certain tools or strategies are more appropriate for assessing particular goals in particular settings. For example, we determined that our goals for the MacroCITI project were limited to increasing (1) knowledge of standards and issues and (2) ethical sensitivity. Because of the national, online nature of the study we determined that those goals would be best assessed using only quantitative measures. We used some items developed in the MicroMacro project and developed new ones specific to the instructional materials created for the MacroCITI project.
As Antes and colleagues (2009) pointed out, it is also important to consider the instructional process. There are several ways to do this. For example, the MicroMacro project included open-ended questions for students to identify best and worst instructional methods used in their instructional models. We also used several existing scales that measure instructor-student communication and classroom climates (see Canary et al. 2012 for description of these measures). These strategies would be less useful for online instructional environments, of course. However, it is interesting that, in our MicroMacro project, students in the hybrid course demonstrated the highest posttest scores of all instructional groups in measures of their knowledge of relevant standards, ethical sensitivity, and ethical judgment (although the difference was statistically significant only for knowledge). Indeed, 32 percent of participants in that study indicated that using a combination of instructional methods was most effective for ethics/social responsibility instruction.
Clearly there are multiple ways to reach students, engage their thinking about ethics and social responsibility, and guide them toward improving their knowledge, sensitivity, and judgment concerning these issues.
By reviewing other published accounts of ethics education assessment and assessing ethics education in two multiyear research programs and an ethics center teaching fellows program, we have garnered experiential knowledge that we share here as practical guidance.
• Take instructional design seriously. Incorporate multiple methods of instruction and varied learning activities, as appropriate for the particular content and context.
• Consider what goals are appropriate for each instructional endeavor and clearly articulate them with center/program faculty.
• Use the content, context, and goals to determine assessment strategies and design assessment tools. For many programs/centers, this will involve the use of multiple strategies to capture multiple foci, contexts, and goals.
• When possible, make use of project workshops and contacts with experts to fine-tune assessment methods. Build such workshops and consultations into program/center budgets.
• Make use of informal assessments as well. Assessment need not always be expensive or time consuming. Informal assessments of center projects can provide valuable information to faculty and administrators.
• When appropriate, use existing resources developed in previous studies. For example:
The Engineering and Science Issues Test (Borenstein et al. 2010) measures ethical judgment in research and practice settings.
Our MicroMacro project website (www.cspo.org/projects/eese-daim/) provides quantitative measures of knowledge of relevant standards and ethical sensitivity for four content areas frequently addressed in ethics education. It also provides qualitative items to compare students’ personal reflections and evaluate the instructional process.
The research projects discussed in this paper were supported by grants from the US National Science Foundation (SES-0832944 and SES-1033111). The views expressed are those of the authors and do not necessarily represent the views of the National Science Foundation or the US government. We wish to acknowledge the contributions of our project collaborators: Karin Ellison, Jameson Wetmore, and Karen Wellner. Additional support for this work was provided by the Center for Teaching and Learning Excellence and Department of Communication, both at the University of Utah; and the School of Letters and Sciences, Lincoln Center for Applied Ethics, Consortium for Science, Policy and Outcomes, and Center for Biology and Society, all at Arizona State University.
Antes AL, Murphy ST, Waples EP, Mumford MD, Brown RP, Connelly S, Devenport LD. 2009. A metaanalysis of ethics instruction effectiveness in the sciences. Ethics and Behavior 19:379–402.
Borenstein J, Drake MJ, Kirkman R, Swann JL. 2010. The Engineering and Science Issues Test (ESIT): A discipline-specific approach to assessing moral judgment. Science and Engineering Ethics 16:387– 407.
Canary HE. 2011. Using an ethics lens for teaching communication: Focus on small group and leadership communication. Teaching Ethics 11(2):25–35.
Canary H, Herkert JR, Ellison K, Wetmore J. 2012. Microethics and macroethics in graduate education for scientists and engineers: Developing and assessing instructional models. Proceedings of the 2012 American Society for Engineering Education Annual Conference. Available online at www.cspo.org/projects/eese-daim/publications/ASEE-2012-Paper.pdf.
Davis M. 1999. Teaching ethics across the engineering curriculum. Proceedings of the Online Ethics Center International Conference on Ethics in Engineering and Computer Science. Available online at www.onlineethics.org/Education/instructessays/curriculum.aspx.
Davis M, Feinerman A. 2012. Assessing graduate student progress in engineering ethics. Science and Engineering Ethics 18:351–367.
Herkert JR. 2011. The Lincoln Teaching Fellows Program at the ASU Polytechnic Campus. Teaching Ethics 11(2):1–5.
Herkert JR, Wetmore J, Canary HE, Ellison K. 2009. Integrating microethics and macroethics in graduate science and engineering education: Developing instructional models. Proceedings of the 2009 American Society for Engineering Education Annual Conference. Available online at www.cspo.org/projects/eese-daim/publications/ASEE-2009-paper.pdf.
Lind G. 2009. The Moral Judgment Test. Information is available online at www.uni-konstanz.de/agmoral/mut/mjt-engl.htm.