Developing Metrics for Assessing Engineering Instruction: What Gets Measured Is What Gets Improved (2009)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

1 Background, Framing, and Concepts In this report, we present a case for using metrics to evaluate teaching and other kinds of instruction in engineering, based on the premise that the quality of their teaching should be a factor in decisions about faculty appointments, promotions, and advancement. This will be possible only if metrics for evaluations are widely available, easy to use, and recognized and respected by all. Metrics for evaluating teaching and learning are also important for ensuring that students receive the best possible education. In addition, meaningful evaluations provide all faculty members with opportunities for continuing their professional development and improving their teaching knowledge and skills. Practical factors also support the need for evaluating teaching and learning. State budgets for higher education have become increasingly constrained and subject to competition from other pressing needs. This long-term, sustained trend has led to more rigorous scrutiny of public institutions of higher education by state governments and agencies, as well as agencies at the federal level, as reflected in A Test of Leadership: Charting the Future of U.S. Higher Education, more frequently called the “Spellings Commission Report” (United States Department of Education, 2006). A natural result of this oversight movement is a growing demand for accountability in higher education (i.e., the ability to show that public dollars are being used wisely and are engendering tangible, positive results). The primary area of accountability is teaching, or more correctly learning. Do students receive an educational benefit that justifies the dollars they and their families spend, and does society receive an appropriate benefit in return for the public dollars spent on education? The capability to document the quality of teaching and learning is a necessary part of institutional accountability, and the documentation methods used must be understandable, transparent, and cogent. The ability of faculty to promote student learning is a natural consequence of their preparation for their role as teachers. In the past few decades, some institutions have established future-faculty programs for graduate students and/or teaching-resource centers for current faculty. In addition, the National Science Foundation (NSF) and other groups and organizations have supported numerous projects and centers focused on teaching and learning. A listing of some of the various centers and coalitions that are concerned with engineering and are located throughout the United States is given in Figure 1.1 (Atman, 2007). Although these programs are usually voluntary and their success varies, faculty-development centers have generated moderate to high interest among engineering educators (Van Note and Szabo, 1996). 4 

Figure 1.1 Engineering Education Organizations and Centers (adapted from Atman, 2007). To a large extent, however, graduate students who plan to pursue academic careers receive little or no supervised instruction in teaching. Thus new faculty members must usually develop their own approaches and styles of teaching. The application of metrics and institutional resources for evaluating progress would make it much easier for all faculty members to develop and improve their teaching skills continually. Additionally, a formal evaluation of teaching effectiveness would likely benefit graduate students’ exposure to teaching and learning concerns as well as their preparation for future teaching responsibilities. Another reason for evaluating teaching and learning is that the demands on practicing engineers are changing, and the system for engineering education must necessarily change with those demands. In this fast-moving environment, it is important to assess how teaching is changing and whether the changes are effective.1 DRIVERS OF CHANGE IN ENGINEERING EDUCATION The need for effective evaluation of teaching is an ongoing one; however, there are a number of recent developments that impact upon engineering education and make evaluation all 1 Concerns about the quality and efficacy of higher education have elicited a variety of responses. Two prominent examples are a workshop in the Association of American Medical Schools, Advancing Educators and Education: Defining the Components and Evidence of Educational Scholarship (2007). Washington, D.C.: AAMC, and a review of indicators of quality teaching and learning by the Carrick Institute (Australia) in Chalmers. D., A Review of Australian and International Quality Systems and Indicators of Learning and Teaching (2007). Chippendale, NSW: The Carrick Institute for Learning and Teaching in Higher Education, Ltd. [Note: The Carrick Institute is now known as the Australian Learning and Teaching Center.] 5 

the more critical. The rapid development of high-bandwidth technologies has enabled instantaneous communication, as well as rapid access to and transmission of information. Industry, business, research, and education are now all global activities. In addition, engineering projects and practices today are intertwined with public issues and policies, such as energy, the environment, health care, and government. For reasons of both globalization and public interaction, engineers must have an understanding of people with different backgrounds and different cultural values and must be able to interact with them effectively. Thus engineers must be more broadly educated than in the past, and they must be able to understand the wider context and effects of their work. Successful engineers in the global workplace need much more than technical knowledge and skills.2 ABET, the engineering accreditation agency, has also acknowledged these changes in the engineering environment. Criterion 3, Sections a-k, was recently expanded to include elements of increased breadth with respect to knowledge regarding environmental and economic assessment, and Criterion 5 now calls for “a general education component that complements the technical content of the curriculum and is consistent with the program and institution objectives”(ABET, 2008). In addition to the changes brought about by globalization and the increasingly public nature of engineering, there are other drivers related to the rapid increase in new knowledge, the changing characteristics and capabilities of students, and the utilization of engineers in the workplace. • The knowledge base is increasing exponentially, and with new technologies, this knowledge is readily retrievable through digital libraries and the Internet. We have long since passed the point at which all of the knowledge necessary for a career in engineering can be packed into the undergraduate engineering curriculum. Students today must be taught to be lifelong learners able to think critically and be adept at locating, evaluating, and assimilating information from many sources. • Students entering college today have grown up in an information society enriched by technological capabilities, an environment in which instant communication, information searching, and multi-tasking are routine.3 Engineering educators must recognize these capabilities, make use of them to advance learning, and build upon them when appropriate. • A vast majority of engineering graduates, 94 percent by some estimates, pursue careers in engineering practice rather than in academia (NSF, 2002). However, educators at research universities have a natural tendency to emphasize fundamental knowledge and research methodologies in their courses rather than practical experience and engineering practice. To meet the needs of today’s students, the authoring committee believes that increased focus should be given to disciplinary aspects of engineering practice and to 2 These issues are discussed in detail in two recent reports from the National Academy of Engineering, The Engineer of 2020: Visions of Engineering in the New Century (2004) and Educating the Engineer of 2020: Adapting Engineering Education to the New Century (2005) (Washington, D.C.: The National Academies Press), and a report by the University of Michigan Millennium Project, Engineering for a Changing World. 3 In some studies, multi-tasking has been shown to interfere with the learning of abstract and complicated material. (Foerde, K., Knowlton, B., and Poldrack, R. (2006). Modulation of competing memory systems by distraction, in Proceedings of the National Academy of Sciences, vol. 103 no. 31, 11778-11783. Washington, DC.) 6 

teaching these aspects explicitly. One of the conclusions of Educating the Engineer of 2020 stated that we need to put more emphasis upon engineering practice in university engineering education programs, and movements in this direction will impact on both who is teaching as well as how we teach (NAE, 2005). IMPROVING METHODS OF EVALUATION Most college faculty members have had little or no formal training in the complex and intellectually sophisticated skills necessary for designing and delivering instruction or in assessing student learning outcomes (Felder, 1993; Brint, 2008), Therefore, effective teaching requires that educators adapt, develop, and hone their teaching skills to increase the level of student learning, and institutions must provide the resources to help them to develop, support and improve their teaching performance—especially if teaching performance is to be evaluated. The US Department of Education, has emphasized the importance of assessing learning outcomes as part of the accreditation process (which mirrors the intent of changes to engineering accreditation efforts in the ABET EC2000 criteria). This emphasis has been expressed in a number of accreditation standards including standards that refer specifically to faculty evaluation (see, for example, Comprehensive Standard 3.7.2 of the Southern Association of Colleges and Schools accrediting agency). Thus, as the accreditation standards for higher education begin to focus more on learning outcomes and the assessment procedure used to ensure quality teaching, the development and use of a consistent set of quality metrics for assessing faculty teaching becomes increasingly important in all aspects of higher education. An informative resource for efforts to identify the component parts of teaching that should be included in a matrix of skills and attributes is the National Research Council report called Evaluating and Improving Undergraduate Teaching in Science, Technology and Mathematics. The report provides an excellent overview of the area of teaching effectiveness (NRC, 2003). Practical realities must be taken into consideration in the development of metrics for faculty evaluations. University faculty members have very full schedules and many commitments inside and outside the classroom. Because change requires an investment of time, there is a strong tendency among them to maintain the status quo. Thus the methods and metrics for evaluating teaching and learning must be efficient in terms of how much time they require, and they must be user friendly. Current assessment methods are heavily dependent on student ratings, which may provide only a single dimension of the classroom experience. Effective metrics should also include diverse and complementary methods. Information technology can support the documentation of faculty activities that reflect their growth and development with respect to improved teaching and learning efforts. As professionals, most faculty members value self-governance and self-policing. Therefore, for change to be effective, they must “buy in” to the need for a systematic approach for measuring teaching effectiveness. Buy-in is more likely if faculty members have the opportunity to participate in the identification and implementation of methodologies through, for example, academic senates and other activities. They should also be involved in determining how assessments by other faculty members will be used in making decisions and evaluating their skills. 7 

Finally, the leaders of a profession define the values and priorities of that profession, and their support carries considerable weight. The National Academy of Engineering (NAE), engineering societies, and other stakeholders can provide valuable support by emphasizing the benefits of evaluating teaching and learning. Leaders of NAE and other societies should put forward statements indicating the need for quality teaching that can transform students from vessels of knowledge into sophisticated seekers and users of knowledge. 8