National Academies Press: OpenBook
« Previous: 4 Evaluating Teaching in Science, Technology, Engineering, and Mathematics: Principles and Research Findings
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

PART II
Applying What Is Known: Strategies for Evaluating Teaching Effectiveness

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
This page in the original is blank.
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

5
Evaluation Methodologies

Part I of this report describes recent research on ways to rethink and restructure teaching and learning, coupled with new approaches to evaluation and professional development for faculty. Those findings have the potential to reshape undergraduate education in science, technology, engineering, and mathematics (STEM) for a much larger number of undergraduates. However, developing strategies for implementing and sustaining such changes requires the commitment of all members of a college or university community.

In a teaching and learning community, the most effective evaluation is that which encourages and rewards effective teaching practices on the basis of student learning outcomes (Doherty et al., 2002; Shapiro and Levine, 1999). Assessment of student learning at its best enables students to identify their own strengths and weaknesses and to determine the kinds of information they need to correct their learning deficiencies and misconceptions. When such evaluation is properly employed, students learn that they can engage in self-assessment and continuous improvement of performance throughout their lives.

Accordingly, this chapter offers practical guidance to postsecondary faculty and administrators on ways to institute a system of both evaluation and professional development that can contribute to significant gains in teaching effectiveness for faculty who teach undergraduates. The chapter describes how input from students (undergraduates and graduate teaching assistants), colleagues, and faculty self-evaluation can be used for evaluating individual instructors. It also describes the advantages and disadvantages of these various approaches.

As stated in Chapter 1, ongoing formative assessment of student learn-

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

ing can have powerful benefits both in improving learning and in helping faculty improve their teaching on the basis of the feedback they receive from a variety of sources. The information gathered during such assessments also can serve as a basis for more formal, summative evaluations that have an impact on important personnel decisions.

The technique of outcomes assessment as a means of measuring student learning and the use of that information to improve teaching are considered first. Additional strategies and methods for formative evaluation follow. The chapter concludes with a series of suggestions for improving summative evaluation of faculty. The committee emphasizes that the approaches described in this chapter are but a sampling of the techniques that appear in the research literature on improving the evaluation of teaching and student learning. They are

Assessment Is More Than Grades

To many, the word “assessment” simply means the process by which we assign students grades. Assessment is much more than this, however. Assessment is a mechanism for providing instructors with data for improving their teaching methods and for guiding and motivating students to be actively involved in their own learning. As such, assessment provides important feedback to both instructors and students.

Assessment Is Feedback for Both Instructors and Students

Assessment gives us essential information about what our students are learning and about the extent to which we are meeting our teaching goals. But the true power of assessment comes in also using it to give feedback to our students. Improving the quality of learning in our courses involves not just determining to what extent students have mastered course content at the end of the course; improving the quality of learning also involves determining to what extent students are mastering content throughout the course.

SOURCE: Excerpted from National Institute for Science Education (2001b).

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

included here on the basis of the committee’s analysis of the research literature and the expertise of individual committee members, and with the expectation that each institution will adapt or modify these approaches according to its individual needs.

IMPROVING TEACHING BY EXAMINING STUDENT LEARNING: OUTCOME ASSESSMENT

One approach to improving student learning is outcome assessment—the process of providing credible evidence that an instructor’s objectives have been obtained. Outcome assessment enables faculty to determine what students know and can do as a result of instruction in a course module, an entire course, or a sequence of courses. This information can be used to indicate to students how successfully they have mastered the course content they are expected to assimilate. It can also be used to provide faculty and academic departments with guidance for improving instruction, course content, and curricular structure. Moreover, faculty and institutions can use secondary analysis of individual outcome assessments to demonstrate to prospective students, parents, college administrators, employers, accreditation bodies, and legislators that a program of study produces competent graduates (Banta, 2000).

Outcome Assessment Activities

Faculty members, both individually and as colleagues examining their department’s education programs, have found the following activities helpful when undertaking outcome assessment:

  • Developing expected student learning outcomes for an individual course of study, including laboratory skills.

  • Determining the point in a student’s education (e.g., courses, laboratories, and internships) at which he/she should develop the specified knowledge and skills.

  • Incorporating the specified learning outcomes in statements of objectives for the appropriate courses and experiences.

  • Selecting or developing appropriate assessment strategies to test student learning of the specified knowledge and skills.

  • Using the results from assessment to provide formative feedback to individual students and to improve curriculum and instruction.

  • Adjusting expected learning outcomes if appropriate and assessing learning again. Such a process can lead to continual improvement of curriculum and instruction.

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

Faculty in STEM are challenged in their teaching by a set of circumstances that most faculty in other disciplines do not encounter, such as designing laboratory and field components of courses, incorporating modern technology into courses, or supervising students involved with original research (see Chapter 2 for additional detail). However, faculty in these disciplines also have an array of assessment methodologies from which to choose that address particular learning outcomes (e.g., see Doherty et al., 2002). Student responses in each of the following formats can first be studied for the information they provide about individual student learning and performance, and then compared across students and classes for clues about the strengths and weaknesses of curriculum and instruction:

  • Classroom quizzes and exams

  • Projects

  • Poster presentations of library or laboratory research

  • Cooperative experiences

  • Portfolios (collections of work)

  • Standardized tests both within and across disciplines

  • Student journals

  • Questionnaires

  • Interviews

  • Focus groups

Scoring of Outcome Assessments: Primary Trait Analysis

Increasingly, primary trait analysis (Lloyd-Jones, 1977) is being used as a scoring mechanism in outcome assessment (Walvoord and Anderson, 1998). Primary trait analysis is a technique whereby faculty members consider an assignment or test and decide what traits or characteristics of student performance are most important in the exercise. They then develop a scoring rubric (Freedman, 1994) for these traits and use it to score each student’s performance.

For example, Emert and Parish (1996) developed multiple-choice and short-answer tests for undergraduate students enrolled in courses in algebra, discrete mathematics, and statistics. Students were asked to submit supporting work to provide additional insight into their thought processes and the extent to which they had developed an understanding of mathematical concepts. Emert and Parish developed the following scoring rubric to assess performance on each item their students provided:

Score

Criterion

3

Conceptual understanding apparent; consistent notation, with only an occasional error; logical formulation; complete

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

 

or near-complete solution/ response

2  

Conceptual understanding only adequate; careless mathematical errors present (for example, algebra, arithmetic); some logical steps lacking; incomplete solution/response

1

Conceptual understanding not adequate; procedural errors; logical or relational steps missing; poor or no response to the question posed

0

Does not attempt problem, or conceptual understanding totally lacking

By studying the aggregate scores for each item, Emert and Parish and their colleagues discovered that students missed most items because they lacked the conceptual understanding to address the problem appropriately (as opposed to making careless errors). By inspecting the items missed by large numbers of students, faculty discovered which concepts needed to be addressed through instruction again, perhaps in alternative ways. Understanding such misconceptions by students can provide instructors with valuable insights into how they might adjust their teaching techniques or emphases to address these kinds of problems (see, e.g., National Research Council [NRC], 1997a, 1999b).

Benefits of Outcome Assessment

It can be difficult and time-consuming for faculty to redesign course objectives to focus on student learning outcomes, to agree with colleagues on comprehensive learning outcomes for the entire curriculum, and to select or develop appropriate assessment tools. It can be equally or more difficult for faculty to adopt a routine of systematically collecting and studying assessment data and then making improvements based on that feedback. However, some examples of positive, multidimensional change have been documented from departments that have taken assessment seriously. These departments update curricula continuously. They develop new courses and phase out others as needs change. And they can document improvement in student learning (Wergin, 1995; Wergin and Swingen, 2000).

Other changes that have been prompted by outcome assessment include faculty employing more active learning strategies that enable students to practice the concepts they are learning in class. Alumni and employers are being asked to comment on curriculum and instruction and even to serve as evaluators of teaching and learning. For example, at Virginia Polytechnic Institute and State University, the Department of Civil Engineering created an alumni advisory board and asked its

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

members to debrief a group of juniors and seniors regarding the department’s curriculum. The students discussed such issues as overcrowding due to space limitations. In response, the soil mechanics laboratory was expanded through privately sponsored renovation. In addition, students’ concerns about opportunities to learn to use the latest software led to the development of a new computer laboratory. And a perceived need for improved communication skills encouraged faculty to develop new writing-intensive courses and introduce them into the civil engineering curriculum (Banta et al., 1996).

Outcome assessment can be difficult to implement because it requires that faculty reorient their course and curriculum objectives to focus on what students learn rather than what faculty teach. Nonetheless, the committee has concluded that outcome assessment can be an important approach to emphasizing and focusing on what and how students learn.

OTHER STRATEGIES AND METHODS FOR FORMATIVE EVALUATION

Formative Evaluation by Undergraduate Students

Research has shown that the best way to improve teaching is to provide individual faculty members, particularly in their first years of teaching, with ongoing individualized formative feedback from students and colleagues (Brinko, 1993; Cambridge, 1996; Centra, 1993; Hutchings, 1996). Instructors are best served by informal evaluation activities that take place throughout a course, especially when coupled with consultations with learning experts.1 Such informal activities can help instructors identify what is working and what needs to be improved while the course is still in progress.

For example, helpful and regular feedback from students allows midcourse corrections in such areas as organization, methods of teaching, and the introduction or modification of activities designed to enhance learning. Many institutions have already recognized the benefits of such midcourse corrections and offer faculty guidance and appropriate forms for conducting various levels of student surveys (see Appendix B). The National Institute for Science Education (NISE) provides a “Student Assessment of Learning Gains” website where faculty can use and modify questionnaires designed to

1  

In contrast, Marsh and Roche (1993) report that feedback gathered at the end of a course had significantly greater long-term impact on the improvement of teaching than midcourse evaluations.

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

offer both formative and summative feedback from their students about how various elements of their courses are helping the students learn. This innovative website also allows students to complete the survey form on line and provides instructors with a statistical analysis of the students’ responses.2

The results of studies on formative evaluations of student learning indicate that the techniques described below require modest effort, are easy to carry out, and consume very little class time. In addition, faculty can obtain regular feedback from their students through the use of course listservs, electronic mail, or a website for student feedback connected to a course’s website.

Repeated Measurements of Student Learning and Teaching Effectiveness

The typical end-of-course student evaluation form is an indirect assessment tool that can help an instructor understand what worked to assist learning in a course and what did not. Instructors may feel that students’ scores on final examinations in their courses provide a valid measure of student learning and that this measure can also be used to assess their effectiveness as a teacher summatively. However, many factors other than the instructor’s teaching competence can affect examination results, including prior knowledge; students’ preconceptions; and their ability, interest, and skills in the subject area (Centra, 1993).

Another factor is student effort. Even the most effective teachers can do only so much to motivate students. Although most college teachers try to motivate students to learn, in the end students must take responsibility for their own learning and academic achievement. For the past three years, the Carnegie Foundation for the Advancement of Teaching and Pew Forum on Undergraduate Learning (2002) have published annually the National Survey of Student Engagement: The College Student Report. Each of these reports is compiled from responses to a questionnaire whose respondents consist of thousands of first-year and senior undergraduates at 4-year colleges and universities.3 The students are asked about the extent to which they participate in classroom and campus activities shown by research studies to be important to learning. Questions from the

2  

Additional information and links to the survey forms are available at <http://www.wcer.wisc.edu/salgains/instructor/>.

3  

The list of institutions that participated in this project is available at <http://www.indiana.edu/~nsse/>.

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

2001 survey instrument are provided in Appendix B.4 This instrument and its parent, the College Student Experiences Questionnaire (Indiana University, 2000), can provide important information about the quality of effort students are committing to their work.

If a teaching evaluation form is distributed only at the end of a course, it cannot help the instructor make useful modifications for students who are currently enrolled. A better way to assess student learning and teaching effectiveness is to test students at the beginning and then again at the end of a course and inspect the “gain scores.” An instructor’s willingness and ability to use gain scores to improve a course may be considered favorably during a summative evaluation of teaching. At the same time, gain scores are easily misinterpreted and manipulated and may not be statistically reliable (both pre- and post-tests are characterized by unreliability that is compounded when the two are used together). Therefore, they should not be used exclusively to examine student learning for purposes of summative evaluation.

Another indirect measure of student learning that some faculty have found particularly useful is a questionnaire that lists the learning outcomes for a course or series of courses. Students may be asked to indicate how much the course or the entire curriculum increased their knowledge and skills in the specified areas.

For maximum usefulness, teachers may want to add their own course-related items to student evaluation forms, as well as encourage written or oral communication from students, including computer-assisted feedback. Evaluations of laboratory, field, and extra clinical or discussion sections require special questions, as do evaluations of student advising (NISE, 2001a).

Direct Questioning of Students

The easiest way to find out whether students understand what is being said is to ask them directly. But unless instructors have developed sufficient rapport and mutual respect among the students in their class, they should avoid questions or situations that could make it awkward for students to respond (“Who is lost?”) or are so generic as to lead to nonresponses (“Are there any questions?”). Instead, instructors should pose questions that encourage more specific responses, (e.g., “How many of you are understanding what we are talking about?”). Various forms of information technology, such as in-class response keypads, can facilitate asking such questions, allowing students to

4  

The survey instruments for both 2000 and 2001 are also available at <http://www.indiana.edu/~nsse/>.

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

answer without fearing that they will be singled out or ridiculed by their peers if they indicate their lack of understanding.

Even better, instructors can ask students to paraphrase briefly the key points or essence of a discussion or lecture. At the end of a class session, students can be asked individually or in pairs to write a brief summary of the main ideas presented and submit it to the instructor (anonymously). If this method is used, students should clearly understand that the written summary is not a quiz and will not be graded.

Minute Papers and Just-in-Time Teaching

At the end of a class, instructors can ask students to write for a minute or two on one of the following kinds of questions: “What is the most significant thing you’ve learned today?” “What points are still not clear?” or “What question is uppermost in your mind at the end of today’s class?” Responses can help instructors evaluate how well students are learning the material. Student responses to the second and third questions also can help instructors select and structure topics for the next class meeting. Large numbers of such short papers can be read quickly, and a review of unclear concepts can take place at the next class meeting (Angelo and Cross, 1993; Schwartz, 1983).

A similar approach, developed by the physics education community, is “just-in-time” teaching (Dougherty, 1999). Students are asked to respond to one or two short questions posed by the instructor the day before a subject is to be taught. They submit their responses via e-mail or to a website. These responses give the instructor a good idea of what the students do and do not understand about the concepts to be considered. The instructor can then adjust the amount of time spent on explaining the concepts, working through problems, or providing examples that will help the students learn and understand the concepts.

Student Teams

Another documented approach involves asking a team of students to work throughout the term on continuous course evaluation (Baugher, 1992; Greene, 2000; Wright et al., 1998). The team members are encouraged to administer questionnaires and interview their peers about how the instructor is or is not promoting learning.

For larger classes, a liaison committee of two to four students can be established that meets periodically with the instructor to discuss difficulties or dissatisfactions. Membership on the committee can be rotated from a list of volunteers as long as the entire class knows who the liaisons are at any given

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

time. Alternatively, students who are not enrolled in a course can be hired to attend the class and offer ongoing feedback to the instructor (e.g., Greene, 2000).

Students’ Course Notes

With students’ permission, instructors can ask to borrow a set of notes. This technique allows teachers to see what students consider to be the main points presented and whether there is misinformation or confusion about various topics. Alternatively, to ensure student anonymity, students can be asked to photocopy selected portions of their notes and submit them to the instructor without identifying information (Davis, 1993).

Chain Notes

In small classes, it may be possible to pass around a piece of paper midway through a session and ask students to jot down the main point of what is being discussed at that moment. The instructor then has a listing of what students consider to be the key concepts discussed in that class period, which can be used (Angelo and Cross, 1993).

Student Study Groups

Students can be encouraged to form small study groups and to send representatives to discuss any difficulties or questions with the instructor. Study groups provide students with opportunities to learn from one another, and a group may find it easier to seek assistance from the instructor. In turn, having group representatives rather than individual students approach the instructor can reduce the amount of time required to answer repetitive questions, especially in larger classes.

Informal Conversations

Instructors can seek feedback through informal conversations with students during office hours, before or after class, or through e-mail. They can ask students about what has been working well or what is problematic. Instructors should not pose these questions to students in ways or at times that might force them to answer quickly. Questions should be directed to those students the teacher thinks would be most likely to respond candidly. Whenever this kind of feedback is solicited, instructors should keep in mind that such evidence is anecdotal and may not be representative of the entire class. However, informal responses from individual students can serve as the basis for index card questions to the entire class (discussed next). Asking such questions based on informal conversations with students can also reinforce the message that the instructor is listening to students and takes input from them seriously.

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Index Cards

Several times during the term, an instructor can pass out index cards to students and ask them to respond to two questions, one on the front of the card, the other on the back. General questions can be posed, such as “What are your overall impressions of the course?” “What’s good about the course?” “Do you have any suggestions for changing the course?” or “Are there any problems?” Alternatively, the instructor can ask more specific questions about aspects of the course, such as “Are the problem sets too difficult?” or “Is the laboratory section well connected to other aspects of the course?” Providing prompts (such as “I would like you to do more…” or “I would like you to do less …”) and asking students to complete the sentence is another useful technique (Davis, 1993).

Outside Evaluators

Midway through the term, an instructor can invite an instructional improvement specialist from the campus-wide or discipline-based teaching and learning center or a departmental colleague to conduct an oral evaluation with his or her students. At the beginning of the class, the teacher introduces the guest evaluator and then leaves the room for 20 minutes. During that time, the evaluator asks students to cluster into groups of five or six and take 10 minutes to (1) select a spokesperson who will write down the group’s comments, (2) name something in the course they find very helpful or worthwhile, (3) name something they would like to see changed, and (4) suggest how the course could be improved. After the groups have completed their work, the evaluator asks the spokesperson from each group to report. The evaluator summarizes the points of consensus for the entire class and also clarifies points of disagreement. The evaluator then provides an oral or written summary for the instructor (Clark and Redmond, 1982).

Small Group Instruction Diagnosis5

This technique (also known by its abbreviation, SGID) originated at the University of Washington and is now promoted by teaching and learning centers on a variety of types of cam-

5  

The description of small group instruction diagnosis presented here is based on information taken from the websites of several campus centers for teaching and learning. A more detailed description of this approach, along with links to other websites and resources on the subject, is available from a website at Miracosta Community College, <http://www.miracosta.cc.ca.us/home/gfloren/sgid.htm>. Small Group Instructional Diagnosis, an online journal from the National Teaching and Learning Forum that publishes research on the uses of the method, is available at <http://www.ntlf.com/html/pi/9705/sgid.htm>.

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

puses. The goal of SGID is to align expectations for improving teaching and learning. Consultants are employed to gather information directly from students and instructors. The technique provides feedback to instructors, including suggestions for strengthening their courses, and generally increases communication between students and instructors. The consultative process takes anywhere from 15 to 60 minutes and is most effective when conducted near midsemester so the faculty member will have sufficient time to amend the course.

Classroom interviews involve the consultant interviewing students, in the instructor’s absence, to identify course strengths and areas for change. The consultant summarizes this information and meets with the instructor to discuss students’ perceptions and pedagogical options. Research at the University of Washington on the use of class interviews indicates that students appreciate the opportunity to provide feedback to their instructor before the end of the quarter.

Response to Students’ Concerns

It is important that the issues posed in a midsemester evaluation be ones to which the instructor will be able to respond during the term. Otherwise, students may develop false expectations about the remainder of the course. Instructors should emphasize to students that they would like to receive candid, constructive responses that will help them improve the course.

It also is important for the instructor to respond quickly and candidly to students’ comments and concerns. Davis (1993) discusses strategies for responding to student feedback. She sorts student suggestions for improvement into three categories: (1) improvements that can be instituted immediately during the current semester (e.g., the turnaround time on grading homework assignments); (2) those that must wait until the next time the course is offered (e.g., the textbook or readings assigned); and (3) those that the instructor either cannot or, for pedagogical or curricular reasons, will not change (e.g., the number of tests, specific content).

At the class meeting after an evaluation exercise, the instructor should thank students for their comments and clarify any confusion or misunderstandings noted in those comments about the instructor’s goals or the students’ expectations. The instructor should then indicate which suggestions would be implemented this term, those that must wait until the course is next offered, and those on which action cannot or will not be taken. In the third case, it would be helpful to explain briefly the reasons for this decision.

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Formative Evaluation by Graduate Teaching Assistants

Teaching assistants can be an invaluable source of feedback for faculty members about successes and problems that are occurring in classes, discussion sections, and teaching laboratories. Such feedback can be especially illuminating if teaching assistants are encouraged to attend class sessions regularly and to meet with the faculty member in charge of the course and with each other. Ways in which teaching assistants can provide appropriate feedback to individual faculty and to their academic department include the following:

  • Encouraging teaching assistants to provide information throughout the term about the difficulties students may be having in the courses with which the teaching assistants are involved. Through conversations with and direct observation of students in the course, teaching assistants can tell an instructor what aspects of the course readings, assignments, and presentations are causing problems for students. Such information is more likely to be offered if instructors make it clear that identifying students’ difficulties is a normal and expected part of a teaching assistant’s responsibilities. Some faculty ask teaching assistants to give them brief weekly reports on the one or two things that cause students the most difficulty.

  • Asking teaching assistants to review examinations and quizzes before they are given to students. Having participated in the course, teaching assistants can identify ambiguous or unclear exam items before the tests are administered. After midterms or quizzes have been graded, teaching assistants can provide detailed information about patterns of error or misunderstanding. Collecting this kind of information from a number of teaching assistants from different courses, from sections within a course, and over an extended period of time can also enable departments to determine which concepts need to be reinforced in several courses or which misconceptions persist as students advance through the curriculum.

  • Soliciting from teaching assistants constructive suggestions on aspects of a course or the department’s programs, such as websites, laboratory offerings, and similarities and differences in approaches to teaching and assessing student learning in different sections of the same course.

Formative Evaluation by Faculty Colleagues

Traditionally, faculty members have willingly and candidly judged their colleagues’ scholarly work through a variety of means (see Chapter 3) but have hesitated when asked to judge

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

their colleagues’ teaching effectiveness. Yet many senior faculty have the background and perspective needed to provide judgments about such matters as the candidate’s knowledge of the subject, course content, appropriateness of course objectives and instructional materials, examination skills, testing proficiency, and breadth and depth of student learning. Under the right circumstances, these judgments can be used to assist in summative evaluations of faculty (see also Chapter 4). Similar judgments from colleagues also can be useful in formative evaluations for professional development of faculty.

At small institutions or in very small departments, a lack of resources or limited numbers of faculty may make faculty input more difficult to obtain than in larger institutions or departments. In addition, friendships or rivalries that arise within any department may be amplified in smaller departments. In such cases, balanced and objective evaluations of teaching colleagues may be achieved only by including in the evaluation process additional faculty from outside the academic unit of the person being evaluated. Even when these issues do not surface, engaging faculty from outside the department, particularly those who are knowledgeable about effective pedagogies for promoting student learning, should enable review of such critical aspects of teaching as course organization, teaching methods, and the instructor’s choice of appropriate assessment practices.

Observation

Instructors who are being evaluated can ask a mentor, colleague, or instructional improvement specialist at the campus or discipline-based teaching and learning center to visit their classes and provide feedback on their teaching. Prior to each visit, instructors can discuss with observers the specific classroom issues or techniques on which the observers should focus (e.g., student-teacher interaction, the nature of questions posed, use of class time, and other issues important to the instructor).

Faculty also can ask colleagues, particularly those known to be excellent teachers, for permission to visit their courses. Visitors can note the specific techniques used by the colleague in leading discussions, conducting teaching laboratories, and so on. If time permits after class, the observing and observed faculty members can discuss their respective teaching philosophies, goals, instructional methods, out-of-class preparation, and similar matters. It is usually most helpful for a faculty member to attend a series of classes (say, all classes dealing with a specific topic or issue) to obtain a broad per-

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

spective on the range of pedagogical approaches used by the colleague in his or her teaching.

Role of Colleagues in “Formal” Formative Evaluation

Informal discussions and efforts to improve instruction among faculty members take place daily, but some departments and institutions employ more systematic and formal efforts to assist in the improvement of teaching through formative evaluation. In addition to the evaluation questionnaires reprinted in Appendix C, the following approaches to formative evaluation can be especially useful for the purposes of faculty professional development.

Faculty mentoring faculty. Increasingly, departments are assigning senior faculty as mentors to untenured faculty. Boice (1992) found that it was not necessary for successful mentors to be from the same department. Whether from within or outside of the faculty member’s department, the ideal faculty mentor appears to play four major roles: friend, source of information, and career and intellectual guide (NRC, 1997b; Sands et al., 1991).

At a variety of higher education institutions, Katz and Henry (1988) developed a strategy of transdisciplinary mentoring based on faculty working together to understand both how students learn and how to improve their teaching. Referred to as the Master Faculty Program, this initiative involves faculty working together in pairs or in triads. Faculty members observe each other’s classes and interview each other’s students several times during the semester. Interviewers’ questions emphasize student learning in the course (for example, topics that may be difficult or reactions to specific class sessions). With these observations in hand, the faculty participating in the program meet periodically to discuss candidly, and confidentially, how each participant has or has not fostered student learning. Chandler (1991) has documented the generally positive results of this type of program involving some 300 faculty at 21 different colleges and universities.

Formative evaluation by faculty colleagues from other institutions. Faculty at higher education institutions across the country and around the world can provide formative evaluation to colleagues via the Internet. They can comment on the content of a faculty member’s websites for courses, old examination questions, assignments, and student responses to questions posed by the faculty member. This kind of input from colleagues at other institutions could be included as part of a teaching portfolio or dossier for

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

summative evaluation, but also has great potential for ongoing formative feedback.

Projects of the American Association for Higher Education. The American Association for Higher Education (AAHE) has promoted collaboration in assessing and improving teaching through a variety of projects. One such project, conducted in the mid-1990s, involved 12 universities and stressed peer review as a means of formative evaluation. In this project, participants monitored their progress in improving student learning. AAHE’s (1993) Making Teaching Community Property: A Menu for Peer Collaboration and Peer Review provides many other examples of peer review efforts that contribute to formative evaluation and improved professional development in teaching for faculty.

More recently, AAHE, the Carnegie Foundation for the Advancement of Teaching, and the Carnegie Academy for the Scholarship of Teaching and Learning jointly developed a program for peer collaboration based on ideas and criteria advanced by Boyer (1990) and Glassick and colleagues (1997). The goals of the program are to support the development of a scholarship of teaching and learning that will foster significant, long-lasting learning for all students. The program also seeks to enhance the practice and profession of teaching and bring to the scholarship of teaching the same kinds of recognition and reward afforded for other forms of scholarly work (Hutchings, 2000).6 Examples of the criteria being advanced for evaluating a faculty member’s scholarship in teaching are presented in Box 5-1, excerpted from Glassick et al. (1997, p. 36). Centra (2001) has extended these criteria to allow for evaluation of the scholarship of teaching and learning as practiced by academic departments and institutions (see Box 5-2).

Self-Evaluation

Self-reports and self-reflections on an instructor’s teaching and promotion of student learning can be important sources of information for evaluating a teacher’s effectiveness (Hutchings, 1998). These self-reports, which may be part of a required annual report or a teaching portfolio, are more useful and appropriate for formative or professional development purposes than for summative personnel decisions. Faculty who have not previously performed self-evaluation may require assistance from teaching and learning centers.

As a summary of a professor’s major teaching accomplishments and

6  

Additional information about this program is available at <http://www.carnegiefoundation.org/CASTL/index.htm >.

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

Box 5-1. Evaluating the Scholarship of Teaching

Clear Goals: Does the scholar state the basic purposes of his or her work clearly? Does the scholar define objectives that are realistic and achievable? Does the scholar identify important questions in the field?

Adequate Preparation: Does the scholar show an understanding of existing scholarship in the field? Does the scholar bring the necessary skills to his or her work? Does the scholar bring together the resources necessary to move the project forward?

Appropriate Methods: Does the scholar use methods appropriate to the goals? Does the scholar apply effectively the methods selected? Does the scholar modify procedures in response to changing circumstances?

Significant Results: Does the scholar achieve the goals? Does the scholar’s work add consequentially to the field? Does the scholar’s work open additional areas for further exploration?

Effective Presentation: Does the scholar use a suitable style and effective organization to present his or her work? Does the scholar use appropriate forums for communicating work to its intended audiences? Does the scholar present his or her message with clarity and integrity?

Reflective Critique: Does the scholar critically evaluate his or her own work? Does the scholar bring an appropriate breadth of evidence to his or her critique? Does the scholar use evaluation to improve the quality of future work?

SOURCE: Glassick et al. (1997, p. 36).

strengths (Shore et al., 1986), the teaching portfolio may include the following kinds of evidence of teaching effectiveness:

  • Development of new courses

  • Products of good teaching (for example, student workbooks or logs, student pre- and post-examination results, graded student essays)

  • Material developed by the individual (course and curriculum development materials, syllabi, descriptions of how various materials were used in teaching, innovations the instructor has attempted and an evaluation of their success, videotapes of teaching)

  • Material or assessments from others (student work and evaluations, input from colleagues or alumni)

  • Descriptions of how the individual has remained current in the field, such as using knowledge gained from attending professional conferences (Edgerton et al., 1991; Shore et al., 1986)

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

Box 5-2. Framework and Examples of Practices/Policies for Evaluating the Scholarship of Teaching

Dimensions of the Scholarship of Teaching

Making Teaching Public

Focusing on Teaching Practices and Learning Outcomes

Having Content and Pedagogical Knowledge

Departments That Practice the Scholarship of Teaching

• Have a system of peer review of teaching.

• Discuss teaching and subject content topics at department meetings.

• Encourage members to prepare teaching portfolios.

• Have a mentoring system for teaching.

• Encourage classroom visits and discussions of teaching.

• Support attendance at conferences and workshops on teaching.

• Administer major field-level exams or other assessments.

• Encourage team teaching or interdisciplinary courses.

• Encourage teaching innovations.

• Encourage research on teaching and learning.

• Seek student perceptions on teaching practices, learning practices, and learning outcomes.

• Sponsor seminars or workshops on teaching in the discipline.

• Encourage diverse approaches to teaching.

• Reward staff who publish or give conference papers on teaching.

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

Institutions That Practice the Scholarship of Teaching

• Encourage student evaluations of teaching.

• Support a mentoring program for teachers.

• Sponsor seminars, workshops, or conferences on teaching and learning.

• Require/ encourage faculty to prepare teaching portfolios or detailed reports on teaching.

• Periodically review teaching.

• Publish results of learning outcome and teaching environment surveys.

• Weigh teaching performance heavily in hiring and promotion decisions.

• Encourage a peer review program.

• Have active programs or centers to support teaching and learning.

• Have training program for teaching assistants.

• Provide grants to support research on teaching and learning.

• Have a plan for assessing student-learning outcomes.

• Survey students and graduates on learning experiences.

• Use evidence of student learning in hiring and promotion decisions.

• Reward the use and development of effective teaching practices.

• Reward teachers/ departments that promote the use of means by which discipline knowledge can be related to students.

• Have staff development programs that emphasize diverse teaching practices.

 

SOURCE: Centra (2001, pp. 8–9).

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
  • External support obtained for such purposes as improving teaching or purchasing instrumentation for teaching laboratories

Videotaping

Videotaping is a useful strategy that enables instructors to see what they do well and what needs to be improved. In consultation with an expert from the campus’s teaching and learning center, instructors can determine whether they exhibit such classroom behaviors as dominating a discussion, allowing students enough time to think through questions, or encouraging all students to participate in discussions. Faculty who have been videotaped find the experience extremely helpful, especially if they discuss the analysis with someone having expertise in classroom behavior. Videotaping is best used for formative evaluation.

Before-and-After Self-Assessment

Faculty members can use before-and-after self-assessment to determine whether course outcomes meet their expectations. Before a course begins, the instructor writes brief comments about the types of students for whom the course is intended. Given that audience, the instructor lists the most important course and learning goals and the teaching strategies she or he will design to achieve them. Once the semester has been completed, the instructor prepares a similar brief description of the types of students who actually enrolled, the instructional methods that were used, and how the students’ achievement of major goals was measured. The evaluation should address (1) goals the instructor believes were met and evidence of student learning and academic achievement, (2) goals that were not realized, (3) the nature of and possible reasons for discrepancies between the instructor’s original intentions and actual outcomes, and (4) how the instructor might modify the course in the future to achieve more of the intended goals. These self-assessments can become part of a teaching portfolio that can later be used for more summative types of evaluation.

Another form of before-and-after assessment may help instructors who are interested in examining their teaching behaviors and effectiveness rather than course outcomes. For this technique, instructors use the end-of-course evaluation form, but complete the questionnaire before their course begins (predicting how they think they will do) and again at the end of the semester (how they believe they did). They also may wish to fill out a questionnaire at the end of the term based on what they expect, on average, their students will say about their teaching. In most cases, such self-evaluations are

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

likely to be more positive than student ratings (Centra, 1973; Feldman, 1989). In looking at the results, instructors may wish to focus on any deficiencies noted in the self-evaluation or on discrepancies between their own evaluations and those of their students.

SUMMATIVE EVALUATION OF TEACHING

Evaluations from Undergraduate Students

Questionnaires are most commonly used for summative student evaluations of teaching. The questionnaires can be machine-scored and fall into two categories: those developed locally by campus teaching and learning centers by consulting the literature or adapting forms used elsewhere, and those developed by other institutions or organizations and made available for a fee.

Questionnaires vary somewhat in the characteristics of teachers and courses covered, as well as in the quality and usefulness of the scores generated for the instructor. Typically, student evaluation instruments have attempted to identify strengths and weaknesses of instructors in the following areas:

  • organization or planning;

  • teacher-student interactions;

  • clarity and communication skills;

  • workload assigned and perceived difficulty of a course;

  • quality and fairness of grading, assignments, and examinations;

  • students’ ratings of their own learning and progress; and

  • students’ ratings of their level of effort, attendance, and participation in the course, completion of assignments, and motivation.

Questionnaires used for student evaluations sometimes address aspects of a faculty member’s teaching style that may or may not contribute to student learning. For example, they may ask whether the faculty member makes eye contact with students during discussions, how many questions the instructor poses during class (as compared with the nature of the questions), or how often students may be assigned to work in groups rather than work alone. Such questions are appropriate only if they are explicitly intended to provide formative feedback for the instructor, but should not be used for summative purposes. Each instructor has a unique personality, persona, and approach to teaching. The primary concern when developing or analyzing questions on student questionnaires for purposes of summative evaluation should be whether the students are actually learning at the desired level and in ways that are consistent with the course goals (Rosenthal, 1976).

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

Global ratings of the course overall or the teacher’s instructional effectiveness also are common to most student questionnaires. For courses in science and engineering, special questions about the efficacy of laboratories, fieldwork, and research experiences also can be included as part of the standardized form or posed in a separate questionnaire. For example, the University of Washington provides separate evaluation forms for laboratories, as well as for clinics and seminars (e.g., University of Washington Education Office of Educational Assessment7). Appendix B provides more specific information about and several examples of student questionnaires for evaluating undergraduate teaching. See also Davis (1988) for compilation of questions that can be used on an end-of-course questionnaire.

It is important to note that questionnaires usually do not permit students to assess such characteristics as an instructor’s level of knowledge of subject matter. Students cannot and should not evaluate instructors in this regard. Instead, faculty peers and colleagues should assess these characteristics of an instructor’s teaching. Additional detail about the use of student evaluations for summative purposes is provided in Appendix A.

Summative Evaluation by Graduate Teaching Assistants

If a department wishes to involve teaching assistants in performing summative evaluations of faculty or improving a department’s educational offerings and approaches to teaching and learning, both the teaching assistants and faculty must feel confident that the procedures for gathering information will preserve the assistants’ anonymity. Teaching assistants need to know before participating how the information will be used and who will see the data.

When evaluations from teaching assistants are to be used for personnel decisions, the department might consider asking for written assessments. Alternatively, a system might be established whereby teaching assistants would be interviewed informally by a member of the evaluation committee and their comments recorded and submitted collectively. In either case, teaching assistants should be asked to indicate the basis for their assessment. Such information might include the number of courses they have taught with the instructor, descriptions of their training and supervisory activities, the nature and amount of their contact with undergraduate students, whether they

7  

Additional information is available at <http://www.washington.edu/oea/>.

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

were allowed to obtain informal student opinions about the course, and the extent to which they observed each major aspect of the course (e.g., lecture, laboratory).

Teacher assistants can be asked for the following kinds of information:

  • An overall judgment of the effectiveness of the faculty member’s teaching.

  • An analysis of the particular strengths and weaknesses of the teaching as reflected in the design, preparation, and conduct of the course. If the department wants specific comments on particular aspects of teaching, the instructions to the teaching assistants should emphasize the need for supporting evidence.

  • The extent to which working with the instructor contributed to the teaching assistant’s own professional development in teaching.

  • The appropriateness of the instructor’s assignments and expectations of the teaching assistants.

For each question posed, the teaching assistants should be encouraged to supply specific examples. If their responses are summarized for personnel decisions, the summary must indicate the number of teaching assistants who worked with the faculty member and the number from whom information was obtained.

Summative Evaluation by Faculty Colleagues

The following approaches might help some institutions obtain more systematic and complete information on teaching performance for purposes of summative evaluation. When these approaches could also be useful for formative evaluation, this is noted.

Ad Hoc Committees on Teaching Effectiveness

The department might appoint an ad hoc committee on teaching to evaluate each faculty member who is being considered for tenure or promotion. At smaller institutions, where final decisions for promotion and tenure may rest with an institution-wide committee rather than individual departmental committees, a similar panel separate from the committee on tenure and promotion could be established regularly to review the institution’s policies with regard to the process and use of summative evaluations for teaching.

The only responsibility of such ad hoc committees would be to evaluate teaching performance. The committee could consist of senior faculty members, one or two junior faculty members, and one or more graduate or senior-level undergraduate students. One or more of these ad hoc committee members should be from outside the candidate’s department.

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

The materials to be considered by the committee could include a variety of teaching-related materials, all of which would be supplied by the candidate: course syllabi and examinations, teaching and learning aids, and evidence of the impact of the candidate’s teaching on students’ learning and intellectual growth. The faculty member also could be asked to submit documentation for the following: currency of course content, participation in the design of courses, contributions to curriculum and instruction, supervision of student research, advising duties, preparation of teaching assistants (if appropriate), and individual and collaborative efforts to improve teaching effectiveness.

Candidates should also prepare and submit a self-assessment of their teaching effectiveness. The self-assessment could address questions such as the following: What are the goals of your teaching? Why were these goals selected? How did you know whether students were gaining competence and learning the material? How well did the courses meet your learning goals for your students, and how do you know? What problems, if any, did you encounter in attempting to meet these goals? How did you conduct the course and challenge and engage students? How did your methods take into account the levels and abilities of students? How satisfied were you with the course? What were the strong and weak points of your teaching? What would you change or do differently the next time you teach the course? What did you find most interesting and most frustrating about the course?

The candidate’s department chair also could provide the committee with student evaluations from courses taught previously, names and addresses of student advisees, dissertation advisees, enrollees in past and current courses, and the candidate’s cumulative teaching portfolio if one has been prepared. The candidate should see the list of materials submitted to the committee and be given the opportunity to supplement it.

Through brief interviews, telephone calls, letters, or brief survey questionnaires issued to the candidate’s current and former students from a variety of courses, the committee could compile a picture of students’ views of the teacher that would supplement the written evaluation reports from past courses. In addition, each committee member could observe and evaluate at least two of the candidate’s classes.

Studies of such ad hoc committees revealed that members met several times to discuss their individual findings, used a rating form, and prepared a report, which was then submitted to a departmental tenure and promotion committee (see Centra, 1993, pp. 129– 131 for details). Given the highly

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

positive reliability coefficients reported by Root (1987) for colleague evaluations when the colleagues are properly prepared, one can conclude that the assessments of a faculty member’s teaching effectiveness thus provided are reliable.

Colleagues’ Evaluation Questionnaires

Several questionnaires have been designed to elicit colleagues’ evaluation of a candidate’s teaching effectiveness for summative evaluation purposes, although they may also be used for formative evaluation. Two forms developed at Syracuse University and the University of Texas at Austin provide scaled-response items and open-ended questions that faculty colleagues and department chairs can use to guide their analysis of a candidate’s chosen instructional materials, as well as teaching behaviors they observe during classroom visits. These forms, printed in their entirety in Appendix C, cover questions grouped under the following five characteristics of good teaching:

  • organization of subject matter and course,

  • effective communication,

  • knowledge of and enthusiasm for subject matter and teaching,

  • fairness in examinations and grading, and

  • flexibility in approaches to teaching.

A form designed by French-Lazovik (1981) is also provided in Appendix C. This form offers five broad questions with which faculty peers can evaluate such dimensions as the quality of materials used in teaching. The form also lists which portfolio materials should be reviewed and suggests a focus for colleagues when examining these materials. Other institutions have developed more extensive guides to help candidates prepare for peer evaluation and to assist faculty colleagues in conducting such evaluations effectively (e.g., the University of Texas’s Preparing for Peer Evaluation;8 see also the many resources available through the websites of college and university teaching and learning centers throughout the United States and in other countries).9

While the kinds of forms included in Appendix C have proven helpful to faculty in identifying what materials and characteristics of a candidate’s teaching to assess, the reliability and validity of their evaluations depend on the use of

8  

Additional information is available at <http://www.utexas.edu/academic/cte/PeerObserve.html>.

9  

A list of websites for teaching and learning centers of colleges and universities in Asia, Australia and New Zealand, Europe, and North America is available at <http://eagle.cc.ukans.edu/~cte/resources/websites.html>.

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

appropriate procedures. For example, as noted above, Root’s (1987) study indicated that a minimum of three departmental colleague evaluators should use the form. They should discuss the evaluation criteria before reviewing materials and making classroom visits. Evaluators also should be provided with examples of evaluations from other candidates, both internal and external, that illustrate high and low ratings.

EFFECTIVE IMPLEMENTATION OF EVALUATION METHODOLOGIES

Before revising and implementing policies and procedures for evaluating teaching, especially for summative evaluation, stakeholders should proceed in ways that will confer maximum credibility on the results of their efforts. Depending on the institution in question, administrators, the academic senate or committee on tenure and promotion, and faculty must accept that the results of evaluation efforts will be helpful both in personnel decisions and in improving the teaching effectiveness of faculty. Policies and procedures that could assist in the process include the following:

  • Closely involving the institution’s faculty in selecting evaluation methods, drafting the policies and procedures to be implemented at the departmental and institutional levels, and determining the procedures to be used for analyzing and reviewing the results of summative evaluations of teaching.

  • Recognizing and addressing as part of the system of evaluation the full range of teaching styles and activities, both in and out of class. Effective evaluation systems should be able to assess a broad range of teaching styles and approaches.

  • Making evaluation forms and supporting documents freely available to faculty so they understand what information will be considered legitimate and relevant in the evaluation of their teaching performance.

  • Establishing uniform procedures for collecting and using information from students. For example, institution-wide procedures should be defined that protect the anonymity of respondents and ensure that instructors do not see end-of-semester student evaluations until after they have submitted their grade reports.

  • Establishing a uniform and equitable system for the analysis and review of evaluation data, including appropriate response rates for end-of-course student questionnaires.

  • Making clear which letters and surveys will be kept confidential; which can be seen by the faculty under review;

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

and which information, if any, will be shared with students for purposes of selecting future courses.

In addition, the following procedures could make any evaluation process more equitable and more easily accepted.

Regular Meetings Between New Faculty Members and the Department Chair

The department chair should meet with each new faculty member and make clear the department’s and the institution’s general expectations and policies regarding teaching. Norms of grading for assigned courses should be described. The chair also should encourage the new faculty member to consult with other department colleagues who teach the same or related courses to develop policies and procedures for establishing desired learning outcomes, pedagogical approaches, and methods for assessing learning (see, e.g., Annex Box 1-2 in Chapter 1). New faculty members should be encouraged from the beginning of their employment to contribute actively to such discussions. The chair also should encourage and assist new faculty members to work with faculty colleagues both within and outside the department on improving their teaching, and possibly assign a senior mentor to assist them.

Formative Discussions Between the Department Chair and Individual Faculty Members

Optimally, department chairs should meet at least annually with each member of the department to discuss teaching accomplishments and issues. Such meetings are especially critical for any faculty member whose teaching evaluations are substantially below the department’s expectations or those of other departmental colleagues. These meetings should occur well before summative decisions are to be made so that candidates have ample opportunities to develop a plan for improving their teaching. Additional meetings at regular intervals should be scheduled to assess progress in addressing concerns.

Sharing of Faculty-Generated Teaching Portfolios

The department’s academic personnel files could include a teaching portfolio for each faculty member. Faculty members could place in the portfolio copies of their course materials (including learning objectives and expected outcomes), syllabi, reading lists, assignments, examinations, and instructional materials. A website also could be established for this purpose. Depending on institutional policy, student evaluation forms or summaries of students’ course evaluations also could be included in the portfolio. It should be

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

assumed that a faculty member would continue to have access to all materials in his or her portfolio, unless letters solicited or submitted in confidence were protected under rules of the university.

Feedback from Graduating Seniors and Alumni

As part of the department’s regular academic program review, graduating seniors and alumni could be surveyed. Relevant survey information about an individual instructor’s teaching effectiveness would be placed in his or her teaching portfolio. Instructors should be made aware that such information will be included in their portfolios and be allowed to provide written comments or responses, where permissible.

Departmental Panel on Teaching Effectiveness and Expectations

In addition to an ad hoc department committee to monitor candidates’ progress in teaching, as discussed above, the department as a whole could establish a faculty panel that would summarize the department’s policies and procedures regarding expectations for teaching effectiveness, the methods and criteria used to judge that effectiveness, and the role of evaluation in academic personnel decisions. The panel would remind faculty of the resources available to them through the institution for improving their teaching. Members of such a panel might include a former recipient of the campus teaching award, a respected senior faculty member who teaches introductory and lower division courses, and a newly tenured associate professor.

Oversight Committee to Monitor Departmental Curriculum and Instruction

The department chair could establish a permanent faculty committee to monitor the quality and effectiveness of instruction by all members of the department. This committee would also oversee all evaluations of curriculum, teaching, and student learning and, where appropriate, nominate faculty for the campus’s or college’s teaching awards.

Legal Considerations

All stakeholders who are involved with the evaluation of teaching must act in accordance with institutional policies that have been designed to ensure legally equitable and fair treatment of all involved parties. Such policies might require, for example, that:

  • The faculty be involved in the design of an evaluation system, as well as in evaluations of colleagues.

  • The institution complies with all procedures specified in contracts or

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×

handbooks, as both are legal documents.

  • The evidence that is used for personnel decisions be job-related and nondiscriminatory.

  • The faculty members be allowed to respond to individual evaluation reports or to clarify information in their dossiers or portfolios.

  • The procedures used in internal review of decisions be clearly elucidated and made available to all faculty.

Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 69
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 70
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 71
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 72
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 73
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 74
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 75
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 76
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 77
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 78
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 79
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 80
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 81
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 82
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 83
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 84
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 85
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 86
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 87
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 88
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 89
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 90
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 91
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 92
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 93
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 94
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 95
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 96
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 97
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 98
Suggested Citation:"5 Evaluation Methodologies." National Research Council. 2003. Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. Washington, DC: The National Academies Press. doi: 10.17226/10024.
×
Page 99
Next: 6 Evaluation of Individual Faculty: Criteria and Benchmarks »
Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics Get This Book
×
Buy Paperback | $54.00 Buy Ebook | $43.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Economic, academic, and social forces are causing undergraduate schools to start a fresh examination of teaching effectiveness. Administrators face the complex task of developing equitable, predictable ways to evaluate, encourage, and reward good teaching in science, math, engineering, and technology.

Evaluating, and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics offers a vision for systematic evaluation of teaching practices and academic programs, with recommendations to the various stakeholders in higher education about how to achieve change.

What is good undergraduate teaching? This book discusses how to evaluate undergraduate teaching of science, mathematics, engineering, and technology and what characterizes effective teaching in these fields.

Why has it been difficult for colleges and universities to address the question of teaching effectiveness? The committee explores the implications of differences between the research and teaching cultures-and how practices in rewarding researchers could be transferred to the teaching enterprise.

How should administrators approach the evaluation of individual faculty members? And how should evaluation results be used? The committee discusses methodologies, offers practical guidelines, and points out pitfalls.

Evaluating, and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics provides a blueprint for institutions ready to build effective evaluation programs for teaching in science fields.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!