3

Models from Other Disciplines and Nations

Engineering has much to learn from other disciplines and countries in evaluating the impact of faculty members. In the second session of the workshop, four panelists looked at models for evaluating faculty impact, drawing from experiences in a school of medicine, design programs, a law school, and the United Kingdom.

MEDICINE

As in engineering, science is at the foundation of medicine, said William Eley, executive associate dean at the Emory University School of Medicine. Medicine, too, went from a practice-based education to one heavily based in science. In addition, medicine is addressing increasingly complex and difficult problems, as is engineering.

Reflecting these new considerations, the Emory University School of Medicine has developed 28 student-physician activities based on competencies. Eley explained that students do not have to achieve mastery in every competency, because medical education is already too long, but they need to demonstrate ability in these areas to graduate.

Dealing with this complexity requires reaching out to fields beyond medicine, a practice that has changed the role of physicians. Physicians need a broad knowledge to manage all of a patient’s—or a system’s—problems. As Eley put it, “The hero—while nice to think about in the world of problems in which we live—is now the team.”

Promotion and tenure decisions therefore need to acknowledge attributes such as leadership and individual contributions to a team.

Academic medical centers are also asked to engage in addressing the problems of surrounding communities, which injects another new element in the process of evaluating faculty.

Eley reported that the School of Medicine recently developed new promotion and tenure guidelines broadening previous conceptions of service, education, and, to some extent, research, “although research has been the most entrenched.” The new guidelines recognize levels of accomplishment at the regional, state, national, and international levels, with benchmarks for success at each level recognizing work in a broader context than had been the case. For example, he cited a pediatrician faculty member who had worked with the state of Georgia to put school nurses in about 150 elementary schools. “She could not get tenure under our old guidelines,” he said. “When we brought her up for discussion for tenure, everybody in the room had tears in their eyes and said, ‘What have we been doing . . . not to honor that kind of service?’”

Medical education is meant to transform a layperson into a physician. To do this well, Eley said that educators need to know their students better than the students know themselves. Faculty members need to teach in small groups or individually so that they get to know students as human beings and as thinkers and can assess abilities such as clinical reasoning, “which is probably the most important skill our students have.” Yet medicine continues to rely on standardized tests that do not come close to measuring these abilities. “There’s a national tyranny of standardized tests, and medicine does that worse than anyone.”

Finally, physicians need to be professionals, Eley said, which means they need to develop attributes such as kindness, compassion, diligence, truthfulness, and the ability to communicate with colleagues and families alike in critical situations. But “we’re not doing a great job of assessing those things,” he acknowledged. “How do we assess what is most important for our society?”

In response to a question, Eley noted that even clinical faculty need to publish to be promoted, although some of the criteria for research, teaching, and service have been broadened. And individuals can change their faculty track partway through the promotion and tenure process at Emory. “Now we’re letting people be hired without having to commit to a track for the first couple of years.” In addition, the tenure track has gone from 7 to 9 years so that people have more time to decide what they want to do.

DESIGN

Engineers have “many different worlds” from which they can draw in developing new ways to evaluate faculty, said Alan Balfour, former dean of the College of Design at Georgia Tech. He had recently returned from a school called the Architectural Association in London, which he described as the most distinguished school of architecture in the English-speaking world. There, students have no curriculum. They spend each year doing complex projects involving design, engineering, and environmental issues and are judged by a review committee using very high standards at the end of the year. Faculty have one-year appointments, and if not enough students sign up for the classes they propose to teach, they immediately leave the institution. “I can’t imagine how you would apply this model to engineering,” said Balfour, “but at least think about” it.

At Georgia Tech, the College of Design includes schools of planning, building construction, industrial design, architecture, design, and music, each with a very different culture and funding allocations, Balfour reported. Planners engage in conferences and produce some publications, but not much money is available for research. Building construction has some publications and a surprisingly strong base of research activity. Industrial design, which increasingly includes engineers, emphasizes exhibitions but has little money for research. Architects pursue exhibitions, competitions, and some publications but not much research, again because of lack of funding. The school of music, an especially “mixed bag,” according to Balfour, includes musical technologists and electrical engineers in addition to music theorists and performers.

Trying to get engineers to see the substance in what others do is difficult, he observed. “Imagine trying to persuade engineers of the significance of a symphony developed from the sounds on the streets around them,” or “asking engineers to accept that this peculiar building, which won an award, has some significance beyond merely being an attractive drawing.”

Engineers have trouble tolerating uncertainty, he continued, which is characteristic of disciplines like architecture, and creativity is inherently subjective. This subjectivity makes it very difficult to measure or set criteria. Balfour noted that in design, there are no peer-reviewed journals, which makes the assessment of research different than in other fields.

There are programs to incorporate creative arts with engineering. New degrees in music technology combine interests in engineering and music. Some fields of industrial design, particularly applications in computing, are also blending perceptions from different fields. But just a few faculty members have been driving these connects between overlapping interests, Balfour said. “How we get institutes as a whole, societies as a whole, to see this and actively bring these people together is very important” to create a rich interconnection “between scientific and engineering knowledge and creative thinking.”

This type of interaction is already occurring in companies like Amazon, Balfour pointed out, and universities also need to create it. The Inventure Prize at Georgia Tech is one example of an effort to catalyze creative thinking through project-based design.¹ These are the kinds of skills that engineering education needs to instill, Balfour concluded, since engineers have other ways, such as computer-based learning, to master the fundamentals of their disciplines.

LAW

The evolution of legal education is similar to that of engineering, according to David Santacroce, associate dean of experiential education at the University of Michigan Law School. In the early 1900s, legal education moved from an apprenticeship to a theoretical orientation. Law schools became a place to discuss cases and learn the tools of legal reasoning, with the learning of practice left until after law school. Even today, many law school faculty members are heavily involved in research and have not practiced law.

Public interest lawyers began to use law students to help with cases during the civil rights movement, which instigated a “movement” toward clinical education, said Santacroce. “It’s been a fight at times, but now clinical ed is at basically every law school in the country and is required by the accrediting agency [the American Bar Association].” After the first or second year of law school, depending on state rules, students are entitled to practice law as a “first chair attorney,” meaning that they can talk to clients and argue to the court under the supervision of a licensed attorney.

The growth of clinical education has led to a range of systems of employment and retention, from faculty members who are largely separated from practice to those grounded in practice. Santacroce explained that the categories of legal professorship reflect both the “use” of the faculty (what they do) and the mode of how they are recruited, compensated, and evaluated. In the past, use dictated mode but now there are a number of combinations and exceptions.

According to Santacroce, the use categories are as follows:

• Doctrinal or podium faculty members teach doctrine and legal reasoning. They focus on scholarship and are supposed to engage with the world on that level.
• Clinical faculty teach the craft: professional and interpersonal engagement, professional identity, and ethics. “We engage the world and community directly,” he said, noting that legal clinics at the University of Michigan delivered more than 100,000 hours of free legal services in the previous year.
• Legal writing faculty teach the fundamentals of legal research and writing.
• Professors from practice range from craft to doctrine and serve as a connection with the bar and law firms.

The highest mode is the traditional tenure track process of probation, a faculty vote, and tenure. Faculty on the traditional tenure track get full say in institutional matters and the highest level of perks. Podium faculty fall into this mode. They are valued for their scholarly impact and reputation; teaching is comparatively less valued. These faculty members contribute in a major way to law school rankings, a very important business consideration for law schools.

___________________

¹ Information about the Inventure Prize is available at https://inventureprize.gatech.edu.

Clinical faculty can be either on the traditional tenure track or in another mode. Santacroce estimates that 25 percent of clinical faculty are on the traditional tenure track (up from 0 percent in the 1990s). But they are subject to a somewhat different evaluation method, to reflect the difference in what they do compared to the podium faculty. Scholarship is still evaluated but it may be more applied.

The modified tenure track follows a model more common in medical, dental, or nursing schools. About 20 percent of clinical faculty and some legal writing faculty members follow this mode. The criteria for evaluation and the faculty member’s say in institutional governance can vary. For about half the faculty in this mode, scholarship is not even a requirement. Teaching is much more important, as is service, whether in the community or in other ways.

Long-term contracts are the mode for some clinical faculty, legal writing faculty, and a small percentage of professors from practice. These contracts mimic parts of tenure through presumptively renewable contracts that are dischargeable for cause. Teaching is the most important criterion.

Professors from practice are utility players that are similar in use to adjunct faculty. They are on at-will appointments. They often serve an outreach function as they typically have close relationships with law firms, and many have had prominent positions in the nonacademic legal community.

Finally, there are fellows on three-year terminal contracts, who are essentially receiving additional training.

UNITED KINGDOM

Ross Ethier, professor of bioengineering at the Georgia Institute of Technology, has been a department chair at the University of Toronto, Imperial College London, and Georgia Tech. While the system for evaluating faculty is largely the same in Canada and the United States, he noted that it is very different in the United Kingdom.

Tenure was abolished during the administration of Margaret Thatcher, so “your protection as a faculty member is the same as any employee at any other institution,” although faculty are on 12-month salaries. With essentially no private system of higher education, all universities in the United Kingdom are, “practically speaking, state schools.” And the system is highly regulated, which creates what Ethier called “a closed ecosystem.”

The government evaluates departments on separate systems for teaching and research. Technically, these do not involve the evaluation of individual faculty members, he said, but “it’s hard for it to not filter through to the individual faculty member level.”

The teaching evaluation is called the Teaching Excellence and Student Outcomes Framework. Students fill out a detailed online questionnaire. Each institution has a publicly available ranking. “From the point of view of individual faculty members, it’s pretty much the same system as in the US,” said Ethier. “We all get our teaching evaluation scores, and many of the things that go into that teaching evaluation score . . . are outside the control of individual faculty members.”

The research excellence evaluation takes place nationwide every 6 or 7 years and consists of a large three-part document. Part one concerns “outputs,” part two “impact,” and part three “the environment.” The three parts have a 40-40-20 weighting. Outputs are determined on the basis of faculty members submitting their four best papers, books, or other scholarly products accompanied by 100-word descriptions of the value of each paper. The descriptions are graded.

Impacts, which do not include papers or books, measure a department’s productivity in other ways. For example, Ethier’s department might cite a medical device that made it into the marketplace, innovations that led to clinical trials, or new protocols for medical imaging. “It’s supposed to be something that changes or benefits the economy, society, culture, public policy or service, the health of the environment, or the quality of life beyond academia,” he explained.

The environment score depends on factors such as how much money is available in a department through grants.

This assessment determines the amount of general research funding, not specific research grants, an institution gets from the government. It is based on a “prefactor” rating of one, two, three, or four stars. The prefactor for one star is 1, for two stars it is 2, for three stars 4, and for four stars 6. A decline from four stars to three can cut a department’s budget in half—“and that’ll get your attention.” At Imperial College London, any department that gets a one- or two-star rating is closed within 2 years.

A department can decide not to include a staff member in the evaluation, though “if your chair tells you that your outputs aren’t going to be [considered], you should start looking for another job.” After several years of experimenting with an algorithmic approach to evaluation, the system has returned to the method of people seated around a table to evaluate evidence and make judgments.

The system has both good and bad aspects, according to Ethier. One pernicious outcome is that the process never ends. Preparation for an evaluation takes years, after which the countdown to the next one begins. “It’s very deleterious to the life of the department,” he said, noting that departments do things like hiring superstars for a few years to bump up their numbers. But he acknowledged that the system’s emphasis on impacts is also very useful, especially in engineering, where the development of a new product for the marketplace can be undervalued by evaluators if it is not explicitly recognized.

Ethier concluded by noting that being chair of a biomedical engineering department means reporting to both a dean of engineering and a dean of medicine. “The outcomes in medicine are similar to what we care about in engineering: competence, integrity, changing the world in a positive way. Yet you can’t believe how different the metrics are between engineering and medicine in terms of the details of the process.”

EVALUATIONS OF TEACHING

One of the topics discussed in the question-and-answer session was the role of teaching in faculty evaluations. Teaching matters less for faculty on the traditional tenure track and more for faculty on alternative tracks, said Santacroce. The study of the pedagogy of teaching law, publications on teaching, new programs, and new partnerships with community members all matter more for faculty on nontraditional tracks.

Ethier pointed out that there are many opportunities for creativity and innovation in educational programs, which in turn can affect evaluations. A professional master’s program in biomedical engineering established in London went from 15 students to 85 in 2 years. “There was clearly a huge need for that,” he said. An important part of the program was accreditation, which involves the professional society, “so there’s a great opportunity for interaction there. Maybe it’s not as glamorous as some other things, but it fulfills an important need.” This kind of program is still best housed in an academic unit, he said, “because at the end of the day we’re teaching fundamentals of the disciplines, [but] there’s a practical aspect to it as well.”

Eley observed that every subdiscipline in medicine has its own education professional society. In addition, annual meetings in medical disciplines have sessions featuring educators where people exchange ideas, innovate, work together, and disseminate programs to test their generalizability. However, every school does education differently so it is difficult to know what works at a national level in medical education.

Balfour pointed to the fact that students are much more civic minded and socially conscious than in the past. This creates new opportunities for bringing together engineering disciplines with design disciplines in project-driven programs that tackle real-world problems, such as clean water or sanitation. Encouraging and rewarding these types of efforts can be very effective. Another participant pointed out that this is the approach of the Grand Challenges Scholars Program based on the NAE’s Grand Challenges for Engineering.

DEFINITIONS OF SERVICE

Another topic that came up was the definition of service in different disciplines. Eley observed that as students are working more in community clinics, external service has gained more recognition.

In law schools Santacroce noted that service has always been externally focused, not simply activities within the institution. Faculty supervision of legal clinics is increasingly recognized as service, as is involvement with state legislatures. But the formal definitions are broadly written and have gradually been changing with cultural norms. These kinds of service activities need to be more institutionalized, he said, if they are to play a larger role in promotion and tenure.

Ethier noted that activities such as participation in Engineers Without Borders were not recognized in academic circles as valuable until fairly recently.