“Assessment and evidence are critical components of innovation adoption.”
Thomas Holme
“It doesn’t matter whether [students] are going into law, business, economics, or international politics—they have to have a sense of where these technical forces are and how they’re shaping the world.”
James Anderson
The predominant focus of this workshop was to identify the barriers to improving chemistry education and to highlight innovative approaches to overcome the barriers. Seven speakers presented on a range of topics to enhance the learning experience, including key requirements for education reform and innovative approaches that move away from standard lecture and testing formats; those presentations are summarized in this chapter. Thomas Holme, of Iowa State University and director of the American Chemical Socieity (ACS) Exams Institute, spoke about some of the challenges in replicating education reform efforts. Clark Landis of the University of Wisconsin, Madison, presented an example of large-classroom reforms and the challenges with reform assessments using traditional metrics. A new approach integrating chemistry and physics curricula into introductory courses was discussed by James G. Anderson of Harvard University. Scott Auerbach of the University of Massachusetts, Amherst, described an approach to undergraduate education that puts context front and center in course design. Michael Cima of Massachusetts Institute of Technology detailed his work developing a massive open online course (MOOC) on solid-state chemistry. Jeffrey Moore talked about the student-centered organic chemistry class he developed at the University of Illinois in Urbana-Champaign. Last, how the design of exams can influence how and what students learn in their chemistry classes was discussed by Angelica Stacy of the University of California, Berkeley.
What are the barriers that keep successful chemistry education reforms from spreading beyond a local college or university? The fundamental tension, said Thomas Holme, is that teaching is both a personal experience and a corporate enterprise. “It’s a personal activity to each of us, but a corporate enterprise because other people care how we do it,” explained Holme. He added that anyone who has written a textbook from scratch has experienced this tension. This same tension arises when it comes to educational reforms—everyone is interested in a very broad way in propagating successful new methods for teaching chemistry, but such methods need to appeal to individual teachers. “Enthusiasm can get us in the door,” but it must be teamed with an assessment in order for education reform to be effective, argued Holme.
There are resources available to help with education reform efforts. A National Research Council (NRC) report, How People Learn: Brain, Mind, Experience, and School (NRC 2000), provides a good foundation and notes the importance to the learning process of prior knowledge, whether that knowledge is correct or not, and metacognition, the ability to reflect on one’s own thinking and learning processes. Holme pointed out that a fairly robust albeit young field of chemistry education research was among the disciplines that contributed to a 2012 NRC Discipline-Based Educaton Research (DBER) report from the Board on Science Education. The DBER report also stresses the importance of prior knowledge as an obstacle to teaching chemistry. Chemistry students, for example, have a difficult time envisioning the particulate nature of matter, and this difficulty is often associated with the way it is presented (Cooper et al. 2010). Student conceptions of size scale (e.g., nucleus, atom, molecule, and compound) also present a major challenge in chemistry, and evidence suggests that the ability to understand scale may be the best predictor of success in general chemistry (Gerlach et al. 2011). Unfortunately, chemistry educators are to blame for at least part of this difficulty, said Holme.
The DBER report (NRC 2012) also talks about how to use educational research to impact real-world teaching. The report acknowledges that it is hard to turn basic research into interventions and it is harder still to turn local changes into larger scale change. It also notes that while the National Science Foundation (NSF) has funded professional development activities related to teaching for some time, those activities have largely been self-selecting, leading to a “preaching to the choir” effect (Feuer et al. 2002). The good news, Holme explained, is that the chemistry community as a whole is good at thinking about the diffusion of innovative education reforms, and this is where the relative roles of enthusiasm and evidence come into play.
The classic book on this subject is Diffusion of Innovation by Everett Rogers (2003), said Holme, who spent a few minutes summarizing some of the key ideas in the book. The basic definition of diffusion of innovation is “the process in which an innovation is communicated through certain channels over time among the members of a social system.” Communication, said Holme, involves convincing somebody who is listening for evidence about key innovation characteristics such as relative advantage, compatibility, complexity, trialability (can a new technology be implemented in steps), and observability. If evidence for these characteristics is not provided, listeners will make up their own evidence and likely conclude that the status quo is better.
Given that an implicit goal is to convince somebody that a new educational innovation is worth adapting, it is important to understand the five stages of innovation adoption. “At the risk of sounding unduly mercantile,” said Holme, “we need to understand our targeted customer.” The five stages are
1. Knowledge of the innovation
2. Persuasiveness of the innovation—is it better for me?
3. Decision—adopt or reject
4. Implementation—adapt and adopt
5. Confirmation—to keep or not to keep
Enthusiasm, Holme said, plays a key role in the first two stages. Evidence is key in the second and third stages. Assessment is critical for the final two stages. But, Holme cautioned, each of these stages takes place in a social system that is not exchangeable and not particularly amenable. In addition, university faculty will not just be skeptical in the face of enthusiasm, but they will “jump in with their skepticism at every opportunity.” It is important to remember, Holme added, that “our colleagues may, or may not, know how to decide about the efficacy of educational innovations they try, but they probably believe they know.” This is why assessment and evidence are critical components of innovation adoption. The problem, however, becomes one of getting the data that will serve as evidence in the context of the academic social system.
This is where the ACS Exams Institute can help. The ACS Exams Institute writes nationally standardized exams covering all fields of chemistry and provides resources for outcomes measurement. Recently, the Exams Institute, with funding from the NSF, conducted a national survey of 14,000 professors and instructors in the United States to assess their understanding of assessment terminology and techniques. Holme described the survey and the statistical methods used to analyze the responses. The 1,500-plus survey respondents fell into six clusters of understanding. General familiarity with assessment terms was not high across the six clusters, although analytical chemists tended to score higher than other groups in understanding statistical terms and methods. The lesson here, said Holme, is that everyone needs to be careful when conducting assessments; they need to truly understand what they are doing and what the results are telling them in terms of evidence for whether an innovative chemistry education method works.
Holme noted, too, that sampling in assessment surveys remains a challenge because most studies of educational innovation use convenience samplings of the students who come to the course. Studies also tend to build in bias because disaffected students leave the course before data are collected, an issue of particular concern for those who study MOOCs, for example. He remarked that institutional review boards, which become involved with research on human subjects, place an important constraint on building meaningful control-based experiments. “If we know something is fundamentally better for students, it is unethical to train some of our students with something we know is not good,” explained Holme.
Responding to a question from Angelica Stacy of UC Berkeley, Holme acknowledged that it is unlikely to ever have enough evidence to prove an educational innovation is effective, but that it should be possible to have enough evidence to take wise action as to where to go with an innovation. Auerbach then asked what is known about assessing process-oriented, laboratory-type courses, and Holme responded that there are resources available on this subject and added that assessing those courses is more challenging. He added, too, that the ACS Exams Institute has the resources to build assessment tools to help those who are developing educational innovations, but that it will take time to develop tools that the ACS feels are good enough to stamp with its imprimatur.
LARGE-CLASSROOM REFORMS: FIVE BEST TEACHING PRACTICES
The observation that certain groups of students are underperforming in introductory or “gateway” chemistry courses has been an important driver for reforming undergraduate chemistry education at the University of Wisconsin, Madison, explained Clark Landis. Another driver is the desire
to increase the fraction of students successfully completing introductory chemistry and who will then take the second semester of the two-course chemistry sequence. Landis and his colleague Ned Sibert also hoped to make teaching introductory chemistry more exciting for themselves. In addition, UW-Madison compiled student data in 2008 and discovered a gap in terms of adverse events, which Landis defined as getting a grade below C or dropping the class, between targeted minorities and the general, nontargeted student population (see Figure 3-1).
Landis described the comprehensive course reform that he and Sibert designed based on what they call the “five best teaching practices” that influence student success in college courses: learning in context, group-based learning, increased time on a task, increased frequency of feedback, and a positive classroom climate (Brower 2009, Cabrera and La Nasa 2005, Treisman and Surles 2001, University of Wisconsin–Madison 2013). These reforms include using concept tests and clicker questions in lectures as ways of making lecture sections more interactive. Online homework, tutorials, videos, and simulations are important elements of the new Chemistry 103 course design, as are peer-led teaching and review. The new course includes spiral curricula, the idea that it is possible to introduce many concepts at a fairly superficial level early on and then return to those concepts regularly, developing them in greater depth each time the concepts are discussed during the semester. Landis explained that spiral curricula work well in conjunction with big, real-world problems.
The primary focus of these changes was to promote active learning in the context of a course that as many as 2,100 students take each fall. The structure of the course includes three lectures, two discussion sections, and 2 hours of laboratory instruction each week. Inquiry-based cooperative learning activities centered on group-oriented challenge problems are used in the discussion sections and in voluntary evening workshops.
FIGURE 3-1 Gap in rates of adverse outcomes for students across a range of ACT scores. SOURCE: Clark Landis.
From fall semester 2009 through spring semester 2011, Landis and Sibert compared student performance between a reformed and a traditional chemistry course. The study included 189 students in targeted groups and 1,680 students in nontargeted groups in the reformed courses, and 170 targeted students and 1,333 nontargeted students in the traditional sections. Assessment elements included grades and retention; six common questions on the final exam; one common essay question; and student surveys of hours spent outside of class, course perceptions, and student assessment of learning gains. They also monitored gender-based and target group achievement gaps (Seymour et al. 2000).
The results surprised Landis. The reformed course did not appear to affect the achievement gap for targeted students, the fraction of adverse outcomes, or performance on common final exam questions or the essay question. Students in the reform sections did perceive a greater emphasis on collaborative and conceptual learning, worked and discussed more outside of class, attended class more often, and related chemistry more to daily life. They were also more confident in their problem-solving skills.
These surprising results led Landis to ask, “Should we evaluate performance differently?” He cited work from the New Traditions Project, conducted some 15 years ago, showing that students who had been in active-learning sections performed no better than students in traditional sections when tested using standard written exams. However, this study found that “almost uniformly, the students that were in the active learning class were assessed as being better performing in oral exams than the students in the traditional class,” explained Landis. “It could be that we just do not have good ways of assessing effectiveness of these methods.”
Landis also noted that a study conducted at the University of Colorado showed that the gender gap in a second- or third-year physics class disappeared in the course of writing two 15-minute essays on how a student’s values are related to the course. However, when Landis and Sibert conducted the same experiment, they found no differences in performance. He concluded his remarks by saying, “We think value is added by the reformed classes, but we just are not capturing that value in our standardized assessments.”
William Tolman commented that he and his colleagues at the University of Minnesota have done similar experiments in organic chemistry class design and also found that performance as measured on written tests did not improve, but that attendance and both student and teacher satisfaction improved. One area that might be showing improvement, he said, is in discovery-based team learning in the organic chemistry laboratory course, though the results are still preliminary. Scott Auerbach thought that one problem with assessing these new teaching methods is that they have different sets of learning goals that the standard assessment techniques are not designed to capture. Landis agreed and
noted that there is a good chemistry concept inventory being developed that he is eager to use as an assessment tool.
Anne McCoy wondered if differences might start showing up in later courses, where students who had taken the reformed classes might show better retention of the concepts they learned in the new chemistry sections. Landis said that he would like to run such longitudinal studies, but funding is an issue. He reiterated the need for longer-term support when Jodi Wesemann asked what was needed to keep these reform efforts going.
Jeffrey Reimer from the University of California, Berkeley, asked if anyone had conducted studies comparing sections taught by white male faculty and those taught by targeted minority faculty. Landis said he did not know of any work in that area but noted that he and his colleagues found no difference in performance between students taught by male versus female faculty.
TEACHING INTRODUCTORY CHEMISTRY WITH A MOLECULAR AND GLOBAL PERSPECTIVE
James Anderson described Harvard University’s efforts to infuse chemistry and physics into an introductory chemistry course. Their goal is not to recruit more chemistry majors, said Anderson, but to develop a chemistry curriculum that is evolving to keep more students engaged in the physical sciences as a whole and to make the rest of the student population more aware of and appreciative of the physical sciences. The reason for that emphasis, he explained, is that the physical sciences are playing a central and critical role in solving the major problems facing human civilization today.
Harvard’s Physical Sciences 11, Foundations and Frontiers of Modern Chemistry: A Molecular and Global Perspective, reflects the idea that introductory courses in both chemistry and physics have been taught separately and without a compelling context. The Physical Sciences 11 course is based on the premise that decisions on what university graduates face in their academic career directly relate to what they take in their freshmen year. “If the separation between science and society occurs in the freshman year, it’s irreversible for that generation,” said Anderson.
Chemistry and physics faculty are both to blame for the lack of appreciation for and understanding of the physical sciences because the courses they teach create an exclusive club of students who can excel at these subjects instead of an inclusive group of students who understand the basic concepts of the physical sciences. In contrast, said Anderson, the life sciences have clearly demonstrated how important they are in the larger context of society. He explained that by context he meant linking the essential concepts of chemistry and physics to their connection with the big problems that intrigue students today—energy, human health, national security, climate change, and others. He noted that while there are significant differences among universities—yes, he acknowledged, the students at Harvard, MIT, and Caltech are somewhat different—there is a general pattern common to all: attrition from the sciences during and following the freshman year (see Figure 3-2).
For the first few decades after World War II, only 10 percent of entering undergraduates completed their baccalaureate degrees in science, which met the demands of graduate and medical schools. Today, however, “this zone of scientific and technical illiteracy has now become a fundamental problem that we have to deal with because of these issues of national security and competitive economic considerations on a global scale,” said Anderson. The problem of driving scientific and technical literacy to a level where 95 percent of graduates are scientifically and technically literate is what prompted Harvard to completely revamp freshman chemistry, he said.
The current strategy in introductory chemistry, said Anderson, is to present lectures and text material that covers the basic formalism and theory, followed by problem sets and exams. Solid evidence shows, however, that there are two basic failures with this “formalism first” approach to teaching. First, he said, it results in “disembodied knowledge”—students cannot attach the knowledge to a context or their past experience, and so it is largely meaningless symbols and
FIGURE 3-2 Attrition of undergraduate students from science majors. SOURCE: James Anderson.
facts to memorize. Second, as Nobel laureate and noted science educator Carl Wieman has shown, knowledge obtained this way is filed away in the brain in a separate compartment and building links to that compartment after the fact is much harder and less effective than if it had been filed correctly from the start.
Anderson and his collaborator, Harvard physics professor Efthimios Kaxiras, have taken an approach to addressing these problems with a strategy that links concepts with context (see Figure 3-3). As an example, electrochemistry is a great way to teach about Gibbs free energy, electron flow, electromagnetism, and chemical transformation when it is connected to a context of the electric car. In writing the textbook for this course, Anderson and Kaxiras used case studies that are broken into their parts, analyzed, and reassembled. When Anderson gave the first lecture of Physical Sciences 11, 25 students were present. By the fourth lecture, there were 125 students in the lecture hall. A year later, around 300 students had completed the class, which is notable because Physics 11 is a considerably more difficult course than the alternative class the students can take, Anderson said.
FIGURE 3-3 Linking concepts to context is the guiding principle for a new introductory physical science course taught at Harvard University. SOURCE: James Anderson.
David Harwell asked if faculty at other institutions are likely to adopt this approach to teaching, given how much more time and effort it takes. Anderson replied that adopting a concepts-with-context approach for classes at other institutions will be a challenge. This is one reason that he and his colleagues are putting so much effort into developing the textbook and associated materials, which will be made available to students outside of Harvard for $15. Anderson asserted that even though this style of teaching is more difficult, “I cannot imagine going back and teaching it the old way.” Cardillo noted that faculty at other institutions have tried this course with the textbook that the Harvard team authored and he characterized the results as extraordinary. “It’s starting to catch on,” said Cardillo.
TODAY’S STUDENTS AND TOMORROW’S LEADERS: INTEGRATED CONCENTRATION IN SCIENCE
The motivating factor behind Scott Auerbach’s involvement in science education reform is that he does not believe that the nation is training its students to succeed in the important areas of science that are crucial to the future of the society. He and a group of colleagues from several academic departments at the University of Massachusetts have responded to this shortcoming by developing the Integrated Concentration in Science (iCons)1 program, where groups of students with diverse backgrounds work in teams to develop solutions to today’s major problems. The program’s mission, explained Auerbach, is “to produce the next generation of leaders in science and technology who have the attitudes, knowledge, and skills needed to solve the inherently multifaceted problems facing the world.”
In developing the iCons program, faculty developed a long list of desired student outcomes and organized them into a set of 10 learning goals. For example, the first goal is that students will be able to critically evaluate societal challenges and possible scientific solutions, and another goal is to develop quantitative understanding of societal problems and solutions. In general, the students do not have the ability TO discern the quantitative regime, and so an important aspect of iCons for faculty is to provide those skills. A third learning goal is for students to be able to design, carry out, and interpret valid scientific studies related to societal challenges. Connecting the dots to societal challenges—that is the key, said Auerbach.
After developing the learning goals, the iCons faculty realized that the skills that students develop as a result of achieving the learning goals are applicable outside of the classroom. That is, the skills may be interchangeable with the key cognitive, intrapersonal, and interpersonal skills that the NRC noted as being critical to success in life and work in the 21st century (NRC 2012). Auerbach and his colleagues were also gratified to see that their learning goals mapped onto the crucial elements of successful science, technology, engineering, and mathematics (STEM) programs identified by the President’s Council of Advisors on Science and Technology.
In practice, iCons is an 18-credit, 4-year concentration that does not replace the student’s major (see Figure 3-4). “Every student in the program is a major in some field of engineering or science,” explained Auerbach. “That major is the cake; iCons is the icing on that cake, and that icing is in the form of case studies, lab work, and research.” Every iCons project involves a case study (NRC 2011b) and has four essential elements: problem-based science and engineering (Gijbels et al. 2005), multidisciplinary student teams, student-driven collaborative learning, and reflection and self-assessment. As an example, he discussed a case study on high fructose corn syrup that was used to teach carbohydrate chemistry. The case study started with two articles in the popular press, one in 2010 that reported on a study showing that high fructose corn syrup promotes weight gain, the other 2 years later purporting that high fructose corn syrup is no worse than table sugar. The students were charged with getting to the bottom of this conflict, which involved learning not only about carbohydrates and carbohydrate metabolism, but also about the limitations of studies and how to design a new study that addresses those limitations. At the end of this case study, the students reflected on what they had learned, how they had learned it, and what they would do next in terms of gaining more knowledge on the subject and putting their ideas into practice.
As freshmen, iCons students learn about teamwork and take on numerous case studies. Prior to starting their second year, students choose a theme for their future work, either energy or biomedicine. As sophomores, they take theme-specific courses focusing on communication, reading, writing, speaking, and debating on the issues that are relevant to their chosen theme. In year three, students move into the laboratory and begin designing experiments using cutting-edge equipment (when relevant) to address real-world problems as part of a research group. As seniors, students will engage in an interdisciplinary research project, complete a portfolio, and write their honors thesis. iCons is currently in the third year of the program and the three cohorts include 130 students from 20 different majors from the colleges of engineering, science, and public health.
To determine whether iCons is meeting the learning goals, faculty have developed eight assessment instruments that pair three categories of assessment: formative/summative, qualitative/quantitative, and generic/targeted. So far, the program has implemented six of these instruments to assess iCons. For example, one weakness of iCons was found in the Student Response to Instruction Instrument (SRTI), which is a summative, quantitative, generic assessment tool. The SRTI showed that students were receiving insufficient feedback
___________________
FIGURE 3-4 The 4-year iCons program at the University of Massachusetts. SOURCE: Scott Auerbach.
on their performance in the class, and so the program has included more opportunities for feedback. Auerbach noted that the response signature of the assessments has been repeatable across the first two cohorts for the first iCons course. In terms of broader impact, there are now student-driven team projects in general chemistry, organic chemistry, physical chemistry, and integrative graduate education. Faculty who have participated in iCons also report that they have changed the way they teach their other courses. The take-home message is that education is not about “filling a bucket,” but about “lighting a fire” through context, Auerbach said. “Tell me and I forget. Teach me and I remember. Involve me and I learn.”
Jay Labov from the National Academy of Sciences asked if iCons is changing the culture at the University of Massachusetts in terms of promotion, tenure, and teaching. Auerbach replied that the program has not changed how departments view the metrics for tenure, and as a result, he would never ask a junior faculty member to teach in iCons until the minute after they received tenure.
After declaring his support for this type of program, John Kozarich of AxtiveX Bioscience asked about the rigor of the education that the iCons students receive. Auerbach replied that the goal of iCons is to instill passion in the students for a given idea that then prompts the students to drill down deeply into a subject. He also acknowledged that the program does face the challenge of balancing depth of learning with depth of exposure.
When asked how students fit these courses into their busy schedules, Auerbach said that the iCons courses are designed to substitute for existing courses. This is expensive, however, because the original courses still need to be taught for students that are not in iCons. “The only way this will work in the long term is if companies love the product that we have so much they are willing to give us money to continue training students in this way.” He noted, too, that there is a 20 percent dropout rate from the program because of scheduling conflicts. “Finding time to get students from 20 majors together at one time is a big barrier,” he said.
A participant asked if iCons is considering how to turn some of these student ideas into real-world activities. Auerbach said
that the plan is to work with the business school to develop a venture capital track over the next couple of years.
Working from the underlying theme that the chemical bond determines properties, and with an emphasis on linking basic concepts with applications, MIT’s unique first-year chemistry course teaches students fundamental chemical principles through the solid state. The hypothesis around which the course is based is that students will find it is easier to learn, understand, and, most importantly, use chemical principles if they can relate them to the solids around them. Michael Cima has now adapted this course, which has been offered as an option for meeting MIT’s general instructional chemistry requirements since the 1970s, to serve as a MOOC on the edX2 online platform. The course, known as 3.091x,3 is offered free of charge; includes homework, exams, and a final exam; contains the same intellectual content as the classroom-based MIT course on which it is based; and is a certificate-earning rather than credit-earning course. He added that the course is taken by a cohort of students that work together.
Cima explained that 3.091x is an engineering course, not a chemistry course, and that affects the way the students are tested. “We are assessing students not on what they know, but what they can do, and they do a lot of calculations, which turns out to be an advantage for an online course.” After demonstrating the class to the workshop, Cima noted that the experience of doing a screencast was much more interesting for him as a teacher than standing in front of a large lecture hall. He noted that classes on the edX platform are nothing like traditional online courses in that there is no 50-minute lecture. At most, each lesson consists of a 10-minute lecture segment on a single concept followed by self-assessment tools that are graded immediately and serve as a reality check for the student.
For MIT, the reason to offer an online version of an established course was simple: research consistently shows that learning outcomes are about the same for a residence-based course and an online version, but that when the versions are combined, the students do better on both courses. The goal, shared between MIT and Harvard, is to use both versions simultaneously, and Cima was going to do just that starting with the fall 2013 semester.
Course development took considerable resources in time and money. Cima explained that course preparations began in June 2012, 4 months prior to 3.091x’s launch on October 15, 2012. Producing the course took one teaching assistant working over the summer, one full-time and one half-time edX person, production and engineering staff time that totaled 2.5 full-time equivalents, a part-time administrative support person, five paid forum moderators to answer questions posted by students, four volunteer community teaching assistants, and two to five beta checkers. The lecture video derived mostly from his 2011 lecture class. Some 280 lecture segments, 65 screencasts, and six additional video segments were incorporated into the final version of 3.091x. “It was a huge amount of work, and I spent the bulk of the summer getting ready for this,” he said. Now the course is offered, he goes online himself most mornings and answers student questions. He noted, though, that the unpaid community teaching assistants, who are people taking the class in locations worldwide, have been an amazing resource. In fact, Cima has asked one of them—a high school chemistry teacher in the United Kingdom—to serve as a teaching assistant for this coming fall semester. Six of her high school students have also taken the class, he added.
When the course was offered, nearly 29,000 people registered for it. Over 3,400 took the first test, almost 2,200 took the second test, and 2,148 took the final exam. About 15,000 of the students were using the materials throughout the course and, on the basis of results of an exit survey, Cima thinks that the bulk of these people are taking a chemistry course and using 3.091x as a teaching supplement. Of those taking the course, 13 percent were graduate students, 28.9 percent were university students, 1 percent were community college students, and 9.7 percent were high school students, some of whom want to know what taking an MIT class entails. The biggest surprise, he said, was the large number of teachers—almost 9 percent—who took the course, and he has corresponded with many of them. Some 3 percent were K-12 teachers and over 5 percent were university or community college teachers. It may be feasible, he said, to offer these kinds of classes for professional development credits for high school science teachers.
In a retrospective look at outcomes, which he did by designing the residence-based final exam in such a way that he could do a select group of measurements with the online final, it appears that the online students outperformed the residence-based students. Cima believes this “troubling” finding resulted because it may be possible to do a better assessment of student learning in an online setting than in a classroom under time constraints. One of his goals for the future is to develop improved assessment tools.
Another task Cima faces is determining how best to maximize the benefits of having all of the developed content and assessment tools, particularly as he works to integrate the online and residence-based classes. He noted that the decision to integrate the two is highly political at MIT and he spent considerable effort building support and getting approval for this change. This coming fall, Cima is going
___________________
2 This nonprofit organization offers MOOCs and interactive online classes for a variety of subjects, including some STEM subjects. It was founded by individuals from Harvard University and MIT.
3 See https://www.edx.org/course/mit/3-091x/introduction-solid-state/591.
to conduct an experiment that will consist of replacing the course texts with the online content and developing lectures to take advantage of the new “text.” The course will be structured around two 1-hour recitations per week, and assessment will consist of 37 proctored online quizzes that the student will complete within a specified time window. Each quiz will represent a learning outcome measure, and if a quiz is not answered correctly, the student may take the quiz again and for as many times as they want within the time window. There will be a 24-hour lockout between quiz attempts. Each learning outcome quiz will be selected randomly from a group of many related problems. Cima noted that he used this quiz format with the last midterm of the online class without any problem.
A participant asked what the implications of MOOCs might be for university education, and Cima said that while he has no real idea, it could be that the freshman year is spent off campus taking their required foundational courses online. He was then asked if the videos will still be fresh in 5 years or if the course will have to be continually redesigned at significant expense. Again, Cima replied that he did not know what the future held but noted that it would be easy and relatively inexpensive to replace videos with screencasts. He also said replacing content is simple once the course is constructed, claiming that he can use a new software tool to replace content from his desktop computer in 5 minutes.
A participant asked if MIT was considering faculty diversity in its plans to create a catalog of MOOCs. Cima said that the university takes this challenge very seriously since the MOOCs do represent the face of the university and that face has changed considerably over the past 30 years. He noted that the introductory mechanical engineering MOOC is taught by two female faculty.
Wesemann asked if the positive experience he had creating 3.091x was having an effect on other faculty. Yes, said Cima, and in fact, each engineering department is hiring a person who will be dedicated to helping faculty convert their courses to an online format for use by residence-based students. In response to a question from Cardillo about other companies offering online courses, Cima said that his impression, as well as that of other faculty he has talked to, is that there is a wide diversity in terms of the quality of these courses. What he likes about the edX format is that it is based on an open-source system and developers are taking advantage of that to develop assessment tools for community use.
DEALING WITH RISK, FAILURE, AND UNCERTAINTY
The challenge that Jeffrey Moore is tackling at the University of Illinois with his instructional experimentation is to center instruction on the individual learner in what he called the “kilostudent” organic chemistry classroom consisting of a diverse set of nonchemistry majors who, in most cases, are taking their last formal course in chemistry. His approach to meeting that challenge has been to reconfigure the learning outcomes of the course to match the grand challenges in science education enumerated in the 2013 special issue of the journal Science (McNutt 2013) and to design a curriculum that uses theory webcasts to present concepts, pressure-point problems to immerse the student in experience, problem-of-the-day discussions, and peer-to-peer tutoring.
One of the key features of the course is its use of online electronic homework, which for organic chemistry works out wonderfully with machine-read computer drawings that can be automatically graded using a programmed graphical language. There is no textbook in the course, though there is a set of course notes that accompany the 5-minute webcasts that the students view before attending a discussion session. The discussion sessions are held in a computer lab that can accommodate 55 students at a time, or via an Internet connection at two times daily. At the end of the discussion session, the students are presented with a pressure-point problem that they have 5 minutes to solve. Successfully answering the question yields bonus points that are applied to the next exam.
These complex problems, explained Moore, are designed to take students into uncharted territory and force them to take risks and fail, just as scientists do in the real world (see Figure 3-5). He called this “taking off the training wheels,” and said it teaches students about failure and how to respond to that failure. The problems are nonalgorithmic and multifaceted, and multiple steps are involved in solving them. Students are forced to use creative processes to generate a variety of initial-guess solutions and to develop a strategy to initiate a solution. They are also allowed to access the literature or any other online resource—except communicating with another person—to solve the problem.
FIGURE 3-5 Learning through experiences that mimic the real world. SOURCE: Jeffrey Moore.
FIGURE 3-6 Graphical tracking of exam progress. SOURCE: Jeffrey Moore.
No points are given for an incorrect answer, but students are allowed to continue trying to solve the problem until they get the correct answer. Because the students are developing and refining their solutions online, the computer can track the progress they are making toward the solution. When plotted, these data provide a picture of how the exam is going and how each student is doing in real time (see Figure 3-6).
One of the main components of Moore’s course is peer-to-peer tutoring aided by the Internet. “We realized through some of our assessments that one of the most important ways that students were learning was not by using the videos, not even by the problems that they were doing, but by the interactions that they were having with other students,” said Moore. The way this worked was when a student answered a problem correctly, they would be placed in the tutoring pool for that problem. Students who answered a problem incorrectly and who wanted help would then go to the tutoring pool, where they could select a peer tutor to help them. At the end of the process, both students then recorded video reflections of the outcome of that tutoring experience.
“We do not monitor every single one of those videos, but we do keep track of the clusters of people and the information flow, the diffusion of information,” he explained. Using this information, Moore can create a network map of tutor–tutee interactions (see Figure 3-7). One of the surprises that came out of the analysis of the video reflections was how much value the students placed on peer-to-peer tutoring from both the tutor and the tutee. Watching video lectures online was deemed the least valuable method of learning by the students.
Moore has also implemented a semester-long group project in which four-student teams select a small organic molecule, typically a bioorganic molecule, and use the literature to propose a mechanism for how this molecule might be made, its properties, and other relevant information. The idea behind this project is to promote the development of professional scientific skills, and it appears to produce gains in literature searching, scientific writing, and critical reading, though there is evidence that sustaining these gains requires what he called a “super teaching assistant” or an intensely devoted instructor.
In summary, Moore said that this revamping has successfully flipped the two-semester organic sequence without significantly increasing the load of the teaching assistants. “I can say from the data we’ve collected that we’ve done no harm,” he said. “We have not improved things in terms of
FIGURE 3-7 Tutor—tutee interaction network. Arrows illustrate the hypothetical transfer of knowledge from tutor to tutee, and the nodes are scaled by how central they are to the network. SOURCE: Jeffrey Moore.
these learning outcomes and objectives, but at the same time, learning is robust.” In his view, machine-graded homework, discussion problems, daily pressure-point problems, and exam problems offer students many more opportunities for practice with feedback, and that testing with real-time feedback might measure students’ ability to diagnose their errors in a pressure situation. He reiterated that peer-to-peer tutoring appears to be a valuable tool for student learning and added that it requires little effort to implement.
Jeffrey Reimer asked if the students are aware that whatever they do for this course is analyzed and therefore not private. Moore replied that he has received institutional review board approval and that the students agree to participate in the data collection and analysis process. Session chair Emilio Bunel asked if other faculty were involved in teaching this course. Moore said he did have one colleague involved who has since moved to another university, but his impression is that faculty are intimidated by the amount of work that went into developing the course. There is an instructor for the first-semester organic chemistry class who has completely embraced this approach and has abandoned the traditional lecture approach.
Luis Martinez from Rollins College asked about the importance of training for the teaching assistants and if their experience teaching in this format had any impact on their development as future teachers. Moore said that he is careful about selecting teaching assistants for this course who feel comfortable with technology. He has not yet assessed what the teaching assistants are getting from this experience but he thought that would be a good idea to look at going forward. What he has heard from the students who take the class is that they value how much they learn from acting as teachers in the peer-to-peer tutoring process and how much they now value teaching.
WHAT GETS MEASURED IS WHAT GETS LEARNED: ASSESSING STUDENT UNDERSTANDING
One of the problems in the way students are taught science today lies with the way student learning is assessed, said Angelica Stacy. Legacy chemistry exams, she stated, are designed to select for those students who are good at memorizing. “Teachers are a good delivery system if all you want students to do is memorize,” she said. “Teachers ask questions on exams that students can answer, and everyone feels good.” The result? “If you don’t assess what is important, what is assessed becomes important.” Stacy quoted Sir Ken Robinson, an internationally recognized author and leader in education, who said that the way most classes are taught today is turning education into “hours of low-grade clerical work.”
Stacy is optimistic that this sorry state can change because of the new climate of possibilities that starts with the Next Generation Science Standards (NGSS)4 that talk about science practices, crosscutting content, and core ideas. Evidence of coming changes include the new Advanced Placement tests that the College Board is releasing and the redesigned MCAT2015 exam. She echoed the sentiment shared by other individuals at the workshop that teaching methods should be revamped based on the growing body of research that shows how students learn.
At UC Berkeley, Stacy and her colleagues have started writing introductory chemistry course exams differently, and they conducted a study to determine the effect of these changes. As part of this study, they asked students at three times during the semester what they were doing outside of class to prepare for the four exams given during the semester to better understand how they were using resources when they were directing themselves and how they changed their study strategies as they realized that the tests were less about memorization and more about understanding and applying concepts.
The students fell into four groups. She calls the lowest-performing group the fact gatherers—the students who memorize independent facts and then are confused by their low grades given the amount of time they put into their studies. The next group, which performed slightly better, learned procedures—they absorbed information and made small connections, but still relied on others or course materials for answers. Students in the third group work at confirming their understanding. They evaluate information and question why, work more independently as learners, and try to give explanations. These students are trying to understand if they are thinking about a problem correctly. The fourth group, which Stacy characterized as amazing, thinks about chemistry all the time. These students are applying ideas, taking in information and questioning why it is true and how it helps to explain the world around them.
Unfortunately, said Stacy, most of the bright, motivated, talented students who come to the University of California, Berkeley, fall into the first two categories. Over the course of the semester, the numbers do shift, with more students moving into group three and a few rising into group four. The big shifts, she noted, do not happen until after the second exam when the students realize the first exam was not a fluke. Students who improved became more active learners and they moved away from just reading the course textbook and started working with peers and asking more questions. Students who did not improve remained passive learners who stay focused on reading text and were not reflective when studying. The students reported that they made changes to their study habits because the exam questions made them apply what they were learning—they could not memorize
___________________
4 The NGSS are new K-12 science standards to provide students an internationally benchmarked science education. The NGSS are based on the NRC report Framework for K-12 Science Education (NRC 2011a).
an algorithm to solve the questions and expect to do well on the exam.
Stacy acknowledged that exam design is challenging when teaching large lecture classes. For example, she has 1,300 students to teach and a limited number of graders to help her. “The trick is, can we start to learn how to write better multiple-choice questions,” she said. One approach is to use data and observations as the basis for test questions and then structure the classroom experience to develop the skills to explain data, to find patterns, and to understand how to control variables when comparing different pieces of data. She and her colleagues have designed their chemistry curriculum around core ideas in chemistry that are similar to those in the Next Generation of Science Standards: matter, change, energy, and light. She described several examples, including one that uses smells to explore molecular structure and properties. Stacy presented her students with a table with properties of four chemicals found in spices with strong smells or tastes: vanillin, eugenol, zingerone, and capsaicin. The properties given were flavor, molecular formula, structural formula, melting point, boiling point, and water solubility. Students were also provided with space-filling models of the molecules. Along with this information, to the students, they were asked practical questions like “why in the world can I smell vanillin so well?” and “I can’t smell capsaicin, but when it gets into my mouth it doesn’t go away. Why is that?” The exam questions are designed to combine data and observations in such a way that the students must use core concepts in chemistry, like intermolecular attractions, to explain the data.
In closing, Stacy argued that the field needs to do research-based redesign of undergraduate chemistry courses. “I think it begins with the assessments,” she said. “They have to promote understanding as opposed to memorization or we’re really not getting to where we want to be. We have to use students’ ideas and experiences to build knowledge. Let them observe. Let them explore the data. You’ll be amazed at what ideas they do come up with.”
YuYe Tong from Georgetown University remarked that this work shows the value and challenges of moving students from being passive to active learners and of emphasizing concepts over content. Sarah Green from the Michigan Technology University commented that it will be interesting to see what the impact will be of the NGSS, which stress concepts and problem solving as opposed to rote memorization, as those students enter the undergraduate chemistry curriculum. Stacy noted that one effect of using these new course designs to teach college students is that they will become the next generation of teachers and professors, creating what could be a virtuous cycle in science education.
In response to a question from Cardillo about the extent to which these innovative courses have been adopted by other faculty, Stacy said that she has been joined by two of her colleagues and they now coteach this course, where they trade off lectures and critique each other. Other faculty have seen how enjoyable it is to teach chemistry in this manner and are working to change their course design too, noted Stacy. Anderson added that he has strong support from his department for continuing to revamp the introductory physical sciences class, but that other faculty are still in wait-and-see mode because of the amount of work that he and Kaxiras have had to put into both redesigning the course and developing the accompanying text and other course materials.
