Innovations and Barriers

“Assessment and evidence are critical components of innovation adoption.”

Thomas Holme

“It doesn’t matter whether [students] are going into law, business, economics, or international politics—they have to have a sense of where these technical forces are and how they’re shaping the world.”

James Anderson

The predominant focus of this workshop was to identify the barriers to improving chemistry education and to highlight innovative approaches to overcome the barriers. Seven speakers presented on a range of topics to enhance the learning experience, including key requirements for education reform and innovative approaches that move away from standard lecture and testing formats; those presentations are summarized in this chapter. Thomas Holme, of Iowa State University and director of the American Chemical Socieity (ACS) Exams Institute, spoke about some of the challenges in replicating education reform efforts. Clark Landis of the University of Wisconsin, Madison, presented an example of large-classroom reforms and the challenges with reform assessments using traditional metrics. A new approach integrating chemistry and physics curricula into introductory courses was discussed by James G. Anderson of Harvard University. Scott Auerbach of the University of Massachusetts, Amherst, described an approach to undergraduate education that puts context front and center in course design. Michael Cima of Massachusetts Institute of Technology detailed his work developing a massive open online course (MOOC) on solid-state chemistry. Jeffrey Moore talked about the student-centered organic chemistry class he developed at the University of Illinois in Urbana-Champaign. Last, how the design of exams can influence how and what students learn in their chemistry classes was discussed by Angelica Stacy of the University of California, Berkeley.


What are the barriers that keep successful chemistry education reforms from spreading beyond a local college or university? The fundamental tension, said Thomas Holme, is that teaching is both a personal experience and a corporate enterprise. “It’s a personal activity to each of us, but a corporate enterprise because other people care how we do it,” explained Holme. He added that anyone who has written a textbook from scratch has experienced this tension. This same tension arises when it comes to educational reforms—everyone is interested in a very broad way in propagating successful new methods for teaching chemistry, but such methods need to appeal to individual teachers. “Enthusiasm can get us in the door,” but it must be teamed with an assessment in order for education reform to be effective, argued Holme.

There are resources available to help with education reform efforts. A National Research Council (NRC) report, How People Learn: Brain, Mind, Experience, and School (NRC 2000), provides a good foundation and notes the importance to the learning process of prior knowledge, whether that knowledge is correct or not, and metacognition, the ability to reflect on one’s own thinking and learning processes. Holme pointed out that a fairly robust albeit young field of chemistry education research was among the disciplines that contributed to a 2012 NRC Discipline-Based Educaton Research (DBER) report from the Board on Science Education. The DBER report also stresses the importance of prior knowledge as an obstacle to teaching chemistry. Chemistry students, for example, have a difficult time envisioning the particulate nature of matter, and this difficulty is often associated with the way it is presented (Cooper et al. 2010). Student conceptions of size scale (e.g., nucleus, atom, molecule, and compound) also present a major challenge in chemistry, and evidence suggests that the ability to understand scale may be the best predictor of success in general chemistry (Gerlach et al. 2011). Unfortunately, chemistry educators are to blame for at least part of this difficulty, said Holme.

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 15
3 Innovations and Barriers ”Assessment and evidence are critical components of innovation adoption.” Thomas Holme “It doesn’t matter whether [students] are going into law, business, economics, or international politics— they have to have a sense of where these technical forces are and how they’re shaping the world.” James Anderson The predominant focus of this workshop was to iden- is that teaching is both a personal experience and a corpo- tify the barriers to improving chemistry education and to rate enterprise. “It’s a personal activity to each of us, but a highlight innovative approaches to overcome the barriers. corporate enterprise because other people care how we do Seven speakers presented on a range of topics to enhance the it,” explained Holme. He added that anyone who has written learning experience, including key requirements for educa- a textbook from scratch has experienced this tension. This tion reform and innovative approaches that move away from same tension arises when it comes to educational reforms— standard lecture and testing formats; those presentations are everyone is interested in a very broad way in propagating summarized in this chapter. Thomas Holme, of Iowa State successful new methods for teaching chemistry, but such University and director of the American Chemical Socieity methods need to appeal to individual teachers. “Enthusiasm (ACS) Exams Institute, spoke about some of the challenges can get us in the door,” but it must be teamed with an assess- in replicating education reform efforts. Clark Landis of the ment in order for education reform to be effective, argued University of Wisconsin, Madison, presented an example Holme. of large-classroom reforms and the challenges with reform There are resources available to help with education assessments using traditional metrics. A new approach inte- reform efforts. A National Research Council (NRC) report, grating chemistry and physics curricula into introductory How People Learn: Brain, Mind, Experience, and School courses was discussed by James G. Anderson of Harvard Uni- (NRC 2000), provides a good foundation and notes the versity. Scott Auerbach of the University of Massachusetts, importance to the learning process of prior knowledge, Amherst, described an approach to undergraduate education whether that knowledge is correct or not, and metacogni- that puts context front and center in course design. Michael tion, the ability to reflect on one’s own thinking and learn- Cima of Massachusetts Institute of Technology detailed his ing processes. Holme pointed out that a fairly robust albeit work developing a massive open online course (MOOC) young field of chemistry education research was among on solid-state chemistry. Jeffrey Moore talked about the the disciplines that contributed to a 2012 NRC Discipline- student-centered organic chemistry class he developed at the Based Educaton Research (DBER) report from the Board University of Illinois in Urbana-Champaign. Last, how the on Science Education. The DBER report also stresses the design of exams can influence how and what students learn importance of prior knowledge as an obstacle to teaching in their chemistry classes was discussed by Angelica Stacy chemistry. Chemistry students, for example, have a difficult of the University of California, Berkeley. time envisioning the particulate nature of matter, and this difficulty is often associated with the way it is presented (Cooper et al. 2010). Student conceptions of size scale (e.g., PROPAGATING MEANINGFUL REFORM IN nucleus, atom, molecule, and compound) also present a CHEMISTRY EDUCATION AND THE RELATIVE ROLES major challenge in chemistry, and evidence suggests that OF ENTHUSIASM AND EVIDENCE the ability to understand scale may be the best predictor of What are the barriers that keep successful chemistry success in general chemistry (Gerlach et al. 2011). Unfortu- education reforms from spreading beyond a local college or nately, chemistry educators are to blame for at least part of university? The fundamental tension, said Thomas Holme, this difficulty, said Holme. 15

OCR for page 15
16 UNDERGRADUATE CHEMISTRY EDUCATION The DBER report (NRC 2012) also talks about how to This is where the ACS Exams Institute can help. The use educational research to impact real-world teaching. The ACS Exams Institute writes nationally standardized exams report acknowledges that it is hard to turn basic research into covering all fields of chemistry and provides resources for interventions and it is harder still to turn local changes into outcomes measurement. Recently, the Exams Institute, with larger scale change. It also notes that while the National Sci- funding from the NSF, conducted a national survey of 14,000 ence Foundation (NSF) has funded professional development professors and instructors in the United States to assess their activities related to teaching for some time, those activities understanding of assessment terminology and techniques. have largely been self-selecting, leading to a “preaching to Holme described the survey and the statistical methods used the choir” effect (Feuer et al. 2002). The good news, Holme to analyze the responses. The 1,500-plus survey respondents explained, is that the chemistry community as a whole is fell into six clusters of understanding. General familiarity good at thinking about the diffusion of innovative education with assessment terms was not high across the six clusters, reforms, and this is where the relative roles of enthusiasm although analytical chemists tended to score higher than and evidence come into play. other groups in understanding statistical terms and methods. The classic book on this subject is Diffusion of Innova- The lesson here, said Holme, is that everyone needs to be tion by Everett Rogers (2003), said Holme, who spent a few careful when conducting assessments; they need to truly minutes summarizing some of the key ideas in the book. The understand what they are doing and what the results are basic definition of diffusion of innovation is “the process telling them in terms of evidence for whether an innovative in which an innovation is communicated through certain chemistry education method works. channels over time among the members of a social system.” Holme noted, too, that sampling in assessment surveys Communication, said Holme, involves convincing somebody remains a challenge because most studies of educational who is listening for evidence about key innovation character- innovation use convenience samplings of the students who istics such as relative advantage, compatibility, complexity, come to the course. Studies also tend to build in bias because trialability (can a new technology be implemented in steps), disaffected students leave the course before data are col- and observability. If evidence for these characteristics is not lected, an issue of particular concern for those who study provided, listeners will make up their own evidence and MOOCs, for example. He remarked that institutional review likely conclude that the status quo is better. boards, which become involved with research on human Given that an implicit goal is to convince somebody that subjects, place an important constraint on building mean- a new educational innovation is worth adapting, it is impor- ingful control-based experiments. “If we know something tant to understand the five stages of innovation adoption. is fundamentally better for students, it is unethical to train “At the risk of sounding unduly mercantile,” said Holme, some of our students with something we know is not good,” “we need to understand our targeted customer.” The five explained Holme. stages are Responding to a question from Angelica Stacy of UC Berkeley, Holme acknowledged that it is unlikely to ever 1. Knowledge of the innovation have enough evidence to prove an educational innovation 2. Persuasiveness of the innovation—is it better for me? is effective, but that it should be possible to have enough 3. Decision—adopt or reject evidence to take wise action as to where to go with an inno- 4. Implementation—adapt and adopt vation. Auerbach then asked what is known about assess- 5. Confirmation—to keep or not to keep ing process-oriented, laboratory-type courses, and Holme responded that there are resources available on this subject Enthusiasm, Holme said, plays a key role in the first and added that assessing those courses is more challenging. two stages. Evidence is key in the second and third stages. He added, too, that the ACS Exams Institute has the resources Assessment is critical for the final two stages. But, Holme to build assessment tools to help those who are developing cautioned, each of these stages takes place in a social system educational innovations, but that it will take time to develop that is not exchangeable and not particularly amenable. In tools that the ACS feels are good enough to stamp with its addition, university faculty will not just be skeptical in the imprimatur. face of enthusiasm, but they will “jump in with their skep- ticism at every opportunity.” It is important to remember, LARGE-CLASSROOM REFORMS: Holme added, that “our colleagues may, or may not, know FIVE BEST TEACHING PRACTICES how to decide about the efficacy of educational innovations they try, but they probably believe they know.” This is why The observation that certain groups of students are assessment and evidence are critical components of innova- underperforming in introductory or “gateway” chemistry tion adoption. The problem, however, becomes one of get- courses has been an important driver for reforming under- ting the data that will serve as evidence in the context of the graduate chemistry education at the University of Wisconsin, academic social system. Madison, explained Clark Landis. Another driver is the desire

OCR for page 15
INNOVATIONS AND BARRIERS 17 to increase the fraction of students successfully completing From fall semester 2009 through spring semester 2011, introductory chemistry and who will then take the second Landis and Sibert compared student performance between semester of the two-course chemistry sequence. Landis and a reformed and a traditional chemistry course. The study his colleague Ned Sibert also hoped to make teaching intro- included 189 students in targeted groups and 1,680 stu- ductory chemistry more exciting for themselves. In addition, dents in nontargeted groups in the reformed courses, and UW-Madison compiled student data in 2008 and discovered 170 targeted students and 1,333 nontargeted students in the a gap in terms of adverse events, which Landis defined as get- traditional sections. Assessment elements included grades ting a grade below C or dropping the class, between targeted and retention; six common questions on the final exam; one minorities and the general, nontargeted student population common essay question; and student surveys of hours spent (see Figure 3-1). outside of class, course perceptions, and student assessment Landis described the comprehensive course reform that of learning gains. They also monitored gender-based and he and Sibert designed based on what they call the “five target group achievement gaps (Seymour et al. 2000). best teaching practices” that influence student success in The results surprised Landis. The reformed course did not college courses: learning in context, group-based learning, appear to affect the achievement gap for targeted students, increased time on a task, increased frequency of feedback, the fraction of adverse outcomes, or performance on com- and a positive classroom climate (Brower 2009, Cabrera mon final exam questions or the essay question. Students and La Nasa 2005, Treisman and Surles 2001, University in the reform sections did perceive a greater emphasis on of Wisconsin–Madison 2013). These reforms include using collaborative and conceptual learning, worked and discussed concept tests and clicker questions in lectures as ways of more outside of class, attended class more often, and related making lecture sections more interactive. Online homework, chemistry more to daily life. They were also more confident tutorials, videos, and simulations are important elements of in their problem-solving skills. the new Chemistry 103 course design, as are peer-led teach- These surprising results led Landis to ask, “Should we ing and review. The new course includes spiral curricula, the evaluate performance differently?” He cited work from the idea that it is possible to introduce many concepts at a fairly New Traditions Project, conducted some 15 years ago, show- superficial level early on and then return to those concepts ing that students who had been in active-learning sections regularly, developing them in greater depth each time the performed no better than students in traditional sections concepts are discussed during the semester. Landis explained when tested using standard written exams. However, this that spiral curricula work well in conjunction with big, real- study found that “almost uniformly, the students that were world problems. in the active learning class were assessed as being better The primary focus of these changes was to promote active performing in oral exams than the students in the traditional learning in the context of a course that as many as 2,100 stu- class,” explained Landis. “It could be that we just do not dents take each fall. The structure of the course includes three have good ways of assessing effectiveness of these methods.” lectures, two discussion sections, and 2 hours of laboratory Landis also noted that a study conducted at the University instruction each week. Inquiry-based cooperative learning of Colorado showed that the gender gap in a second- or third- activities centered on group-oriented challenge problems year physics class disappeared in the course of writing two are used in the discussion sections and in voluntary evening 15-minute essays on how a student’s values are related to workshops. the course. However, when Landis and Sibert conducted the same experiment, they found no differences in performance. He concluded his remarks by saying, “We think value is added by the reformed classes, but we just are not capturing that value in our standardized assessments.” William Tolman commented that he and his colleagues at the University of Minnesota have done similar experi- ments in organic chemistry class design and also found that performance as measured on written tests did not improve, but that attendance and both student and teacher satisfaction improved. One area that might be showing improvement, he said, is in discovery-based team learning in the organic chemistry laboratory course, though the results are still preliminary. Scott Auerbach thought that one problem with assessing these new teaching methods is that they have dif- ferent sets of learning goals that the standard assessment FIGURE 3-1  Gap in rates of adverse outcomes for students across a range of ACT scores. SOURCE: Clark Landis. techniques are not designed to capture. Landis agreed and

OCR for page 15
18 UNDERGRADUATE CHEMISTRY EDUCATION noted that there is a good chemistry concept inventory being he acknowledged, the students at Harvard, MIT, and Caltech developed that he is eager to use as an assessment tool. are somewhat different—there is a general pattern common Anne McCoy wondered if differences might start show- to all: attrition from the sciences during and following the ing up in later courses, where students who had taken the freshman year (see Figure 3-2). reformed classes might show better retention of the concepts For the first few decades after World War II, only 10 per- they learned in the new chemistry sections. Landis said that cent of entering undergraduates completed their baccalaure- he would like to run such longitudinal studies, but funding ate degrees in science, which met the demands of graduate is an issue. He reiterated the need for longer-term support and medical schools. Today, however, “this zone of scientific when Jodi Wesemann asked what was needed to keep these and technical illiteracy has now become a fundamental reform efforts going. problem that we have to deal with because of these issues of Jeffrey Reimer from the University of California, Berkeley, national security and competitive economic considerations asked if anyone had conducted studies comparing sections on a global scale,” said Anderson. The problem of driving taught by white male faculty and those taught by targeted scientific and technical literacy to a level where 95 percent minority faculty. Landis said he did not know of any work of graduates are scientifically and technically literate is what in that area but noted that he and his colleagues found no prompted Harvard to completely revamp freshman chemis- difference in performance between students taught by male try, he said. versus female faculty. The current strategy in introductory chemistry, said Anderson, is to present lectures and text material that covers the basic formalism and theory, followed by problem sets and TEACHING INTRODUCTORY CHEMISTRY WITH A exams. Solid evidence shows, however, that there are two MOLECULAR AND GLOBAL PERSPECTIVE basic failures with this “formalism first” approach to teach- James Anderson described Harvard University’s efforts to ing. First, he said, it results in “disembodied knowledge”— infuse chemistry and physics into an introductory chemistry students cannot attach the knowledge to a context or their course. Their goal is not to recruit more chemistry majors, past experience, and so it is largely meaningless symbols and said Anderson, but to develop a chemistry curriculum that is evolving to keep more students engaged in the physical sciences as a whole and to make the rest of the student population more aware of and appreciative of the physical sciences. The reason for that emphasis, he explained, is that the physical sciences are playing a central and critical role in solving the major problems facing human civilization today. Harvard’s Physical Sciences 11, Foundations and Fron- tiers of Modern Chemistry: A Molecular and Global Per- spective, reflects the idea that introductory courses in both chemistry and physics have been taught separately and with- out a compelling context. The Physical Sciences 11 course is based on the premise that decisions on what university graduates face in their academic career directly relate to what they take in their freshmen year. “If the separation between science and society occurs in the freshman year, it’s irrevers- ible for that generation,” said Anderson. Chemistry and physics faculty are both to blame for the lack of appreciation for and understanding of the physical sciences because the courses they teach create an exclusive club of students who can excel at these subjects instead of an inclusive group of students who understand the basic concepts of the physical sciences. In contrast, said Anderson, the life sciences have clearly demonstrated how important they are in the larger context of society. He explained that by context he meant linking the essential concepts of chem- istry and physics to their connection with the big problems that intrigue students today—energy, human health, national security, climate change, and others. He noted that while FIGURE 3-2 Attrition of undergraduate students from science there are significant differences among universities—yes, majors. SOURCE: James Anderson.

OCR for page 15
INNOVATIONS AND BARRIERS 19 facts to memorize. Second, as Nobel laureate and noted sci- flow, electromagnetism, and chemical transformation when ence educator Carl Wieman has shown, knowledge obtained it is connected to a context of the electric car. In writing this way is filed away in the brain in a separate compartment the textbook for this course, Anderson and Kaxiras used and building links to that compartment after the fact is much case studies that are broken into their parts, analyzed, harder and less effective than if it had been filed correctly and reassembled. When Anderson gave the first lecture from the start. of Physical Sciences 11, 25 students were present. By the Anderson and his collaborator, Harvard physics profes- fourth lecture, there were 125 students in the lecture hall. sor Efthimios Kaxiras, have taken an approach to address- A year later, around 300 students had completed the class, ing these problems with a strategy that links concepts with which is notable because Physics 11 is a considerably more context (see Figure 3-3). As an example, electrochemistry difficult course than the alternative class the students can is a great way to teach about Gibbs free energy, electron take, Anderson said. FIGURE 3-3  Linking concepts to context is the guiding principle for a new introductory physical science course taught at Harvard Univer- sity. SOURCE: James Anderson.

OCR for page 15
20 UNDERGRADUATE CHEMISTRY EDUCATION David Harwell asked if faculty at other institutions are 21st century (NRC 2012). Auerbach and his colleagues were likely to adopt this approach to teaching, given how much also gratified to see that their learning goals mapped onto the more time and effort it takes. Anderson replied that adopt- crucial elements of successful science, technology, engineer- ing a concepts-with-context approach for classes at other ing, and mathematics (STEM) programs identified by the institutions will be a challenge. This is one reason that he President’s Council of Advisors on Science and Technology. and his colleagues are putting so much effort into developing In practice, iCons is an 18-credit, 4-year concentration the textbook and associated materials, which will be made that does not replace the student’s major (see Figure 3-4). available to students outside of Harvard for $15. Anderson “Every student in the program is a major in some field of asserted that even though this style of teaching is more dif- engineering or science,” explained Auerbach. “That major ficult, “I cannot imagine going back and teaching it the old is the cake; iCons is the icing on that cake, and that icing is way.” Cardillo noted that faculty at other institutions have in the form of case studies, lab work, and research.” Every tried this course with the textbook that the Harvard team iCons project involves a case study (NRC 2011b) and has authored and he characterized the results as extraordinary. four essential elements: problem-based science and engi- “It’s starting to catch on,” said Cardillo. neering (Gijbels et al. 2005), multidisciplinary student teams, student-driven collaborative learning, and reflection and self-assessment. As an example, he discussed a case study on TODAY’S STUDENTS AND TOMORROW’S LEADERS: high fructose corn syrup that was used to teach carbohydrate INTEGRATED CONCENTRATION IN SCIENCE chemistry. The case study started with two articles in the The motivating factor behind Scott Auerbach’s involve- popular press, one in 2010 that reported on a study showing ment in science education reform is that he does not believe that high fructose corn syrup promotes weight gain, the other that the nation is training its students to succeed in the 2 years later purporting that high fructose corn syrup is no important areas of science that are crucial to the future of worse than table sugar. The students were charged with get- the society. He and a group of colleagues from several aca- ting to the bottom of this conflict, which involved learning demic departments at the University of Massachusetts have not only about carbohydrates and carbohydrate metabolism, responded to this shortcoming by developing the Integrated but also about the limitations of studies and how to design a Concentration in Science (iCons)1 program, where groups of new study that addresses those limitations. At the end of this students with diverse backgrounds work in teams to develop case study, the students reflected on what they had learned, solutions to today’s major problems. The program’s mission, how they had learned it, and what they would do next in terms explained Auerbach, is “to produce the next generation of of gaining more knowledge on the subject and putting their leaders in science and technology who have the attitudes, ideas into practice. knowledge, and skills needed to solve the inherently multi- As freshmen, iCons students learn about teamwork and faceted problems facing the world.” take on numerous case studies. Prior to starting their second In developing the iCons program, faculty developed a year, students choose a theme for their future work, either long list of desired student outcomes and organized them energy or biomedicine. As sophomores, they take theme- into a set of 10 learning goals. For example, the first goal is specific courses focusing on communication, reading, writ- that students will be able to critically evaluate societal chal- ing, speaking, and debating on the issues that are relevant lenges and possible scientific solutions, and another goal is to their chosen theme. In year three, students move into the to develop quantitative understanding of societal problems laboratory and begin designing experiments using cutting- and solutions. In general, the students do not have the abil- edge equipment (when relevant) to address real-world prob- ity TO discern the quantitative regime, and so an important lems as part of a research group. As seniors, students will aspect of iCons for faculty is to provide those skills. A third engage in an interdisciplinary research project, complete a learning goal is for students to be able to design, carry out, portfolio, and write their honors thesis. iCons is currently in and interpret valid scientific studies related to societal chal- the third year of the program and the three cohorts include lenges. Connecting the dots to societal challenges—that is 130 students from 20 different majors from the colleges of the key, said Auerbach. engineering, science, and public health. After developing the learning goals, the iCons faculty To determine whether iCons is meeting the learning goals, realized that the skills that students develop as a result of faculty have developed eight assessment instruments that achieving the learning goals are applicable outside of the pair three categories of assessment: formative/summative, classroom. That is, the skills may be interchangeable with the qualitative/quantitative, and generic/targeted. So far, the key cognitive, intrapersonal, and interpersonal skills that the program has implemented six of these instruments to assess NRC noted as being critical to success in life and work in the iCons. For example, one weakness of iCons was found in the Student Response to Instruction Instrument (SRTI), which is a summative, quantitative, generic assessment tool. The SRTI 1 See http://www.cns.umass.edu/icons-program/. showed that students were receiving insufficient feedback

OCR for page 15
INNOVATIONS AND BARRIERS 21 FIGURE 3-4  The 4-year iCons program at the University of Massachusetts. SOURCE: Scott Auerbach. on their performance in the class, and so the program has After declaring his support for this type of program, John included more opportunities for feedback. Auerbach noted Kozarich of AxtiveX Bioscience asked about the rigor of the that the response signature of the assessments has been education that the iCons students receive. Auerbach replied repeatable across the first two cohorts for the first iCons that the goal of iCons is to instill passion in the students for a course. In terms of broader impact, there are now student- given idea that then prompts the students to drill down deeply driven team projects in general chemistry, organic chemis- into a subject. He also acknowledged that the program does try, physical chemistry, and integrative graduate education. face the challenge of balancing depth of learning with depth Faculty who have participated in iCons also report that they of exposure. have changed the way they teach their other courses. The When asked how students fit these courses into their busy take-home message is that education is not about “filling a schedules, Auerbach said that the iCons courses are designed bucket,” but about “lighting a fire” through context, Auer- to substitute for existing courses. This is expensive, however, bach said. “Tell me and I forget. Teach me and I remember. because the original courses still need to be taught for stu- Involve me and I learn.” dents that are not in iCons. “The only way this will work in Jay Labov from the National Academy of Sciences the long term is if companies love the product that we have asked if iCons is changing the culture at the University of so much they are willing to give us money to continue train- Massachusetts in terms of promotion, tenure, and teaching. ing students in this way.” He noted, too, that there is a 20 Auerbach replied that the program has not changed how percent dropout rate from the program because of schedul- departments view the metrics for tenure, and as a result, he ing conflicts. “Finding time to get students from 20 majors would never ask a junior faculty member to teach in iCons together at one time is a big barrier,” he said. until the minute after they received tenure. A participant asked if iCons is considering how to turn some of these student ideas into real-world activities. Auerbach said

OCR for page 15
22 UNDERGRADUATE CHEMISTRY EDUCATION that the plan is to work with the business school to develop a 15, 2012. Producing the course took one teaching assistant venture capital track over the next couple of years. working over the summer, one full-time and one half-time edX person, production and engineering staff time that totaled 2.5 full-time equivalents, a part-time administrative ONLINE EDUCATION AND MOOCS support person, five paid forum moderators to answer ques- Working from the underlying theme that the chemical tions posted by students, four volunteer community teach- bond determines properties, and with an emphasis on link- ing assistants, and two to five beta checkers. The lecture ing basic concepts with applications, MIT’s unique first-year video derived mostly from his 2011 lecture class. Some 280 chemistry course teaches students fundamental chemical lecture segments, 65 screencasts, and six additional video principles through the solid state. The hypothesis around segments were incorporated into the final version of 3.091x. which the course is based is that students will find it is easier “It was a huge amount of work, and I spent the bulk of the to learn, understand, and, most importantly, use chemical summer getting ready for this,” he said. Now the course is principles if they can relate them to the solids around them. offered, he goes online himself most mornings and answers Michael Cima has now adapted this course, which has been student questions. He noted, though, that the unpaid com- offered as an option for meeting MIT’s general instructional munity teaching assistants, who are people taking the class chemistry requirements since the 1970s, to serve as a MOOC in locations worldwide, have been an amazing resource. In on the edX2 online platform. The course, known as 3.091x,3 fact, Cima has asked one of them—a high school chemis- is offered free of charge; includes homework, exams, and try teacher in the United Kingdom—to serve as a teaching a final exam; contains the same intellectual content as the assistant for this coming fall semester. Six of her high school classroom-based MIT course on which it is based; and is students have also taken the class, he added. a certificate-earning rather than credit-earning course. He When the course was offered, nearly 29,000 people added that the course is taken by a cohort of students that registered for it. Over 3,400 took the first test, almost 2,200 work together. took the second test, and 2,148 took the final exam. About Cima explained that 3.091x is an engineering course, not 15,000 of the students were using the materials throughout a chemistry course, and that affects the way the students are the course and, on the basis of results of an exit survey, Cima tested. “We are assessing students not on what they know, thinks that the bulk of these people are taking a chemistry but what they can do, and they do a lot of calculations, which course and using 3.091x as a teaching supplement. Of those turns out to be an advantage for an online course.” After taking the course, 13 percent were graduate students, 28.9 demonstrating the class to the workshop, Cima noted that the percent were university students, 1 percent were community experience of doing a screencast was much more interesting college students, and 9.7 percent were high school students, for him as a teacher than standing in front of a large lecture some of whom want to know what taking an MIT class hall. He noted that classes on the edX platform are nothing entails. The biggest surprise, he said, was the large number like traditional online courses in that there is no 50-minute of teachers—almost 9 percent—who took the course, and he lecture. At most, each lesson consists of a 10-minute lecture has corresponded with many of them. Some 3 percent were segment on a single concept followed by self-assessment K-12 teachers and over 5 percent were university or com- tools that are graded immediately and serve as a reality check munity college teachers. It may be feasible, he said, to offer for the student. these kinds of classes for professional development credits For MIT, the reason to offer an online version of an estab- for high school science teachers. lished course was simple: research consistently shows that In a retrospective look at outcomes, which he did by learning outcomes are about the same for a residence-based designing the residence-based final exam in such a way that course and an online version, but that when the versions are he could do a select group of measurements with the online combined, the students do better on both courses. The goal, final, it appears that the online students outperformed the shared between MIT and Harvard, is to use both versions residence-based students. Cima believes this “troubling” simultaneously, and Cima was going to do just that starting finding resulted because it may be possible to do a better with the fall 2013 semester. assessment of student learning in an online setting than in a Course development took considerable resources in time classroom under time constraints. One of his goals for the and money. Cima explained that course preparations began future is to develop improved assessment tools. in June 2012, 4 months prior to 3.091x’s launch on October Another task Cima faces is determining how best to maximize the benefits of having all of the developed content and assessment tools, particularly as he works to integrate 2 This nonprofit organization offers MOOCs and interactive online classes the online and residence-based classes. He noted that the for a variety of subjects, including some STEM subjects. It was founded by decision to integrate the two is highly political at MIT and individuals from Harvard University and MIT. he spent considerable effort building support and getting 3 See https://www.edx.org/course/mit/3-091x/introduction-solid- approval for this change. This coming fall, Cima is going state/591.

OCR for page 15
INNOVATIONS AND BARRIERS 23 to conduct an experiment that will consist of replacing the meeting that challenge has been to reconfigure the learning course texts with the online content and developing lec- outcomes of the course to match the grand challenges in sci- tures to take advantage of the new “text.” The course will ence education enumerated in the 2013 special issue of the be structured around two 1-hour recitations per week, and journal Science (McNutt 2013) and to design a curriculum assessment will consist of 37 proctored online quizzes that that uses theory webcasts to present concepts, pressure-point the student will complete within a specified time window. problems to immerse the student in experience, problem-of- Each quiz will represent a learning outcome measure, and the-day discussions, and peer-to-peer tutoring. if a quiz is not answered correctly, the student may take the One of the key features of the course is its use of online quiz again and for as many times as they want within the electronic homework, which for organic chemistry works time window. There will be a 24-hour lockout between quiz out wonderfully with machine-read computer drawings that attempts. Each learning outcome quiz will be selected ran- can be automatically graded using a programmed graphical domly from a group of many related problems. Cima noted language. There is no textbook in the course, though there is that he used this quiz format with the last midterm of the a set of course notes that accompany the 5-minute webcasts online class without any problem. that the students view before attending a discussion session. A participant asked what the implications of MOOCs The discussion sessions are held in a computer lab that might be for university education, and Cima said that while can accommodate 55 students at a time, or via an Internet he has no real idea, it could be that the freshman year is connection at two times daily. At the end of the discussion spent off campus taking their required foundational courses session, the students are presented with a pressure-point online. He was then asked if the videos will still be fresh in 5 problem that they have 5 minutes to solve. Successfully years or if the course will have to be continually redesigned answering the question yields bonus points that are applied at significant expense. Again, Cima replied that he did not to the next exam. know what the future held but noted that it would be easy These complex problems, explained Moore, are designed and relatively inexpensive to replace videos with screencasts. to take students into uncharted territory and force them to He also said replacing content is simple once the course is take risks and fail, just as scientists do in the real world constructed, claiming that he can use a new software tool (see Figure 3-5). He called this “taking off the training to replace content from his desktop computer in 5 minutes. wheels,” and said it teaches students about failure and how A participant asked if MIT was considering faculty diver- to respond to that failure. The problems are nonalgorithmic sity in its plans to create a catalog of MOOCs. Cima said and multifaceted, and multiple steps are involved in solv- that the university takes this challenge very seriously since ing them. Students are forced to use creative processes to the MOOCs do represent the face of the university and that generate a variety of initial-guess solutions and to develop face has changed considerably over the past 30 years. He a strategy to initiate a solution. They are also allowed to noted that the introductory mechanical engineering MOOC access the literature or any other online resource—except is taught by two female faculty. communicating with another person—to solve the problem. Wesemann asked if the positive experience he had creat- ing 3.091x was having an effect on other faculty. Yes, said Cima, and in fact, each engineering department is hiring a person who will be dedicated to helping faculty convert their courses to an online format for use by residence-based students. In response to a question from Cardillo about other companies offering online courses, Cima said that his impression, as well as that of other faculty he has talked to, is that there is a wide diversity in terms of the quality of these courses. What he likes about the edX format is that it is based on an open-source system and developers are taking advan- tage of that to develop assessment tools for community use. DEALING WITH RISK, FAILURE, AND UNCERTAINTY The challenge that Jeffrey Moore is tackling at the Uni- versity of Illinois with his instructional experimentation is to center instruction on the individual learner in what he called the “kilostudent” organic chemistry classroom consisting of a diverse set of nonchemistry majors who, in most cases, are FIGURE 3-5  Learning through experiences that mimic the real taking their last formal course in chemistry. His approach to world. SOURCE: Jeffrey Moore.

OCR for page 15
24 UNDERGRADUATE CHEMISTRY EDUCATION FIGURE 3-6  Graphical tracking of exam progress. SOURCE: Jeffrey Moore. No points are given for an incorrect answer, but students are be made, its properties, and other relevant information. The allowed to continue trying to solve the problem until they idea behind this project is to promote the development of get the correct answer. Because the students are develop- professional scientific skills, and it appears to produce gains ing and refining their solutions online, the computer can in literature searching, scientific writing, and critical reading, track the progress they are making toward the solution. though there is evidence that sustaining these gains requires When plotted, these data provide a picture of how the what he called a “super teaching assistant” or an intensely exam is going and how each student is doing in real time devoted instructor. (see Figure 3-6). In summary, Moore said that this revamping has success- One of the main components of Moore’s course is peer- fully flipped the two-semester organic sequence without to-peer tutoring aided by the Internet. “We realized through significantly increasing the load of the teaching assistants. some of our assessments that one of the most important ways “I can say from the data we’ve collected that we’ve done no that students were learning was not by using the videos, not harm,” he said. “We have not improved things in terms of even by the problems that they were doing, but by the interac- tions that they were having with other students,” said Moore. The way this worked was when a student answered a problem correctly, they would be placed in the tutoring pool for that problem. Students who answered a problem incorrectly and who wanted help would then go to the tutoring pool, where they could select a peer tutor to help them. At the end of the process, both students then recorded video reflections of the outcome of that tutoring experience. “We do not monitor every single one of those videos, but we do keep track of the clusters of people and the informa- tion flow, the diffusion of information,” he explained. Using this information, Moore can create a network map of tutor– tutee interactions (see Figure 3-7). One of the surprises that came out of the analysis of the video reflections was how much value the students placed on peer-to-peer tutoring from both the tutor and the tutee. Watching video lectures online was deemed the least valuable method of learning by the students. Moore has also implemented a semester-long group FIGURE 3-7  Tutor–tutee interaction network. Arrows illustrate project in which four-student teams select a small organic the hypothetical transfer of knowledge from tutor to tutee, and the molecule, typically a bioorganic molecule, and use the lit- nodes are scaled by how central they are to the network. SOURCE: erature to propose a mechanism for how this molecule might Jeffrey Moore.

OCR for page 15
INNOVATIONS AND BARRIERS 25 these learning outcomes and objectives, but at the same time, Generation Science Standards (NGSS)4 that talk about sci- learning is robust.” In his view, machine-graded homework, ence practices, crosscutting content, and core ideas. Evidence discussion problems, daily pressure-point problems, and of coming changes include the new Advanced Placement exam problems offer students many more opportunities tests that the College Board is releasing and the redesigned for practice with feedback, and that testing with real-time MCAT2015 exam. She echoed the sentiment shared by other feedback might measure students’ ability to diagnose their individuals at the workshop that teaching methods should be errors in a pressure situation. He reiterated that peer-to-peer revamped based on the growing body of research that shows tutoring appears to be a valuable tool for student learning and how students learn. added that it requires little effort to implement. At UC Berkeley, Stacy and her colleagues have started Jeffrey Reimer asked if the students are aware that what- writing introductory chemistry course exams differently, ever they do for this course is analyzed and therefore not pri- and they conducted a study to determine the effect of these vate. Moore replied that he has received institutional review changes. As part of this study, they asked students at three board approval and that the students agree to participate in times during the semester what they were doing outside of the data collection and analysis process. Session chair Emilio class to prepare for the four exams given during the semester Bunel asked if other faculty were involved in teaching this to better understand how they were using resources when course. Moore said he did have one colleague involved who they were directing themselves and how they changed their has since moved to another university, but his impression is study strategies as they realized that the tests were less about that faculty are intimidated by the amount of work that went memorization and more about understanding and applying into developing the course. There is an instructor for the concepts. first-semester organic chemistry class who has completely The students fell into four groups. She calls the lowest- embraced this approach and has abandoned the traditional performing group the fact gatherers—the students who lecture approach. memorize independent facts and then are confused by their Luis Martinez from Rollins College asked about the low grades given the amount of time they put into their stud- importance of training for the teaching assistants and if their ies. The next group, which performed slightly better, learned experience teaching in this format had any impact on their procedures—they absorbed information and made small development as future teachers. Moore said that he is careful connections, but still relied on others or course materials about selecting teaching assistants for this course who feel for answers. Students in the third group work at confirming comfortable with technology. He has not yet assessed what their understanding. They evaluate information and question the teaching assistants are getting from this experience but he why, work more independently as learners, and try to give thought that would be a good idea to look at going forward. explanations. These students are trying to understand if they What he has heard from the students who take the class is are thinking about a problem correctly. The fourth group, that they value how much they learn from acting as teachers which Stacy characterized as amazing, thinks about chemis- in the peer-to-peer tutoring process and how much they now try all the time. These students are applying ideas, taking in value teaching. information and questioning why it is true and how it helps to explain the world around them. Unfortunately, said Stacy, most of the bright, motivated, WHAT GETS MEASURED IS WHAT GETS LEARNED: talented students who come to the University of California, ASSESSING STUDENT UNDERSTANDING Berkeley, fall into the first two categories. Over the course One of the problems in the way students are taught sci- of the semester, the numbers do shift, with more students ence today lies with the way student learning is assessed, moving into group three and a few rising into group four. said Angelica Stacy. Legacy chemistry exams, she stated, The big shifts, she noted, do not happen until after the sec- are designed to select for those students who are good at ond exam when the students realize the first exam was not a memorizing. “Teachers are a good delivery system if all you fluke. Students who improved became more active learners want students to do is memorize,” she said. “Teachers ask and they moved away from just reading the course textbook questions on exams that students can answer, and everyone and started working with peers and asking more questions. feels good.” The result? “If you don’t assess what is impor- Students who did not improve remained passive learners who tant, what is assessed becomes important.” Stacy quoted stay focused on reading text and were not reflective when Sir Ken Robinson, an internationally recognized author and studying. The students reported that they made changes to leader in education, who said that the way most classes are their study habits because the exam questions made them taught today is turning education into “hours of low-grade apply what they were learning—they could not memorize clerical work.” Stacy is optimistic that this sorry state can change because 4 of the new climate of possibilities that starts with the Next The NGSS are new K-12 science standards to provide students an internationally benchmarked science education. The NGSS are based on the NRC report Framework for K-12 Science Education (NRC 2011a).

OCR for page 15
26 UNDERGRADUATE CHEMISTRY EDUCATION an algorithm to solve the questions and expect to do well “I think it begins with the assessments,” she said. “They have on the exam. to promote understanding as opposed to memorization or Stacy acknowledged that exam design is challenging we’re really not getting to where we want to be. We have to when teaching large lecture classes. For example, she has use students’ ideas and experiences to build knowledge. Let 1,300 students to teach and a limited number of graders to them observe. Let them explore the data. You’ll be amazed help her. “The trick is, can we start to learn how to write at what ideas they do come up with.” better multiple-choice questions,” she said. One approach is YuYe Tong from Georgetown University remarked that to use data and observations as the basis for test questions this work shows the value and challenges of moving students and then structure the classroom experience to develop the from being passive to active learners and of emphasizing skills to explain data, to find patterns, and to understand concepts over content. Sarah Green from the Michigan Tech- how to control variables when comparing different pieces of nology University commented that it will be interesting to see data. She and her colleagues have designed their chemistry what the impact will be of the NGSS, which stress concepts curriculum around core ideas in chemistry that are similar to and problem solving as opposed to rote memorization, as those in the Next Generation of Science Standards: matter, those students enter the undergraduate chemistry curriculum. change, energy, and light. She described several examples, Stacy noted that one effect of using these new course designs including one that uses smells to explore molecular structure to teach college students is that they will become the next and properties. Stacy presented her students with a table with generation of teachers and professors, creating what could properties of four chemicals found in spices with strong be a virtuous cycle in science education. smells or tastes: vanillin, eugenol, zingerone, and capsaicin. In response to a question from Cardillo about the extent to The properties given were flavor, molecular formula, struc- which these innovative courses have been adopted by other tural formula, melting point, boiling point, and water solubil- faculty, Stacy said that she has been joined by two of her col- ity. Students were also provided with space-filling models of leagues and they now coteach this course, where they trade the molecules. Along with this information, to the students, off lectures and critique each other. Other faculty have seen they were asked practical questions like “why in the world how enjoyable it is to teach chemistry in this manner and can I smell vanillin so well?” and “I can’t smell capsaicin, are working to change their course design too, noted Stacy. but when it gets into my mouth it doesn’t go away. Why is Anderson added that he has strong support from his depart- that?” The exam questions are designed to combine data ment for continuing to revamp the introductory physical and observations in such a way that the students must use sciences class, but that other faculty are still in wait-and-see core concepts in chemistry, like intermolecular attractions, mode because of the amount of work that he and Kaxiras to explain the data. have had to put into both redesigning the course and develop- In closing, Stacy argued that the field needs to do ing the accompanying text and other course materials. research-based redesign of undergraduate chemistry courses.