2
Learning with Simulations and Games

This chapter discusses research evidence related to the use of simulations and games for science learning. The first section presents the committee’s framework for its review of the research, identifying five science learning goals. The next two sections review and discuss research on the effectiveness of simulations and games in advancing each of these goals. The fourth section synthesizes research findings related to a set of design features that appear to influence the effectiveness of simulations and games in supporting learning, and the fifth section describes limitations of the research. The chapter concludes with a summary of key findings—both about the effectiveness of simulations and games and about the state of the research.

LEARNING GOALS

The committee views science learning as a complex, multifaceted process that involves not only mastering science concepts, but also skills in designing and carrying out scientific investigations and feelings and attitudes toward science. To identify the learning goals of simulations and games, the committee drew on a previous definition of informal science learning (National Research Council, 2009). That study identified six interwoven strands as valued goals of informal science learning:


Strand 1: Experience excitement, interest, and motivation to learn about phenomena in the natural and physical world (motivation).


Strand 2: Come to generate, understand, remember, and use concepts, explanations, arguments, models, and facts related to science (conceptual understanding).



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 25
2 Learning with Simulations and Games This chapter discusses research evidence related to the use of simulations and games for science learning. The first section presents the committee’s framework for its review of the research, identifying five science learning goals. The next two sections review and discuss research on the effectiveness of simulations and games in advancing each of these goals. The fourth section synthesizes research findings related to a set of design features that appear to influence the effectiveness of simulations and games in supporting learn- ing, and the fifth section describes limitations of the research. The chapter concludes with a summary of key findings—both about the effectiveness of simulations and games and about the state of the research. LEARNING GOALS The committee views science learning as a complex, multifaceted process that involves not only mastering science concepts, but also skills in designing and carrying out scientific investigations and feelings and attitudes toward science. To identify the learning goals of simulations and games, the com- mittee drew on a previous definition of informal science learning (National Research Council, 2009). That study identified six interwoven strands as valued goals of informal science learning: Strand 1: Experience excitement, interest, and motivation to learn about phenomena in the natural and physical world (motivation). Strand 2: Come to generate, understand, remember, and use concepts, explanations, arguments, models, and facts related to science (conceptual understanding). 

OCR for page 25
 Learning Science Through Computer Games and Simulations This strand emphasizes understanding of fundamental concepts rather than memorization of unconnected facts. Strand 3: Manipulate, test, explore, predict, question, observe, and make sense of the natural and physical world (science process skills). This may include making observations, formulating a research question, developing a hypothesis (perhaps in the form of a model), using a range of methods to gather data, data analysis, and confirmation or revision of the hypothesis. Strand 4: Reflect on science as a way of knowing; on processes, concepts, and institutions of science, as well as on the learners’ own process of learning about phenomena (understanding of the nature of science). Strand 5: Participate in scientific activities and learning practices with others, using scientific language and tools (scientific discourse). This strand flows out of the notion that science takes place in a com- munity that shares norms, practices, and a common language and that learners should be introduced to these norms and practices as they engage with science. Strand 6: Think about themselves as science learners and develop an identity as someone who knows about, uses, and sometimes contributes to science (identity). This strand may be reflected in one’s ability to effectively apply scientific knowledge to life situations (e.g., health decisions) or at work, whether or not one works in a science-related job. These six strands of informal science learning are closely intertwined and mutually supportive. They reflect the theory that mastery of science concepts and understanding of the nature of science are supported and accelerated when students engage in the processes of science. This theory is supported by a growing body of research evidence (National Research Council, 2005b, 2007). The strands are also based on a growing body of research that illuminates the importance of motivation, the social and cul- tural context, and feelings of identity and self-efficacy in supporting learn- ing generally and science learning in particular (National Research Council, 2005b, 2007, 2009). The strands are well aligned with other recent theories of how people learn, such as theories that view education as a process of preparing for future learning and problem solving (Bransford and Schwartz, 1999; Schwartz, Bransford, and Sears, 2005). Because science process skills and understanding of the nature of sci- ence are especially closely related, the committee merged them, reducing

OCR for page 25
Learning with Simulations and Games  the number of learning goals from six to five. These five goals provided a valuable framework for the committee’s deliberations about the use of gaming and simulations to support science learning and they serve as a template in the following review of the research. Although the review is organized by separate goals, it illuminates the capacity of some simulations and games to simultaneously advance multiple science learning goals. EFFECTIVENESS OF SIMULATIONS The available research on the effectiveness of simulations for learning is more extensive and stronger than the research on games. However, both simulations and games are relatively young learning technologies, and devel- opers have focused primarily on design, with less attention to research. Some studies have examined how a simulation affects a single group of learners without a control group of similar learners who receive science instruction targeted to the same learning goal but without the simulation. Other studies compare one or more groups of learners who interact with different versions of a simulation. In these studies, the lack of control of other variables that may influence learning makes it unclear whether any reported learning gains can be attributed to the simulation (or one version of it) alone. A related challenge is that simulations are often embedded within a larger curriculum unit, making it difficult to disentangle the effects of the simulation(s). Ma and Nickerson (2006) discuss this problem in their review of the literature comparing hands-on, virtual, and remote laboratories in undergraduate science education. They found that investigators often con- founded the effects of many different factors and perhaps over-attributed learning gains to simulations or other learning technologies. The research also includes a few studies focusing on the goal of con- ceptual understanding, in which investigators used control or comparison groups or other elements of the study design to try to limit the influence of other variables. These studies provide stronger evidence that simulations are effective. It is important to keep in mind the strengths and weaknesses of study designs when reviewing research findings. Overall, the research provides promising evidence that the use of simulations can enhance conceptual understanding in science and moder- ate evidence that simulations can motivate interest in science and science learning. There is more limited or no evidence that simulations advance the other science learning goals defined above. Motivation Research over the past three decades indicates that simulations can encourage learners to experience excitement, interest, and motivation to

OCR for page 25
 Learning Science Through Computer Games and Simulations learn about phenomena in the natural and physical world (Clark et al, 2009). Building on these findings, more recent research indicates that simulations and simulation-based curriculum units motivate learners by providing them with authentic, interesting tasks and contexts (e.g., Adams et al., 2008a, 2008b; Cognition and Technology Group at Vanderbilt, 1990; Edelson, Gordin, and Pea, 1999). Some examples follow. In a study of the PhET suite of simulations (see Box 1-2), Adams et al. (2008a) conducted over 200 structured interviews with 89 undergraduate student volunteers, focusing on 52 simulations targeting different physics concepts. For each simulation, the authors interviewed a diverse group of four to six students with equal numbers of male and female students and a representative share of minority students. The volunteers (typically non- science majors) included students who had not yet received formal instruction on the topics covered by the simulations. Trained interviewers with advanced physics knowledge asked students to describe their understanding of an idea or concept before seeing the simulation and allowed them to revise their answers while interacting with the simulation or afterward; they also asked students to think aloud as they freely explored the simulations. The results suggest that the simulations’ effectiveness in motivating learners was closely related to their effectiveness in supporting conceptual understanding. The authors found that a PhET simulation can be highly en- gaging and effective for mastering physics concepts, but only if the student’s interaction with the simulation is directed by the student’s own questioning—a process they refer to as “engaged exploration.” Through this process, most study participants were able to accurately describe the concepts covered in the simulation and apply the concepts to correctly predict behaviors in the simulation. The participants also frequently volunteered correct predictions or explanations about related phenomena. Although the study did not in- clude a control group, the authors described the study participants’ level of conceptual understanding as much greater than the level typically reached by students taught about these concepts in a physics course. They also noted that study participants regularly reported playing with several simulations for fun during their leisure time—suggesting that the simulations are motivating and engaging. In another study, Edelson et al. (1999) found that incorporating the challenge of global warming in the WorldWatcher visualization-based cur- riculum unit enhanced motivation for learning. The researchers used an informal evaluation approach in a rapid cycle of iterative design and testing of the unit, seeking a design that would motivate students to engage and persist in the investigations included within the unit. Formative evaluation of early versions of the curriculum unit led to the decision to incorporate the challenge of global warming. In a pilot test of the revised curriculum unit in three schools, the authors observed and videotaped students interacting

OCR for page 25
Learning with Simulations and Games  with the visualization and obtained teacher and student journals and informal teacher feedback. The data indicated that students found four aspects of the global warming challenge motivating: It was familiar, it had potential direct implications for students, the policy issues appealed to students’ sense of fairness, and it was a subject of current scientific debate and controversy. Both Edelson, Gordin, and Pea (1999) and Adams et al. (2008a, 2008b) caution that encouraging students’ interest, engagement, and motivation is a very challenging task for the designer of a simulation or simulation-based learning environment. Klopfer, Yoon, and Rivas (2004) studied two participatory simula - tions, comparing the relative ability of two different technology platforms to motivate students to persist through the difficulties of inquiry learning incorporated within these simulations. Students from two Boston area high schools—one public (N = 71 in four classes) and one private (N = 117 in six classes)—played Live Long and Prosper, a game focusing on Mendelian genetics. Students at one private middle school (N = 82 in five classes) played the Virus game, which simulates transmission of a virus. Within each school, half of the classes were randomly assigned to use either wearable computers or Palm Pilots while participating in the simulation. Data from pre- and post-activity questionnaires revealed no significant differences between schools, classes, or technology in students’ ratings of engagement. The pooled data showed that students felt like they had fun and expressed a strong interest in playing other participatory simulation games. After playing the games, students felt more strongly that they could learn a lot about science from games. They also highly rated their learning about science content and experimental design and expressed strong agreement with the statement that the technology used positively impacted their learning. Conceptual Understanding Most studies of simulations focus on the goal of enhancing conceptual understanding (de Jong, 2009; Quellmalz, Timms, and Schneider, 2009). They provide promising evidence that simulations can help students generate, understand, remember, and use science concepts, particularly when they are supported by other forms of instruction within a larger curriculum unit (Clark et al., 2009). Many studies have examined the potential of simulations to help students replace their intuitive alternative explanations of natural phenomena with scientifically correct explanations. For example, Meir et al. (2005) hypoth- esized that students’ deep-rooted misconceptions about diffusion and osmosis might be partly due to their inability to see and explore these processes at the molecular level. To investigate this, they developed OsmoBeaker, a set of two simulated laboratories, one focusing on diffusion and the other on

OCR for page 25
0 Learning Science Through Computer Games and Simulations osmosis. Each laboratory included a series of simulated experiments and a workbook. To test OsmoBeaker, the researchers recruited student volunteers from 11 Boston-area colleges, ranging from large, well-known universities to small community colleges. Among the volunteers, 83 percent were fresh- men or sophomores and 71 percent were women. Eighty-four percent had received instruction on osmosis in an introductory college biology class, and most of the others had studied osmosis in high school biology. At least half had completed a wet lab on osmosis. Each participant met with a researcher for a 2-hour session and was financially compensated. The participant first read a description of osmosis and diffusion, then completed a written pretest, and then worked through the simulated laboratory experiments for about 45-60 minutes before complet- ing a posttest. Both the pretest and posttest focused on alternative concep- tions of diffusion and osmosis. The authors tested the diffusion laboratory on 15 students and the osmosis laboratory on 31 students. On the diffusion laboratory, 13 out of 15 students showed statistically significant gains from pretest to posttest, and on the osmosis laboratory, 23 of 31 students demon- strated statistically significant gains. Based on these results and interviews with study participants, the authors concluded that the simulated experiments helped students overcome several common alternative conceptions about diffusion and osmosis. Although this study lacked a comparison group, all study participants had received previous instruction on diffusion, osmosis, or both prior to engaging with the simulation. This gives greater strength to the conclusion by Meir et al. (2005, p. 245) that “the improvements observed after the com- puter laboratories are above and beyond what students learn by reading or listening to material on the topic.” Another strand of research on the use of simulations to address alterna- tive conceptions focuses NetLogo simulations (Wilensky, 1999). Sengupta and Wilensky (2008a, 2008b, 2009) studied NetLogo Investigations in Electromagnetism (NIELS). This sequence of simulations allows learners to manipulate representations of electrons at the microscopic level to help them understand the behavior of electric current moving through a wire at the macroscopic level. Sengupta and Wilensky (2008a) studied a group of fifth and seventh graders who interacted with a revised version of NIELS. The revised version framed the motion of electrons in terms of a process of accumulation inside the positively-charged end of a battery—a change designed to address intui- tive conceptions about electric current that appeared, from earlier research, to pose a barrier to the correct scientific understanding. Two science classes of 20 students each worked with the revised version of the simulation dur- ing one 45-minute class period, recording their observations in detail on log sheets. The researchers analyzed the log sheets and interviewed a sample

OCR for page 25
Learning with Simulations and Games  of four randomly selected students within each class to gain insight into students’ thinking. Over 90 percent of students in both classes using the revised version of NIELS displayed correct reasoning about the behavior of electrons in an electric current. The performance of these novice learners was not statisti- cally different from the performance of 12th graders who had used the pilot version of NIELS. The authors concluded that the reframing of the motion of electrons helped the younger students build on their naïve ideas about electricity to develop a correct understanding. A related study of middle school students suggests that their interaction with NetLogo-based simulations enhanced their understanding of statistical mechanics, a topic that is traditionally taught using equation-based repre- sentations in college-level physics (Wilensky, 2003). To address students’ alternative conceptions in chemistry, researchers developed ChemCollective, a collection of simulated laboratories and other learning activities (Yaron et al., 2010). Cuadros and Yaron (2007) investigated the use of the virtual laboratories, assigned as homework, in a second-semester chemistry class of 144 students. Students completed a pretest focusing on chemistry concepts, and also took three midterm exams and a final exam focusing on the same concepts; the homework assignments were graded. The authors found that the homework grades accounted for 24 percent of the variation in exam scores, suggesting that engaging in science processes with the virtual laboratories increased students’ conceptual understanding. In addition, the lack of a significant relationship between the homework grades and the pretest scores suggests that virtual laboratory activities developed additional understanding beyond what students brought to the class. In one of the few controlled studies of simulations, Evans, Yaron, and Leinhardt (2008) studied the simulated laboratories, integrated with other forms of instruction in an online stoichiometry course. The course included an overarching narrative designed to motivate student learning, a variety of virtual laboratory activities, and rapid feedback during laboratory practice. The comparison course was a text-based study guide addressing the same topics presented in the online course. Both the online and text-based courses were designed for self-study, because all first-semester chemistry students were required to study stoichiometry on their own time in preparation for a mastery exam. Entering college freshmen volunteers were randomly assigned to either the online class or the text-based class. A total of 45 students (27 male and 18 female) completed either the online course (21 students) or the text-only course (24 students) over a period of 10 to 24 days. After the end date of the study, participants completed a proctored test of stoichiometry concepts and proce- dures on campus. Statistical analysis revealed a significant gain in test scores among the online group when compared with the text-only group. However,

OCR for page 25
 Learning Science Through Computer Games and Simulations regression analysis of posttest scores indicated that only 6 percent of the vari- ability in performance was explained by treatment (i.e., participation in either the online or text-based course). Among the students in the online course, nearly 40 percent of the variation in posttest scores was related to the degree to which the student interacted with the virtual laboratory. Although the study does not demonstrate that simulations are more effective than other forms of science instruction, it provides further evidence that simulations can help students master science concepts by engaging them in science processes. Recent Syntheses of Research on Simulations Linn and Eylon (in press) synthesized findings from three types of studies: (1) laboratory investigations that compare static diagrams to dynamic simula- tions, (2) classroom comparison studies that compare simulation-supported instruction with typical text-based instruction, and (3) classroom studies of the use of simulations without comparison groups that use a pretest-posttest design. The laboratory-based studies generally indicated that well-designed simulations are more effective for learning than static diagrams, but the studies had mixed results, with effect sizes ranging from –0.5 to 1.76. The authors’ analysis of classroom comparison studies found that simulations are more effective than typical instruction, with consistently positive effects averag- ing 0.49 across the studies. Analysis of the third group of studies found that simulations had a large positive effect, averaging 1.17. However, the authors note that these studies lack control groups and sometimes confound the larger instructional design with the specific effects of the simulation. Noting these mixed findings about the effectiveness of simulations, Linn et al. (2010) identify three design principles to improve learning outcomes. First, simulations should minimize irrelevant cognitive demand to avoid dis- tracting students from the primary learning goal. Second, simulations should be presented in a personally meaningful scientific context, allowing students to draw on what they already know, ask more effective questions, and recog- nize unlikely findings. Third, simulations should be embedded in supportive instruction, such as guidance on how to conduct simulated experiments. For example, Chang (2009) compared two approaches to learning about heat and temperature, in which students either read about how to conduct virtual experiments or critiqued the experiments of others before conducting their own virtual experiments. Pretests and posttests indicated that both groups of students made considerable progress in understanding thermal conductivity and equilibrium; however, the critique group had larger learning gains than the other group. In addition, the critique group was more successful than the other group in responding to an assessment item that asked students to plan a second trial after being given a research question and the results of a first trial related to the research question.

OCR for page 25
Learning with Simulations and Games  In another recent synthesis, Scalise et al. (2009) identified 79 articles that examined the use of simulations, including virtual laboratories, in grades 6-12 and included reports of measured learning outcomes. The most frequent research design, used in slightly over half of the studies, was to compare results from pretests and posttests of student learning on goals and objec- tives. Approximately 40 percent of the studies also, or alternatively, used a quasi-experimental research design, comparing a treatment group that received the simulations with a group that received another type of science instruction not involving virtual laboratories or simulations. In addition, just over 15 percent of the articles used literature synthesis of results from other studies, 10 percent were qualitative case studies, and the remaining 10 per- cent used other approaches. None of the 79 studies used a true experimental research design, with random assignment of participants to either a treatment group or a control group. Across these 79 studies, slightly over half (53 percent) reported gains in learning among those taught with the simulations, about 25 percent found mixed outcomes in which some groups showed learning gains but others did not, 18 percent found gains under the right conditions, and approximately 4 percent reported no gain in learning. Scalise et al. (2009) note that many of the studies that lacked comparison groups were designed to quickly obtain feedback from students or teachers for the purpose of developing a simula- tion product and caution that the reported learning gains might not align well with findings that would result from more systematic research designs. Conceptual Understanding in Domains Outside Science Simulations for military training have demonstrated effectiveness in enhancing the conceptual understanding and related skills needed to per- form specific jobs; cost-effectiveness is a key measure of success (Fletcher, 2009a, 2009b). For example, SHERLOCK is a simulation-based training system designed to prepare technicians to solve electronics problems when main- taining avionics equipment. Lesgold et al. (1992) estimated that a trainee who spent 20 hours interacting with the system developed problem-solving ability equivalent to that of an avionics technician with 4 years of learning on the job. Another example focuses on the sophisticated knowledge of oceanogra- phy needed by Navy personnel who use advanced sonar to detect submarines. The Interactive Multisensor Analysis Training (IMAT) simulation-based train- ing system presents trainees with a comprehensive range of virtual situations representing the required knowledge and skill. Wulfeck, Wetzel-Smith, and Baker (2007) found that IMAT graduates scored higher on an assessment of oceanography knowledge and skills than fleet personnel with 3 to 10 years of experience, and IMAT-trained officers performed as well on an assessment

OCR for page 25
4 Learning Science Through Computer Games and Simulations of search planning as officers with 4 to 6 years of experience in planning sonar searches for submarines. Science Process Skills and Understanding of the Nature of Science The goal of developing students’ ability to manipulate, test, explore, predict, question, observe, and make sense of the natural and physical world (science process skills) is closely related to the goal that students reflect on science as a way of knowing; on processes, concepts, and institu- tions of science; and on their own science learning (understanding of the nature of science). Although simulations and simulation-based curriculum units often engage students in selected science processes (see Box 1-2), only a few studies have focused on—or directly assessed—their potential to advance these two learning goals. One study that that specifically examined science process skills focused on ThinkerTools, a simulation-based curriculum unit addressing Newton’s laws of motion (White and Frederiksen, 1998). In the curriculum, students formulate a research question, generate alternative hypotheses and predic- tions, design and carry out both real-world and simulated experiments, analyze the resulting data, construct a conceptual model with scientific laws that would predict and explain what they found, and apply their model to different situations thereby leading to new research questions. White and Frederiksen (1998) tested two different versions of Thinkertools, one with formative assessments integrated throughout, designed to encour- age students to self-assess and reflect on core aspects of inquiry and their own learning, and another without these self-assessment prompts. The researchers implemented the curriculum unit in 12 urban seventh-, eighth-, and ninth-grade classrooms, incorporating it in daily science instruction over a period of about 10.5 weeks. The classes included 343 students taught by 3 teachers, and two-thirds of the students were minorities. Classrooms were randomly assigned to either the reflective self-assessment version or the control version. The researchers evaluated understanding of scientific investigations using a pre-post inquiry test and compared gains in scores for the reflective self- assessment classes with gains in scores in control classrooms. Results were also broken out by students categorized as high and low achieving, based on performance on a standardized test conducted before the intervention. The test results showed gains for all students in their understanding of scientific investigations, and the self-assessment classes exhibited greater gains. This was especially true for low-achieving students. ThinkerTools also appeared to advance conceptual understanding, as measured by a posttest focusing on force and motion. On one difficult test

OCR for page 25
Learning with Simulations and Games  item that had been included in an earlier study, the middle school students performed significantly better, on average, than did a comparison group of 40 high school students who had completed a high school-level physics class. In terms of the goal of motivation, student surveys conducted before and after participation in the curriculum unit indicated that students felt more positive about their ability to learn and understand science following their interactions with ThinkerTools. More recently, Schwarz and White (2005) developed and studied the Model-­Enhanced ThinkerTools (METT) curriculum unit, focusing on three learning goals: (1) science process (inquiry) skills, (2) understanding of the nature of science (specifically, knowledge of models and modeling), and (3) conceptual understanding of physics. The METT curriculum unit extended ThinkerTools by allowing students to create, evaluate, and discuss computer models of their ideas about force and motion, and it included instruction on the nature of models and modeling. Schwarz and White (2005) tested METT in four seventh-grade science classes in an urban school that met daily for 45 minutes over the course of 10.5 weeks. Approximately 44 percent of the school’s students were black, 31 percent were white, 13 percent were Asian, and 11 percent were Hispanic. Additionally, 34 percent of students qualified for free or reduced-price meals, and 20 percent came from families who received Aid for Dependent Children. Study participants’ scores on the Individual Test of Academic Skills varied, with a median percentile score of 66—higher than the median score of 60 on the Comprehensive Test of Basic Skills among students involved in the earlier test of ThinkerTools. Student scores on three written pretests and posttests—a modeling as- sessment, an inquiry test, and a conceptual physics test—showed significant gains. Comparison of METT students’ gains in inquiry and physics concepts with those of the prior ThinkerTools students revealed no significant differ- ence overall. However, the METT students performed better on one section of the inquiry test focusing on conclusions, which suggested that the em- phasis on modeling helped them to draw appropriate conclusions from their experimental data. Finally, analysis of METT students’ test results suggested that their gains in knowledge of modeling (a dimension of understanding of the nature of science) and science process skills supported their gains in physics knowledge. Two studies of participatory simulations examined development of science process skills. In a pilot study by Colella (2000), urban high school biology students wearing small portable computers acted as agents in a dynamic simu- lation of the transmission of a virus in the closed system of the classroom. The class consisted mainly of tenth-grade students, who were described by their teacher as traditionally poor performers in science. Sixteen students, seven girls and nine boys, along with their teacher, participated in the activities.

OCR for page 25
4 Learning Science Through Computer Games and Simulations because design features that may enhance conceptual learning may not be the same as design features that aim to motivate learners to pursue careers in science (Ketelhut, 2009). Researchers have established focusing on clearly-defined learning goals as a design principle to improve the effectiveness of simulations. For ex- ample, Clark and Mayer (2003) drew on empirical evidence to propose the coherence principle. This principle emphasizes that all elements of a simu- lation should be directly related to the learning goals, avoiding extraneous information that could distract the learner, disrupt the learning process, or seduce them into incorrect understandings. More recently, Linn et al. (2010) stated, based on a review of the research, that simulations should minimize irrelevant cognitive demand that could otherwise distract students from the primary learning goal. Plass et al. (2009, p. 48) state that there is enough research evidence to identify the following design principle for simulations: “The efficacy of a simulation depends on the degree to which it is in line with learning objectives.” The more limited research on games also suggests that it is important to focus on clear learning goals. For example, in the study of SURGE described above, Clark et al. (2010) found that the game caused a significant decrease in scores on one posttest item by unintentionally focusing students’ attention on another physics relationship that was not an intended learning goal. Clear learning goals are critical for the design of assessments to measure the effectiveness of a simulation or game (Quellmalz et al., 2009). The learn- ing goal must be clearly established as a basis for evaluating the effectiveness of any game or simulation, and such evaluations support further research and continued improvement. Provide External Scaffolding To address the challenges involved in inquiry learning, research cur- rently focuses on developing scaffolds, or cognitive tools, to support learn- ing (de Jong, 2006). Learning scaffolds for simulations and games may be internal, including many of the other design features discussed below, or they may be external (see Box 2-1). The research discussed in this chapter highlights the value of external scaffolding. Many of the examples provide evidence that simulations enhance conceptual understanding of science when they are scaffolded with other forms of instruction in larger curriculum units (e.g., ThinkerTools, NIELS, Biologica). Linn et al. (2010) recommend that designers embed simulations in supportive instruction as an important design principle to enhance effec- tiveness. This design principle is similar to de Jong’s (2005) guided-discovery principle, which focuses on addressing students’ documented difficulty in all aspects of inquiry learning, whether in the classroom or laboratory or in

OCR for page 25
Learning with Simulations and Games 4 BOX 2-1 Scaffolding Learning in Simulations and Games Traditionally, scaffolding is a process by which adults or more able peers provide supportive structures to help learners perform mature behaviors before they are ready to do so on their own. Scaffolds can also be built into an activity itself, as in the example of training wheels on a bicycle. Once learners exhibit mature or independent behavior, the scaffolds are removed or faded. Taken more expansively, scaffolding can also be viewed as a progression of just-manageable challenges that enable learners to climb to greater understanding and skills. Thus, as they develop independence at one activity, a new, more challenging activity can lead to the next round of support. New technologies create new opportunities for scaffolding, for ex- ample, with adaptive systems that provide just-in-time hints or change problem difficulty. Simulations and games can be designed to permit learners to pursue different progressions to the same outcomes, depend- ing on various factors, including student interest, prior knowledge, and success so far. Scaffolding can be proactive and built into learners’ first attempts at an activity, or it can be reactive in response to when they are faced with a challenge that they can solve with a hint, question, prompt, or interactive resource. Games demonstrate that providing challenges and scaffolds in an appropriate balance can keep motivation high. Ideally, they also help students develop important dispositions that include identify- ing with scientific activities and content to help reach important science learning goals. Building effective scaffolding is a multifaceted process. First, experts in a subject identify suitable learning tasks or challenges that will guide the learner to grapple with the important ideas or skills in productive ways. Second, it is important to develop the resource framework that learners can use to help achieve the task, for example, through experimentation, explanation, peer networking, or reading. Scaffolding is therefore provided both in the selection of the important ideas or skills and in the related edu- cational tasks and resources that best support the learning. Third, when developing a complex set of ideas or skills, the developer must consider the progression of learning over time. Fourth, the high interactivity of games and simulations provides opportunities for contingent feedback and system responsiveness. When learners encounter a challenge or question that is beyond their immediate capacity, scaffolding of various forms allows them to make progress (e.g., hints, guidance, or simply turning off options).

OCR for page 25
4 Learning Science Through Computer Games and Simulations a simulation (Mayer, 2004). The guided-discovery principle (de Jong, 2005) states that inquiry learning is more effective when simulations or simulation- based curriculum units provide guidance, such as domain-specific explana- tions or direct advice on when to perform certain actions. External scaffolding also appears to enhance learning through games. Neulight et al. (2007) found that study participants who experienced the simulated Whypox virus in a classroom setting, in which they also learned about infectious diseases through other forms of instruction, experienced gains in conceptual understanding and in identification with the scientific enterprise. Other study participants, who played the game at home, did not advance in these two dimensions of science learning. In another study, Mayer, Mautone, and Prothero (2002) found that providing pretraining in the Profile Game before playing it, by showing players pictures of possible geological features that would need to be identified through the game, led to significantly better performance on identifying those geological features in the game. Representation Research on how people react to, and learn from, different forms of visual stimuli has been under way for decades. Early studies compared pic- torial with text representations (Plass et al., 2009). More recent studies of simulations and games have focused on how information is represented on a continuum from more detailed and realistic to more stylized or abstract. Some research suggests that more realistic representations can be more effective than abstract symbols. For example, Plass et al (2009) report on two experiments, both involv- ing 80 to 90 students aged 16 to 18 in a large public high school in rural Texas. Nearly 90 percent of the students were of Hispanic descent, 40 percent were female, and they had not previously studied the topic addressed by the simulation—the behavior of a gas when heated. For the first experiment, participants were randomly assigned to one of two forms of the simulation, one of which incorporated only abstract symbols (e.g., numbers), while the other also incorporated icons—small pictures of flames representing tem- perature and weights representing pressure. After completing a question- naire about prior chemistry experience and pretests of chemistry knowledge and self-efficacy, participants worked with the simulation for approximately 20 minutes. They then completed posttests of comprehension and transfer knowledge. When the authors found no significant difference in learning outcomes between the two groups, they hypothesized that it was because the learning task placed a low cognitive load (demand on working memory) on the students. For the second experiment, the investigators increased the simulation’s cognitive load by including a chart that displayed the effects

OCR for page 25
Learning with Simulations and Games 4 of changing the temperature or pressure of the gas. Comparing test results, the authors found significant positive differences in comprehension and self-efficacy for the group using the simulations with icons. Further analysis indicated that the added icons were especially beneficial for students with low prior knowledge of chemistry. Other research suggests that representations that are too realistic may impede learners’ ability to transfer their understanding to another domain. Son and Goldstone (2009) conducted a series of three experiments focusing on the scientific principle of competitive specialization. First, they compared intuitive descriptions with concrete (i.e., realistic) representations and found that intuitive descriptions led to enhanced domain-specific learning but also deterred transfer. Second, they alleviated the limited transfer by combining intuitive descriptions with idealized graphical elements. In the third experi- ment, they found that idealized graphics were more effective for learning and transfer than concrete graphics, even when unintuitive descriptions were applied to them. They concluded that idealized graphics enhance learning and transfer when compared with highly realistic graphics. In addition, re- search on the two-dimensional, cartoonlike Whyville game discussed above suggests that a high degree of realism is not always necessary to support science learning. Based on their review of research on education and train- ing with games, Wilson et al. (2009) propose that as the degree of realism of the task in a game increases, psychomotor skill learning will also increase but then level off. Finally, representation is related to the learning goals of the simulation or game. Clear learning goals can help designers focus on the perceptual salience of the information displayed. For example, in a simulation about harmonic motion, Parnafes (2007) noted that students typically tended to attend to the perceptually salient features of the simulation rather than the conceptually important features (features an expert would attend to). This study suggests that, when designing simulations, it is important that the salient features of the simulation are ones that will be most productive in terms of the targeted learning goals. Narrative/Fantasy Narrative, sometimes called fantasy, is an extremely important feature of games. It engages learners, allows them to interact with the game without fear of real-life consequences, and makes them feel immersed in the game (Wilson et al., 2009). Without a strong narrative, a game designed for infor- mal use may not attract players and a game designed for classroom use will not generate excitement, interest, or enthusiasm for science learning. Barab, Arici, and Jackson (2005) found, based on their iterative design process in creating and modifying Quest Atlantis, that a strong narrative was one ele-

OCR for page 25
0 Learning Science Through Computer Games and Simulations ment supporting engagement. In a further study of Quest Atlantis, Barab et al. (2007) observed that students saw an erosion diagram as part of the narrative, rather than an abstract representation of the scientific process of erosion. The authors suggest that too much narrative might hinder learning of formal scientific concepts, principles, and methods, making it difficult for students to distinguish these concepts from the particular situation in the game. Thus, game developers must carefully balance context with content. The narrative in games designed for science learning often presents players with a question, problem, or mission that requires information to respond. Wilson et al. (2009) refer to this type of narrative as “mystery” and propose that learner motivation is positively related to the level of mystery in a game. Feedback An extensive body of research supports the view that providing learners with feedback enhances learning, and this also appears to be the case when using simulations and games. For example, the “reflection prompts” in ThinkerTools encouraged students to reflect on their own thinking, which in turn led to gains in both science process skills and conceptual understanding (White and Frederikson, 1998). Rieber, Tzeng, and Tribble (2004) found that students given graphical feedback during a simulation on laws of motion with short explanations far outperformed those given only textual information. Moreno and Mayer (2000, 2004) conducted a series of studies to investi- gate the impact of design principles applied to computer games on student retention of science content and on problem-solving transfer questions. In one of these studies, undergraduate university students played a computer game about environmental science that included personalized instructional content, delivered as narrated speech by a pedagogical agent. Students who heard personalized content outperformed students who received neutral content. In another study, Moreno and Mayer (2005) compared using the pedagogical agent to give only corrective feedback (communicating to the learner whether she or he is right or wrong) with using it to give explanatory feedback (learners were told whether or not they were correct and were also given an explanation of why the answer was right or wrong). They found that providing explanatory feedback increased retention and transfer of the targeted concepts. Nelson (2007) conducted a River City study in which he explored the impact of embedded guidance messages on student understanding of real- world science inquiry processes and knowledge. He found that increased viewing of guidance messages was associated with significantly higher score gains on a test focusing on knowledge of disease transmission.

OCR for page 25
Learning with Simulations and Games  User Control Clark et al. (2009) identified the degree of user control as a dimension of simulations. However, the research reviewed above suggests that user control is an important feature of games as well. The optimal degree of user control in a given simulation or game is related to the science learning goal or goals targeted. For example, the limited degree of control provided to users of PhET simulations appears to be well aligned with the goals of these simulations—to increase conceptual understanding of specific sci- ence topics. If the goal of a simulation or game is to increase science process skills and understanding, the research suggests that the degree of user control must be carefully balanced. On one hand, providing some autonomy to design and carry out virtual experiments appears to engage and motivate users of River City and Quest Atlantis. On the other hand, students often become confused when allowed to engage in open-ended inquiry—whether in a school science laboratory or in a virtual inquiry environment (Mayer, 2004; Moreno and Mayer, 2005). Providing students with guidance along with some control—such as the feedback from a pedagogical agent described above (Moreno and Mayer, 2005)—appears to enhance learning of science processes as well as science content. Plass, Homer, and Hayward (2009), based on their review of the re- search, identify manipulation of content as a design principle for effective simulations, proposing that, “learning from visualizations is improved when learners are able to manipulate the content of a dynamic visualization com- pared to when they are not able to do so” (p. 49). Among other studies supporting this principle is a comparative study of two forms of a chemistry simulation—one that allowed the user to manipulate the content (e.g., the temperature and pressure of a gas) and one that allowed the user to only control pacing (Plass et al., 2007). Study participants who interacted with the simulation that allowed content manipulation demonstrated larger learning gains than those who were only allowed to control pacing. Wilson et al.(2009), in their review of the research on gaming, report that allowing learners to navigate through a computer program based on their personal preferences leads to more positive attitudes and higher cogni- tive outcomes (Vogel et al., 2006). They also found that game players value control at all levels, from simply picking out a wardrobe or specific facial features for their avatars to determining strategies in game play. The authors propose that increasing the amount of control given to learners using games will positively affect skill-based learning.

OCR for page 25
 Learning Science Through Computer Games and Simulations Individual Learner Differences The studies discussed above suggest that differences among individuals influence how they respond to, and learn from, simulations and games. For example, students with lower science achievement, as measured by a pretest, experienced greater gains in inquiry and content knowledge after using ThinkerTools (White and Frederiksen, 1998). Plass et al. (2009) found that adding icons that represented temperature and pressure concretely (as opposed to only abstract symbols) improved understanding of gas laws, espe- cially among learners with low prior knowledge of the topic. These findings suggest that it is important to consider the target audience when designing a simulation or game and also to include adaptive features that modify the pace and type of information, based on user responses. LIMITATIONS OF THE RESEARCH The preceding discussion reveals many gaps and weaknesses in the body of research on the use of simulations and games for science learning. Although both simulations and games have been used for training and educa- tion for over three decades, they have not been studied systematically (Clark et al., 2009). Rapid changes in technology and delivery platforms result in changing definitions of what constitutes a game or a simulation, making it difficult to focus the research. Another problem is that researchers do not always describe the context for the interaction with the simulation or game, including other instructional support that might be provided in a classroom setting or informally by peers, making it difficult to separate out the unique contribution of the simulation or game. In addition, researchers sometimes fail to examine or report important variables related to student abilities and attitudes, such as previous science knowledge and previous experience with simulations or games. Another limitation is that studies have usually involved small groups of students with little diversity, making it difficult to generalize the results to the large, diverse population of U.S. science students. The studies of games and simulations reviewed in this chapter unevenly address the methodological challenge associated with how to model out- comes that are by their very nature “nested” (students within classrooms or recitation sections, classrooms within schools or universities). The authors of several studies randomly assigned classrooms to different treatments (e.g., different versions of a simulation) or to treatment and control conditions, but analyzed and reported on data from individual students. These studies must be interpreted with caution, as the analysis of student-level data may lead to findings of statistically significant effects that are not warranted. 2See Bryk and Raudenbush (1992) for a detailed treatment of this issue.

OCR for page 25
Learning with Simulations and Games  The questions researchers have asked about the effectiveness of simula- tions and games for learning, as well as the methods they have used, reflect a wide range of theoretical perspectives on how people learn. For example, the theoretical perspectives of neuroscientists studying how playing action video games affects visual response times are quite different from the theoretical perspectives underlying studies of how interactions with simulations affect understanding of science concepts. Reflecting these diverse perspectives, investigators have used a range of different research methods to measure the learning outcomes of simulations or games. The wide distribution of the published research evidence across journals in a variety of different disci- plines makes it difficult to build on and extend a coherent base of research across studies and over time. Another problem is that researchers studying games and simulations have not given enough attention to the adequacy of the instruments used to measure student outcomes (Quellmalz, Timms, and Schneider, 2009). Assessments are often designed to measure conceptual understanding alone, rather than other learning goals, and generally rely on paper and pencil tests, rather than taking advantage of digital technology to embed assessments in simulations or games (see Chapter 5). As a result, there is only limited evidence related to many of the five learning goals. The research on games is particularly limited. Game designers often study potential users’ reactions to and experience of a game to gauge consumer acceptance, but they rarely conduct formal research on science learning. Another challenge is that games are often designed for informal learning by self-selected users. Because of these challenges, only a few scholarly studies have been conducted. O’Neil, Wainess, and Baker (2005) searched three databases for studies of the effectiveness of games for learning and training published over a 15-year period and also conducted a hand search of journals for the year 2004-2005. Among the several thousand articles about games, the authors were able to identify only 19 articles that had been published in peer-reviewed journals and provided empirical information on the effec- tiveness of games. Although studies have documented the effectiveness of particular games to support learning among specific populations, it is unclear whether, or to what extent, the study findings can be generalized to other populations of learners (Hays, 2005). All of these challenges make it difficult to build a coherent base of evi- dence that could demonstrate the effectiveness of simulations and games and inform future design improvements. Experts do not agree on the best directions for future research and development to support science learning. The field needs a process that will allow research evidence to accumulate across the variety of simulations and games and in the face of the constant innovation that characterizes them.

OCR for page 25
4 Learning Science Through Computer Games and Simulations CONCLUSIONS Science learning is a complex process involving multiple learning goals. A simulation or game can be designed to advance one or more science learning goals. Conclusion: Simulations and games have potential to advance multiple science learning goals, including motivation to learn science, conceptual understanding, science process skills, understanding of the nature of science, scientific discourse and argumentation, and identification with science and science learning. There is promising evidence that simulations enhance conceptual under- standing, but effectiveness in conveying science concepts requires good design, testing, and proper scaffolding of the learning experience itself. Conclusion: Most studies of simulations have focused on conceptual under-­ standing, providing promising evidence that simulations can advance this science learning goal. There is moderate evidence that simulations motivate students’ interest in science and science learning. Less evidence is available about whether simulations support development of science process skills and other science learning goals. The emerging body of evidence about the effectiveness of games in supporting science learning is much smaller and weaker than the body of evidence about the effectiveness of simulations. Research on a few examples suggests that games can motivate interest in science and enhance conceptual understanding, but overall it is inconclusive. Conclusion: Evidence for the effectiveness of games for supporting science learning is emerging, but is currently inconclusive. To date, the research base is very limited. The available research suggests that differences among individual learners influence how they respond to, and learn from, simulations and games. Some studies of simulations have found that students with lower prior knowledge experienced greater gains in targeted learning goals than students with more prior knowledge related to these goals. Differences across gender and race in young people’s use of commercial games could potentially influence their motivation to use games for science learning; however, a few studies of games have demonstrated gains in science learning across students of different genders, races, English language ability, and socioeconomic status.

OCR for page 25
Learning with Simulations and Games  Conclusion: Emerging evidence indicates that different individuals and groups of learners respond differently to features of games and simulations. Although the research evidence related to science learning through inter- action with simulations is stronger and deeper than that related to games, the overall research base is thin. Development of simulations and games has outpaced research and development of assessment of their learning outcomes, limiting the amount of evidence related to other learning goals beyond conceptual understanding. Conclusion: The many gaps and weaknesses in the body of research on the use of simulations and games for science learning make it difficult to build a coherent base of evidence that could demonstrate their effectiveness and inform future improvements. The field needs a process that will allow research evidence to accumulate across the variety of simulations and games and in the face of the constant innovation that characterizes them.

OCR for page 25