Learning with Simulations and Games
This chapter discusses research evidence related to the use of simulations and games for science learning. The first section presents the committee’s framework for its review of the research, identifying five science learning goals. The next two sections review and discuss research on the effectiveness of simulations and games in advancing each of these goals. The fourth section synthesizes research findings related to a set of design features that appear to influence the effectiveness of simulations and games in supporting learning, and the fifth section describes limitations of the research. The chapter concludes with a summary of key findings—both about the effectiveness of simulations and games and about the state of the research.
The committee views science learning as a complex, multifaceted process that involves not only mastering science concepts, but also skills in designing and carrying out scientific investigations and feelings and attitudes toward science. To identify the learning goals of simulations and games, the committee drew on a previous definition of informal science learning (National Research Council, 2009). That study identified six interwoven strands as valued goals of informal science learning:
Strand 1: Experience excitement, interest, and motivation to learn about phenomena in the natural and physical world (motivation).
Strand 2: Come to generate, understand, remember, and use concepts, explanations, arguments, models, and facts related to science (conceptual understanding).
This strand emphasizes understanding of fundamental concepts rather than memorization of unconnected facts.
Strand 3: Manipulate, test, explore, predict, question, observe, and make sense of the natural and physical world (science process skills).
This may include making observations, formulating a research question, developing a hypothesis (perhaps in the form of a model), using a range of methods to gather data, data analysis, and confirmation or revision of the hypothesis.
Strand 4: Reflect on science as a way of knowing; on processes, concepts, and institutions of science, as well as on the learners’ own process of learning about phenomena (understanding of the nature of science).
Strand 5: Participate in scientific activities and learning practices with others, using scientific language and tools (scientific discourse).
This strand flows out of the notion that science takes place in a community that shares norms, practices, and a common language and that learners should be introduced to these norms and practices as they engage with science.
Strand 6: Think about themselves as science learners and develop an identity as someone who knows about, uses, and sometimes contributes to science (identity).
This strand may be reflected in one’s ability to effectively apply scientific knowledge to life situations (e.g., health decisions) or at work, whether or not one works in a science-related job.
These six strands of informal science learning are closely intertwined and mutually supportive. They reflect the theory that mastery of science concepts and understanding of the nature of science are supported and accelerated when students engage in the processes of science. This theory is supported by a growing body of research evidence (National Research Council, 2005b, 2007). The strands are also based on a growing body of research that illuminates the importance of motivation, the social and cultural context, and feelings of identity and self-efficacy in supporting learning generally and science learning in particular (National Research Council, 2005b, 2007, 2009). The strands are well aligned with other recent theories of how people learn, such as theories that view education as a process of preparing for future learning and problem solving (Bransford and Schwartz, 1999; Schwartz, Bransford, and Sears, 2005).
Because science process skills and understanding of the nature of science are especially closely related, the committee merged them, reducing
the number of learning goals from six to five. These five goals provided a valuable framework for the committee’s deliberations about the use of gaming and simulations to support science learning and they serve as a template in the following review of the research. Although the review is organized by separate goals, it illuminates the capacity of some simulations and games to simultaneously advance multiple science learning goals.
EFFECTIVENESS OF SIMULATIONS
The available research on the effectiveness of simulations for learning is more extensive and stronger than the research on games. However, both simulations and games are relatively young learning technologies, and developers have focused primarily on design, with less attention to research. Some studies have examined how a simulation affects a single group of learners without a control group of similar learners who receive science instruction targeted to the same learning goal but without the simulation. Other studies compare one or more groups of learners who interact with different versions of a simulation. In these studies, the lack of control of other variables that may influence learning makes it unclear whether any reported learning gains can be attributed to the simulation (or one version of it) alone.
A related challenge is that simulations are often embedded within a larger curriculum unit, making it difficult to disentangle the effects of the simulation(s). Ma and Nickerson (2006) discuss this problem in their review of the literature comparing hands-on, virtual, and remote laboratories in undergraduate science education. They found that investigators often confounded the effects of many different factors and perhaps over-attributed learning gains to simulations or other learning technologies.
The research also includes a few studies focusing on the goal of conceptual understanding, in which investigators used control or comparison groups or other elements of the study design to try to limit the influence of other variables. These studies provide stronger evidence that simulations are effective. It is important to keep in mind the strengths and weaknesses of study designs when reviewing research findings.
Overall, the research provides promising evidence that the use of simulations can enhance conceptual understanding in science and moderate evidence that simulations can motivate interest in science and science learning. There is more limited or no evidence that simulations advance the other science learning goals defined above.
Research over the past three decades indicates that simulations can encourage learners to experience excitement, interest, and motivation to
learn about phenomena in the natural and physical world (Clark et al, 2009). Building on these findings, more recent research indicates that simulations and simulation-based curriculum units motivate learners by providing them with authentic, interesting tasks and contexts (e.g., Adams et al., 2008a, 2008b; Cognition and Technology Group at Vanderbilt, 1990; Edelson, Gordin, and Pea, 1999). Some examples follow.
In a study of the PhET suite of simulations (see Box 1-2), Adams et al. (2008a) conducted over 200 structured interviews with 89 undergraduate student volunteers, focusing on 52 simulations targeting different physics concepts. For each simulation, the authors interviewed a diverse group of four to six students with equal numbers of male and female students and a representative share of minority students. The volunteers (typically non-science majors) included students who had not yet received formal instruction on the topics covered by the simulations. Trained interviewers with advanced physics knowledge asked students to describe their understanding of an idea or concept before seeing the simulation and allowed them to revise their answers while interacting with the simulation or afterward; they also asked students to think aloud as they freely explored the simulations.
The results suggest that the simulations’ effectiveness in motivating learners was closely related to their effectiveness in supporting conceptual understanding. The authors found that a PhET simulation can be highly engaging and effective for mastering physics concepts, but only if the student’s interaction with the simulation is directed by the student’s own questioning—a process they refer to as “engaged exploration.” Through this process, most study participants were able to accurately describe the concepts covered in the simulation and apply the concepts to correctly predict behaviors in the simulation. The participants also frequently volunteered correct predictions or explanations about related phenomena. Although the study did not include a control group, the authors described the study participants’ level of conceptual understanding as much greater than the level typically reached by students taught about these concepts in a physics course. They also noted that study participants regularly reported playing with several simulations for fun during their leisure time—suggesting that the simulations are motivating and engaging.
In another study, Edelson et al. (1999) found that incorporating the challenge of global warming in the WorldWatcher visualization-based curriculum unit enhanced motivation for learning. The researchers used an informal evaluation approach in a rapid cycle of iterative design and testing of the unit, seeking a design that would motivate students to engage and persist in the investigations included within the unit. Formative evaluation of early versions of the curriculum unit led to the decision to incorporate the challenge of global warming. In a pilot test of the revised curriculum unit in three schools, the authors observed and videotaped students interacting
with the visualization and obtained teacher and student journals and informal teacher feedback. The data indicated that students found four aspects of the global warming challenge motivating: It was familiar, it had potential direct implications for students, the policy issues appealed to students’ sense of fairness, and it was a subject of current scientific debate and controversy.
Both Edelson, Gordin, and Pea (1999) and Adams et al. (2008a, 2008b) caution that encouraging students’ interest, engagement, and motivation is a very challenging task for the designer of a simulation or simulation-based learning environment.
Klopfer, Yoon, and Rivas (2004) studied two participatory simulations, comparing the relative ability of two different technology platforms to motivate students to persist through the difficulties of inquiry learning incorporated within these simulations. Students from two Boston area high schools—one public (N = 71 in four classes) and one private (N = 117 in six classes)—played Live Long and Prosper, a game focusing on Mendelian genetics. Students at one private middle school (N = 82 in five classes) played the Virus game, which simulates transmission of a virus. Within each school, half of the classes were randomly assigned to use either wearable computers or Palm Pilots while participating in the simulation.
Data from pre- and post-activity questionnaires revealed no significant differences between schools, classes, or technology in students’ ratings of engagement. The pooled data showed that students felt like they had fun and expressed a strong interest in playing other participatory simulation games. After playing the games, students felt more strongly that they could learn a lot about science from games. They also highly rated their learning about science content and experimental design and expressed strong agreement with the statement that the technology used positively impacted their learning.
Most studies of simulations focus on the goal of enhancing conceptual understanding (de Jong, 2009; Quellmalz, Timms, and Schneider, 2009). They provide promising evidence that simulations can help students generate, understand, remember, and use science concepts, particularly when they are supported by other forms of instruction within a larger curriculum unit (Clark et al., 2009).
Many studies have examined the potential of simulations to help students replace their intuitive alternative explanations of natural phenomena with scientifically correct explanations. For example, Meir et al. (2005) hypothesized that students’ deep-rooted misconceptions about diffusion and osmosis might be partly due to their inability to see and explore these processes at the molecular level. To investigate this, they developed OsmoBeaker, a set of two simulated laboratories, one focusing on diffusion and the other on
osmosis. Each laboratory included a series of simulated experiments and a workbook. To test OsmoBeaker, the researchers recruited student volunteers from 11 Boston-area colleges, ranging from large, well-known universities to small community colleges. Among the volunteers, 83 percent were freshmen or sophomores and 71 percent were women. Eighty-four percent had received instruction on osmosis in an introductory college biology class, and most of the others had studied osmosis in high school biology. At least half had completed a wet lab on osmosis.
Each participant met with a researcher for a 2-hour session and was financially compensated. The participant first read a description of osmosis and diffusion, then completed a written pretest, and then worked through the simulated laboratory experiments for about 45-60 minutes before completing a posttest. Both the pretest and posttest focused on alternative conceptions of diffusion and osmosis. The authors tested the diffusion laboratory on 15 students and the osmosis laboratory on 31 students. On the diffusion laboratory, 13 out of 15 students showed statistically significant gains from pretest to posttest, and on the osmosis laboratory, 23 of 31 students demonstrated statistically significant gains. Based on these results and interviews with study participants, the authors concluded that the simulated experiments helped students overcome several common alternative conceptions about diffusion and osmosis.
Although this study lacked a comparison group, all study participants had received previous instruction on diffusion, osmosis, or both prior to engaging with the simulation. This gives greater strength to the conclusion by Meir et al. (2005, p. 245) that “the improvements observed after the computer laboratories are above and beyond what students learn by reading or listening to material on the topic.”
Another strand of research on the use of simulations to address alternative conceptions focuses NetLogo simulations (Wilensky, 1999). Sengupta and Wilensky (2008a, 2008b, 2009) studied NetLogo Investigations in Electromagnetism (NIELS). This sequence of simulations allows learners to manipulate representations of electrons at the microscopic level to help them understand the behavior of electric current moving through a wire at the macroscopic level.
Sengupta and Wilensky (2008a) studied a group of fifth and seventh graders who interacted with a revised version of NIELS. The revised version framed the motion of electrons in terms of a process of accumulation inside the positively-charged end of a battery—a change designed to address intuitive conceptions about electric current that appeared, from earlier research, to pose a barrier to the correct scientific understanding. Two science classes of 20 students each worked with the revised version of the simulation during one 45-minute class period, recording their observations in detail on log sheets. The researchers analyzed the log sheets and interviewed a sample
of four randomly selected students within each class to gain insight into students’ thinking.
Over 90 percent of students in both classes using the revised version of NIELS displayed correct reasoning about the behavior of electrons in an electric current. The performance of these novice learners was not statistically different from the performance of 12th graders who had used the pilot version of NIELS. The authors concluded that the reframing of the motion of electrons helped the younger students build on their naïve ideas about electricity to develop a correct understanding.
A related study of middle school students suggests that their interaction with NetLogo-based simulations enhanced their understanding of statistical mechanics, a topic that is traditionally taught using equation-based representations in college-level physics (Wilensky, 2003).
To address students’ alternative conceptions in chemistry, researchers developed ChemCollective, a collection of simulated laboratories and other learning activities (Yaron et al., 2010). Cuadros and Yaron (2007) investigated the use of the virtual laboratories, assigned as homework, in a second-semester chemistry class of 144 students. Students completed a pretest focusing on chemistry concepts, and also took three midterm exams and a final exam focusing on the same concepts; the homework assignments were graded. The authors found that the homework grades accounted for 24 percent of the variation in exam scores, suggesting that engaging in science processes with the virtual laboratories increased students’ conceptual understanding. In addition, the lack of a significant relationship between the homework grades and the pretest scores suggests that virtual laboratory activities developed additional understanding beyond what students brought to the class.
In one of the few controlled studies of simulations, Evans, Yaron, and Leinhardt (2008) studied the simulated laboratories, integrated with other forms of instruction in an online stoichiometry course. The course included an overarching narrative designed to motivate student learning, a variety of virtual laboratory activities, and rapid feedback during laboratory practice. The comparison course was a text-based study guide addressing the same topics presented in the online course. Both the online and text-based courses were designed for self-study, because all first-semester chemistry students were required to study stoichiometry on their own time in preparation for a mastery exam.
Entering college freshmen volunteers were randomly assigned to either the online class or the text-based class. A total of 45 students (27 male and 18 female) completed either the online course (21 students) or the text-only course (24 students) over a period of 10 to 24 days. After the end date of the study, participants completed a proctored test of stoichiometry concepts and procedures on campus. Statistical analysis revealed a significant gain in test scores among the online group when compared with the text-only group. However,
regression analysis of posttest scores indicated that only 6 percent of the variability in performance was explained by treatment (i.e., participation in either the online or text-based course). Among the students in the online course, nearly 40 percent of the variation in posttest scores was related to the degree to which the student interacted with the virtual laboratory. Although the study does not demonstrate that simulations are more effective than other forms of science instruction, it provides further evidence that simulations can help students master science concepts by engaging them in science processes.
Recent Syntheses of Research on Simulations
Linn and Eylon (in press) synthesized findings from three types of studies: (1) laboratory investigations that compare static diagrams to dynamic simulations, (2) classroom comparison studies that compare simulation-supported instruction with typical text-based instruction, and (3) classroom studies of the use of simulations without comparison groups that use a pretest-posttest design. The laboratory-based studies generally indicated that well-designed simulations are more effective for learning than static diagrams, but the studies had mixed results, with effect sizes ranging from –0.5 to 1.76. The authors’ analysis of classroom comparison studies found that simulations are more effective than typical instruction, with consistently positive effects averaging 0.49 across the studies. Analysis of the third group of studies found that simulations had a large positive effect, averaging 1.17. However, the authors note that these studies lack control groups and sometimes confound the larger instructional design with the specific effects of the simulation.
Noting these mixed findings about the effectiveness of simulations, Linn et al. (2010) identify three design principles to improve learning outcomes. First, simulations should minimize irrelevant cognitive demand to avoid distracting students from the primary learning goal. Second, simulations should be presented in a personally meaningful scientific context, allowing students to draw on what they already know, ask more effective questions, and recognize unlikely findings. Third, simulations should be embedded in supportive instruction, such as guidance on how to conduct simulated experiments. For example, Chang (2009) compared two approaches to learning about heat and temperature, in which students either read about how to conduct virtual experiments or critiqued the experiments of others before conducting their own virtual experiments. Pretests and posttests indicated that both groups of students made considerable progress in understanding thermal conductivity and equilibrium; however, the critique group had larger learning gains than the other group. In addition, the critique group was more successful than the other group in responding to an assessment item that asked students to plan a second trial after being given a research question and the results of a first trial related to the research question.
In another recent synthesis, Scalise et al. (2009) identified 79 articles that examined the use of simulations, including virtual laboratories, in grades 6-12 and included reports of measured learning outcomes. The most frequent research design, used in slightly over half of the studies, was to compare results from pretests and posttests of student learning on goals and objectives. Approximately 40 percent of the studies also, or alternatively, used a quasi-experimental research design, comparing a treatment group that received the simulations with a group that received another type of science instruction not involving virtual laboratories or simulations. In addition, just over 15 percent of the articles used literature synthesis of results from other studies, 10 percent were qualitative case studies, and the remaining 10 percent used other approaches. None of the 79 studies used a true experimental research design, with random assignment of participants to either a treatment group or a control group.
Across these 79 studies, slightly over half (53 percent) reported gains in learning among those taught with the simulations, about 25 percent found mixed outcomes in which some groups showed learning gains but others did not, 18 percent found gains under the right conditions, and approximately 4 percent reported no gain in learning. Scalise et al. (2009) note that many of the studies that lacked comparison groups were designed to quickly obtain feedback from students or teachers for the purpose of developing a simulation product and caution that the reported learning gains might not align well with findings that would result from more systematic research designs.
Conceptual Understanding in Domains Outside Science
Simulations for military training have demonstrated effectiveness in enhancing the conceptual understanding and related skills needed to perform specific jobs; cost-effectiveness is a key measure of success (Fletcher, 2009a, 2009b). For example, SHERLOCK is a simulation-based training system designed to prepare technicians to solve electronics problems when maintaining avionics equipment. Lesgold et al. (1992) estimated that a trainee who spent 20 hours interacting with the system developed problem-solving ability equivalent to that of an avionics technician with 4 years of learning on the job.
Another example focuses on the sophisticated knowledge of oceanography needed by Navy personnel who use advanced sonar to detect submarines. The Interactive Multisensor Analysis Training (IMAT) simulation-based training system presents trainees with a comprehensive range of virtual situations representing the required knowledge and skill. Wulfeck, Wetzel-Smith, and Baker (2007) found that IMAT graduates scored higher on an assessment of oceanography knowledge and skills than fleet personnel with 3 to 10 years of experience, and IMAT-trained officers performed as well on an assessment
of search planning as officers with 4 to 6 years of experience in planning sonar searches for submarines.
Science Process Skills and Understanding of the Nature of Science
The goal of developing students’ ability to manipulate, test, explore, predict, question, observe, and make sense of the natural and physical world (science process skills) is closely related to the goal that students reflect on science as a way of knowing; on processes, concepts, and institutions of science; and on their own science learning (understanding of the nature of science). Although simulations and simulation-based curriculum units often engage students in selected science processes (see Box 1-2), only a few studies have focused on—or directly assessed—their potential to advance these two learning goals.
One study that that specifically examined science process skills focused on ThinkerTools, a simulation-based curriculum unit addressing Newton’s laws of motion (White and Frederiksen, 1998). In the curriculum, students formulate a research question, generate alternative hypotheses and predictions, design and carry out both real-world and simulated experiments, analyze the resulting data, construct a conceptual model with scientific laws that would predict and explain what they found, and apply their model to different situations thereby leading to new research questions.
White and Frederiksen (1998) tested two different versions of Thinkertools, one with formative assessments integrated throughout, designed to encourage students to self-assess and reflect on core aspects of inquiry and their own learning, and another without these self-assessment prompts. The researchers implemented the curriculum unit in 12 urban seventh-, eighth-, and ninth-grade classrooms, incorporating it in daily science instruction over a period of about 10.5 weeks. The classes included 343 students taught by 3 teachers, and two-thirds of the students were minorities. Classrooms were randomly assigned to either the reflective self-assessment version or the control version.
The researchers evaluated understanding of scientific investigations using a pre-post inquiry test and compared gains in scores for the reflective self-assessment classes with gains in scores in control classrooms. Results were also broken out by students categorized as high and low achieving, based on performance on a standardized test conducted before the intervention. The test results showed gains for all students in their understanding of scientific investigations, and the self-assessment classes exhibited greater gains. This was especially true for low-achieving students.
ThinkerTools also appeared to advance conceptual understanding, as measured by a posttest focusing on force and motion. On one difficult test
item that had been included in an earlier study, the middle school students performed significantly better, on average, than did a comparison group of 40 high school students who had completed a high school-level physics class. In terms of the goal of motivation, student surveys conducted before and after participation in the curriculum unit indicated that students felt more positive about their ability to learn and understand science following their interactions with ThinkerTools.
More recently, Schwarz and White (2005) developed and studied the Model-Enhanced ThinkerTools (METT) curriculum unit, focusing on three learning goals: (1) science process (inquiry) skills, (2) understanding of the nature of science (specifically, knowledge of models and modeling), and (3) conceptual understanding of physics. The METT curriculum unit extended ThinkerTools by allowing students to create, evaluate, and discuss computer models of their ideas about force and motion, and it included instruction on the nature of models and modeling.
Schwarz and White (2005) tested METT in four seventh-grade science classes in an urban school that met daily for 45 minutes over the course of 10.5 weeks. Approximately 44 percent of the school’s students were black, 31 percent were white, 13 percent were Asian, and 11 percent were Hispanic. Additionally, 34 percent of students qualified for free or reduced-price meals, and 20 percent came from families who received Aid for Dependent Children. Study participants’ scores on the Individual Test of Academic Skills varied, with a median percentile score of 66—higher than the median score of 60 on the Comprehensive Test of Basic Skills among students involved in the earlier test of ThinkerTools.
Student scores on three written pretests and posttests—a modeling assessment, an inquiry test, and a conceptual physics test—showed significant gains. Comparison of METT students’ gains in inquiry and physics concepts with those of the prior ThinkerTools students revealed no significant difference overall. However, the METT students performed better on one section of the inquiry test focusing on conclusions, which suggested that the emphasis on modeling helped them to draw appropriate conclusions from their experimental data. Finally, analysis of METT students’ test results suggested that their gains in knowledge of modeling (a dimension of understanding of the nature of science) and science process skills supported their gains in physics knowledge.
Two studies of participatory simulations examined development of science process skills. In a pilot study by Colella (2000), urban high school biology students wearing small portable computers acted as agents in a dynamic simulation of the transmission of a virus in the closed system of the classroom. The class consisted mainly of tenth-grade students, who were described by their teacher as traditionally poor performers in science. Sixteen students, seven girls and nine boys, along with their teacher, participated in the activities.
The author’s analysis of video and audio recordings of the students as they engaged in the simulation over the course of six class sessions indicated that they advanced in science process skills. After exploring and observing the transmission of the virus in the first few sessions, they developed hypotheses about how the virus spread and then began to systematically collect data to confirm or deny their hypotheses. By the final session, they could articulate the underlying rules of the simulation.
In another study, Klopfer, Yoon, and Um (2005) examined a virus simulation that was similar to that studied by Colella (2000) and Live Long and Prosper, a simulation of Mendelian genetic inheritance. The fifth- and seventh-grade students gained an understanding of the importance of several scientific practices, such as repeated testing and revision of a hypothesis; they also increased their understanding of key concepts, such as random genetic variation.
Buckley, Gobert, and Horwitz (2006) conducted a study of BioLogica, a software system linking simulations to text in the domain of genetics. The researchers analyzed logs of students’ interactions with the instructional system as they attempted to solve science process tasks. For example, one task asked the student to manipulate the model of a genome so that a trait appears to skip a generation. Successful performance on these tasks, as well as a systematic approach to the tasks, correlated with gains in conceptual understanding measured by pretests and posttests, suggesting that development of science process skills supported the gains in conceptual understanding.
Although the committee found no evidence that the use of simulations develops students’ scientific discourse or argumentation, a few studies focusing on the goal of conceptual understanding illuminate the relationship between discourse and conceptual understanding. For example, research on BGuiLE, a simulation-based curriculum unit for high school biology, suggests that by supporting and scaffolding students’ construction of scientific arguments, the unit helps students to develop deep and accurate understanding of scientific phenomena (Sandoval, 2003; Sandoval and Reiser, 2004).
In another study, Keller et al. (2006) showed that the PhET Circuit Construction Kit simulation, which models the behavior of electric circuits, can be an effective tool for engaging students in productive discourse. Undergraduate students who were shown the simulation during the lecture demonstrated a comparatively much higher and statistically significant gain in conceptual understanding after discussing the modeled phenomenon with their peers, compared with students who were shown a physical demonstration or who were provided with an equivalent verbal explanation to discuss with their peers.
The committee did not find any research evidence about whether the use of simulations may encourage students to think about themselves as science learners and develop an identity as someone who knows about, uses, and sometimes contributes to science.
EFFECTIVENESS OF GAMES
Evidence about the effectiveness of games in supporting science learning is only beginning to emerge, and the body of evidence is much smaller and weaker than the body of evidence related to the effectiveness of simulations. Most studies have not included control groups that would allow comparisons with other forms of science instruction. The limited evidence available, based on only a few examples, suggests that games can motivate interest in science and enhance conceptual understanding. Overall, however, the research remains inconclusive.
Because one of the defining features of games is that they are fun, it is not surprising that researchers have studied the role of games in sparking interest in science and science learning. Several studies of River City suggest that it increases middle school students’ motivation for science learning (this game-based curriculum unit is described in Chapter 1).
An early study compared two Boston area public middle school classes using the pilot version of River City with two matched control classes in which students received inquiry-oriented instruction focusing on the same science learning goals. The control instruction presented the same challenge as the game—to identify the causes of diseases in a historic American city—and engaged students in designing and conducting experiments—but did not include the game. There were 45 students in the two River City classes and 36 students in the control classes, split evenly by gender. About half of the ethnically diverse study participants were English language learners, and the majority qualified for free and reduced-price meals. All participants completed a pretest and posttest focusing on affective dimensions of science learning, including scales designed to measure motivation and perceived self-efficacy. On the motivation scales, the River City group, on average, gained more from pretest to posttest than the control group. On the perceived self-efficacy scales, the River City group’s gains were significantly higher than the control group (Dede, Ketelhut, and Ruess, 2002).
Later, Ketelhut et al. (2006) compared middle school classrooms in which students were randomly assigned to one of three variations of River City with
matched control classrooms in which students received similar instruction without the game, as described above. Approximately 2,000 ethnically diverse adolescents in eight public schools participated in the study, including large percentages of minorities, English language learners, and students eligible for free and reduced-price meals. All participants completed a pretest and posttest focusing on affective dimensions of science learning; one subscale measured interest in a scientific career. The River City group gained 5 percent more on this subscale than the control group. In addition, the authors found that in the River City classrooms, students and teachers were highly engaged, student attendance improved, and disruptive behavior dropped during the 3-week implementation period. In interviews, students who played River City reported that they were motivated by the ability to conduct inquiry, along with the ability to use virtual tools, such as bug catchers and microscopes to aid in their investigations (Ketelhut, 2007).
Tuzan (2004) found that students participating in Quest Atlantis (described later in this chapter) were motivated by a large number of game elements centered on identity, play, immersion, and social relationships. Barab, Arici, and Jackson (2005) found, based on their iterative design process in creating and modifying Quest Atlantis, that a strong narrative was one element supporting engagement.
Another study focused on the introduction of the Whypox disease in the virtual gaming community of Whyville (Neulight et al., 2007). Two classes with a total of 46 sixth-grade students, including an equal number of boys and girls, joined Whyville and played the game both at home and in science class. The students attended a laboratory school affiliated with a large urban university. They were ethnically diverse (27 percent Hispanic, 13 percent black, 13 percent Asian, and 47 percent white), and two-thirds received tuition assistance. Over 85 percent had computer and Internet access at home.
In this study, Whyville was integrated into a 10-week, teacher-led curriculum about infectious diseases that also included watching videos, examining cell structures under the microscope, doing experiments, and completing worksheets. Whyville was introduced in week 3, and the Whypox epidemic arrived in week 5. When a student’s avatar contracted Whypox, its appearance changed, and the student’s ability to chat declined. The researchers videotaped the classrooms and administered a survey about infectious disease before and after the curriculum unit.
Responding to the survey, the majority of study participants (61.5 percent) reported that they were motivated to learn more about the scientific phenomenon of the Whypox epidemic by the emotional impact of the disease. Those whose avatars were infected described the experience as “terrible,” “annoying,” and “frustrating,” partly because sneezing interrupted their conversations with friends. Players increased their visits to the virtual Centers for Disease Control and Prevention to learn more about Whypox,
where they could use two simulators to make and test predictions about the spread of the disease. In a further study, Kafai, Quintero, and Feldon (2010) found that, during Whypox outbreaks, simulation usage peaked with more than 1,400 simulations performed by 171 players. The authors found that 68 percent of the players conducted some form of systematic investigation by running the simulations three or more times.
Motivation Among Subpopulations
Like the disparities in science interest and achievement among young people of different genders, races, ethnicities, and socioeconomic status, there are also disparities in the engagement of different populations with video and computer games. A recent national survey of media use by children and youth aged 8 to 18 (Rideout, Foehr, and Roberts, 2010) indicates that, in 2009, boys spent an average of nearly an hour per day playing console video games and girls spent less than a quarter hour. Boys also spent more time with computers than girls, primarily because they spent an average of 25 minutes daily playing computer video games, whereas girls spent only 8 minutes per day playing such games. Black and Hispanic youth of both genders spent significantly more time playing video games than did white youth.
In light of these disparities, research on the role of games in sparking excitement and interest in science and science learning among diverse student groups is particularly important. To date, however, only a few investigators have examined this important issue. In the large study of River City described above, all students (regardless of gender, ethnicity, or English language proficiency) were more engaged in inquiry when playing River City, and preliminary data analysis suggests that they gained as much or more in content knowledge than the students in the control group (Ketelhut et al., 2007; Nelson, 2007).
Plass et al. (2009) conducted another study that addressed interest in science (computer science) among different groups of students. The researchers created a gaming environment, called Peeps, that was designed to engage girls in learning computer programming by inviting them to design parts of the game. In the game, students (both boys and girls) play a female character that interacts with the inhabitants of the virtual world by dancing with them. Students create dances by using increasingly complex computer programming skills, and they must also avoid a character designed to steal pieces of computer code that players have developed or acquired during the game.
The 59 study participants, sixth graders in an urban school in a large Northeastern city, included approximately equal numbers of ethnically diverse boys and girls. Participants played the game during four sessions over the course of one month, completed two missions designed to assess pro-
gramming knowledge, and responded to surveys. The results suggested that playing the game increased feelings of general self-efficacy among female students and general self-esteem among both female and male students. Playing the game also appeared to increase self-efficacy in using computers among male, but not female, students. Finally, although the game did not appear to increase programming knowledge among students of either gender, it did appear to increase feelings of self-efficacy in the area of computer programming among both female and male students.
Some recent research suggests that gender differences in interest in games and responses to games may be diminishing (see Chapter 4).
Evidence about the role of games in conceptual understanding of science topics is only beginning to emerge and is inconclusive. Some examples are provided below.
Moreno and Mayer conducted a series of laboratory studies to investigate the impact of a computer game on students’ retention of science content and their ability to transfer their knowledge to solve new types of problems. In two experiments (Moreno and Mayer, 2000), undergraduate student volunteers played a computer game about environmental science that included personalized (first- and second-person language) instructional content, delivered as narrated speech by a pedagogical agent.1 Students who heard personalized content outperformed students who received neutral content on assessment questions focused on retention of science content as well as on questions focused on transfer of problem-solving skills. When Moreno and Mayer (2004) continued these studies and added a dimension that involved wearing a head-mounted display, they found that this display did not impact learning. In a follow-up study using the same game but with personalized content delivered via text (not voice), Moreno and Mayer (2007) found similar results. Students who saw personalized content outperformed students who received neutral content on questions designed to measure transfer of problem-solving skills and retention.
Another series of studies focused on the Supercharged game (Barnett et al., 2004; Jenkins, Squire, and Tan, 2004). In this three-dimensional game, players use the properties of charged particles and field lines to navigate their ship through space. Three middle school classes participated in a mixed methods pilot study comparing learning outcomes among students playing
Supercharged with outcomes among students using a guided inquiry curriculum. Average posttest scores were significantly higher for the students who played Supercharged. Anderson and Barnett (in press) continued the investigation of Supercharged with preservice elementary teachers. The control group in the study learned through a series of guided inquiry methods, and the experimental group played Supercharged during the lab sessions of the course. The students who played the game significantly outperformed the control students in terms of pre-post assessment gains.
In the study of Whyville discussed above, Neulight et al. (2007) investigated conceptual understanding of disease transmission among the sixth-grade students. Analyzing pretest and posttest results, the authors found a significant shift toward biologically correct explanations. By the end of the game-centered instructional unit, twice as many students applied biological reasoning when reasoning about natural infectious disease.
Clark et al. (2010) analyzed pre-post test data from 24 undergraduate and graduate students playing SURGE, a game focused on increasing students’ understanding of specific relationships that are central to Newtonian mechanics. The data not only reinforce the potential of games to help students learn, but also underscore their potential to reinforce alternative conceptions. The game actually resulted in a significant decrease in scores on one posttest item by unintentionally focusing students’ attention on another physics relationship that was not supported by the game. When this posttest item was excluded, the students demonstrated significant gains on the rest of the posttest. Data from interviews with the students indicate that players made successful (although variable) use of growing tacit understanding of the physics concepts involved to complete levels of the game.
There is some evidence that commercial games, not designed for science, can support conceptual understanding of science topics. For example, Holbert (2009) conducted ethnographic observations of and individual clinical interviews with children playing popular video games (Mario Kart Wii and Burnout Paradise). He found that children’s intuitive thoughts about velocity, acceleration, and momentum were activated as they played these games. These intuitive ideas have been shown to play productive roles in the development of understanding of physics (diSessa, 1993; Roschelle, 1991).
Science Process Skills and Understanding of the Nature of Science
Research on two games designed for use in science classrooms—Quest Atlantis and River City—has examined development of science process skills.
In Quest Atlantis, players use an avatar to travel to virtual places and carry out quests, talk with other users and mentors, and develop their avatars. A
quest is designed to be both entertaining and educational, as players participate in real-world and simulated activities focusing on the story of Atlantis—a complex civilization in need of help. The game includes a virtual world called Taiga Park and a story line in which the park experiences a decline in fish numbers, causing the fishing company, which generates revenue for the park, to threaten to leave. Three groups involved with the park—an indigenous population, a logging company, and a fishing company—disagree about the cause of the declining fish population. Working in small groups, students assist the park ranger by interviewing people with different perspectives on the problem, collecting and analyzing data to develop a hypothesis about the problem, and then proposing informed solutions (Barab et al., 2007).
Hickey, Ingram-Goble, and Jameson (2009) conducted a comparative study of the Taiga Park curriculum unit. A single sixth-grade teacher taught four science classes, using Taiga Park in two classes and a custom textbook addressing all of the same topics in the other two classes, over the course of 4 weeks. The teacher was in a school serving relatively high-achieving students in a Midwestern university town. Less than 20 percent of the schools’ students qualified for free and reduced-price meals and about 90 percent were white. The authors measured content and inquiry skills using two types of assessments. The first consisted of open-ended performance assessment items that asked students to solve new water quality problems and provide a rationale for the solutions they proposed. The other was a pool of randomly sampled, released achievement test items that were aligned to targeted science content and inquiry standards but independent of the water quality focus of the Taiga curriculum. The authors compared pre-post scores on both types of assessments among the Taiga Park and control classes and found that the Taiga Park classes showed significantly larger gains in conceptual understanding and science process skills. Two new types of virtual formative feedback included in the game the following year resulted in substantially larger gains in both conceptual understanding and science processes as measured by the two assessments.
Several studies of River City have also investigated science process skills. In the large comparative study described above, Ketelhut et al. (2006) administered a pre- and post-affective assessment that included questions designed to measure thoughtfulness of inquiry. On these questions, the average gains of students using two of the three versions of River City were significantly higher than those of students in the control classrooms. The authors also measured learning outcomes using a pretest and posttest focusing on content and science process skills, and found no significant difference among the three experimental groups and the control group. In a further analysis, Ketelhut et al. (in press) looked for and scored evidence of inquiry in a random sample of 224 student “letters to the mayor,” written at the end of their investigations by students in the experimental and control groups. Letters by students using
a version of River City with additional guidance scored significantly higher in overall quality than letters by students in the control and other treatment groups. They were also significantly higher in specific dimensions of inquiry, such as stating a testable hypothesis, awareness that different symptoms were related to different diseases, and stating a conclusion.
Another study suggests that one form of River City may help students gain in understanding of the nature of science. Nelson (2007) explored the impact of embedded guidance messages in River City. The author found that increased viewing of guidance messages was associated with significantly higher score gains on assessment questions related to the nature of scientific inquiry and on questions about conceptual knowledge of disease transmission.
Perceptual and Spatial Abilities
Researchers are studying how games may influence perceptual and spatial abilities that play a role in development of science process skills. For example, experimental neuroscientists investigate how “action games”—that is, fast-paced, first-person shooter games in three-dimensional environments—may influence the ability to focus on a topic of interest while ignoring all other information. Green and Bavelier (2006) conducted a comparison study of action and nonaction gamers. The action gamers spent 10 hours playing an action game (Medal of Honor, Unreal Tournament, or Call of Duty), while the nonaction gamers played a control game (Tetris or The Sims) for 10 hours. The action gamers were better than the nonaction game players in two different types of performance tasks designed to measure visual attention.
Other studies focus on visual acuity. Li et al. (2009) compared the effects of training young people for 50 hours in either an action game (Unreal Tournament 2004 or Call of Duty) or a nonaction game (The Sims 2). The action gamers experienced a marked improvement in contrast sensitivity. Calzato et al. (2010) found that, when action game players switched tasks, they lost less time than individuals that typically do not play such fast-paced, action-packed games. Finally, Dye, Green, and Bavelier (2009) found that action gamers were on average 12 percent faster than nonaction gamers at several visual tasks while being equally accurate.
Few researchers have examined whether the use of games may affect students’ scientific discourse or argumentation. In one study, Kafai (2009) analyzed chat data among Whyville players in an after-school setting following the onset of Whypox. Conversations about the disease rose dramatically and
players engaged in “serious argumentation” about the epidemic. In another study of Whyville, Kafai, Quintero, and Feldon (2010) identified increases in students’ use of type 2 vocabulary—that is, words, such as “contamination,” that are not everyday words but are also not entirely scientific (“E. coli” is a scientific term). Type 2 vocabulary has been shown to be critically important for struggling readers’ success in school (see Beck, McKeown, and Kucan, 2002).
Squire (2010) also reported increasing use of, and understanding of, type 2 scientific vocabulary among students playing an augmented reality game in science classrooms. The students used type 2 words in discussions, reports, and presentations as they played the role of scientists and gained proto-experiences of “authentic” scientific inquiry. Although this study reported findings about increased scientific discourse in classroom settings, such increases have also been identified among students playing augmented reality games in museums and after-school settings (Klopfer, 2008; Squire and Jan, 2007).
Barab et al. (2007), in the study of the Taiga Park curriculum unit in Quest Atlantis described above, found that the Taiga Park students were actively engaged in discourse related to the inquiry tasks of the curriculum and that they participated actively and productively in the inquiry practices of data gathering, negotiation, and data interpretation. Anderson (in press) found that embedded scaffolds in Quest Atlantis supported students in dialogue about the inquiry activities and in expressing as well as acquiring science content and process skills.
Steinkuehler and Duncan (2008) studied the discussion forums around the commercial, massively multiplayer, online role-playing game World of Warcraft, which focuses on fantasy themes. The authors’ analysis of nearly 2,000 posts by users in 85 different discussion threads found that 86 percent of the posts involved social knowledge construction, more than 50 percent evidenced systems-based reasoning, roughly 10 percent evidenced model-based reasoning, and 65 percent displayed approaches to evaluating information that would support argumentation as a way to construct knowledge. Steinkuehler and Duncan argue that this is evidence that even popular commercial games without a direct connection to science can support discussions and thought processes that are similar to scientific discourse and reasoning.
Games have potential to help young people identify with science and science learning. Barab and Dede (2007, p. 1) propose, “Game-like virtual learning experiences can provide a strong sense of engagement and opportunities to learn for all students, even helping learners with low self-efficacy start afresh with a new ‘identity’ not tagged as an academic loser.”
Many of the games described above engage students in playing the role of scientists, technicians, or others who need scientific knowledge to solve problems, and there is some evidence that this encourages them to identify with science. In interviews conducted as part of a River City study, students using the curriculum unit reported feeling like real scientists for the first time (Clarke and Dede, 2005). Researchers found significantly higher levels of “global science self-efficacy” among River City classes than among the matched control classes (Dede and Ketelhut, 2003), as well as significant gains in self-efficacy in scientific inquiry among River City classes (Dieterle, 2009).
Rosenbaum, Klopfer, and Perry (2006) studied 21 urban high school students playing Outbreak @ The Institute, an augmented reality game in which players take on the roles of doctors, technicians, and public health experts trying to contain a disease outbreak. Surveys, video, and interviews of the students showed that they perceived the game as authentic, felt embodied in the game, engaged in the inquiry, and understood the dynamic nature of the model in the game.
While this research suggests that some games have encouraged some students to identify with science, no evidence is available on whether these feelings of identity persist over time. Longitudinal studies are needed to address this question.
DESIGN FEATURES THAT INFLUENCE LEARNING
The research discussed above suggests that certain design features of games and simulations influence learning as students engage in inquiry and discourse and grapple with complex science concepts. These design features may be part of a simulation or game, or they may be part of the larger instructional context. The preliminary list below should not be considered definitive or complete. Because most simulations and games incorporate multiple features, it is difficult to disentangle the unique contribution of any single one (Wilson et al., 2009). Nevertheless, the committee offers these design features as a guide for continuing, design-based research on simulations and games. Design-based research is an ongoing process of developing, testing, and revising a simulation or game to enhance effectiveness.
Focus on Clear Learning Goals
In the committee’s view, carefully targeting one or more learning goals is an important design feature for both simulations and games. Science learning, including learning through inquiry, is enhanced when instruction is targeted to clearly defined learning goals (National Research Council, 2005b). Clearly defining the learning goal or goals of a simulation or game is an essential first step before considering other design features. This is
because design features that may enhance conceptual learning may not be the same as design features that aim to motivate learners to pursue careers in science (Ketelhut, 2009).
Researchers have established focusing on clearly-defined learning goals as a design principle to improve the effectiveness of simulations. For example, Clark and Mayer (2003) drew on empirical evidence to propose the coherence principle. This principle emphasizes that all elements of a simulation should be directly related to the learning goals, avoiding extraneous information that could distract the learner, disrupt the learning process, or seduce them into incorrect understandings. More recently, Linn et al. (2010) stated, based on a review of the research, that simulations should minimize irrelevant cognitive demand that could otherwise distract students from the primary learning goal. Plass et al. (2009, p. 48) state that there is enough research evidence to identify the following design principle for simulations: “The efficacy of a simulation depends on the degree to which it is in line with learning objectives.”
The more limited research on games also suggests that it is important to focus on clear learning goals. For example, in the study of SURGE described above, Clark et al. (2010) found that the game caused a significant decrease in scores on one posttest item by unintentionally focusing students’ attention on another physics relationship that was not an intended learning goal.
Clear learning goals are critical for the design of assessments to measure the effectiveness of a simulation or game (Quellmalz et al., 2009). The learning goal must be clearly established as a basis for evaluating the effectiveness of any game or simulation, and such evaluations support further research and continued improvement.
Provide External Scaffolding
To address the challenges involved in inquiry learning, research currently focuses on developing scaffolds, or cognitive tools, to support learning (de Jong, 2006). Learning scaffolds for simulations and games may be internal, including many of the other design features discussed below, or they may be external (see Box 2-1).
The research discussed in this chapter highlights the value of external scaffolding. Many of the examples provide evidence that simulations enhance conceptual understanding of science when they are scaffolded with other forms of instruction in larger curriculum units (e.g., ThinkerTools, NIELS, Biologica). Linn et al. (2010) recommend that designers embed simulations in supportive instruction as an important design principle to enhance effectiveness. This design principle is similar to de Jong’s (2005) guided-discovery principle, which focuses on addressing students’ documented difficulty in all aspects of inquiry learning, whether in the classroom or laboratory or in
Scaffolding Learning in Simulations and Games
Traditionally, scaffolding is a process by which adults or more able peers provide supportive structures to help learners perform mature behaviors before they are ready to do so on their own. Scaffolds can also be built into an activity itself, as in the example of training wheels on a bicycle. Once learners exhibit mature or independent behavior, the scaffolds are removed or faded. Taken more expansively, scaffolding can also be viewed as a progression of just-manageable challenges that enable learners to climb to greater understanding and skills. Thus, as they develop independence at one activity, a new, more challenging activity can lead to the next round of support.
New technologies create new opportunities for scaffolding, for example, with adaptive systems that provide just-in-time hints or change problem difficulty. Simulations and games can be designed to permit learners to pursue different progressions to the same outcomes, depending on various factors, including student interest, prior knowledge, and success so far. Scaffolding can be proactive and built into learners’ first attempts at an activity, or it can be reactive in response to when they are faced with a challenge that they can solve with a hint, question, prompt, or interactive resource. Games demonstrate that providing challenges and scaffolds in an appropriate balance can keep motivation high. Ideally, they also help students develop important dispositions that include identifying with scientific activities and content to help reach important science learning goals.
Building effective scaffolding is a multifaceted process. First, experts in a subject identify suitable learning tasks or challenges that will guide the learner to grapple with the important ideas or skills in productive ways. Second, it is important to develop the resource framework that learners can use to help achieve the task, for example, through experimentation, explanation, peer networking, or reading. Scaffolding is therefore provided both in the selection of the important ideas or skills and in the related educational tasks and resources that best support the learning. Third, when developing a complex set of ideas or skills, the developer must consider the progression of learning over time. Fourth, the high interactivity of games and simulations provides opportunities for contingent feedback and system responsiveness. When learners encounter a challenge or question that is beyond their immediate capacity, scaffolding of various forms allows them to make progress (e.g., hints, guidance, or simply turning off options).
a simulation (Mayer, 2004). The guided-discovery principle (de Jong, 2005) states that inquiry learning is more effective when simulations or simulation-based curriculum units provide guidance, such as domain-specific explanations or direct advice on when to perform certain actions.
External scaffolding also appears to enhance learning through games. Neulight et al. (2007) found that study participants who experienced the simulated Whypox virus in a classroom setting, in which they also learned about infectious diseases through other forms of instruction, experienced gains in conceptual understanding and in identification with the scientific enterprise. Other study participants, who played the game at home, did not advance in these two dimensions of science learning. In another study, Mayer, Mautone, and Prothero (2002) found that providing pretraining in the Profile Game before playing it, by showing players pictures of possible geological features that would need to be identified through the game, led to significantly better performance on identifying those geological features in the game.
Research on how people react to, and learn from, different forms of visual stimuli has been under way for decades. Early studies compared pictorial with text representations (Plass et al., 2009). More recent studies of simulations and games have focused on how information is represented on a continuum from more detailed and realistic to more stylized or abstract. Some research suggests that more realistic representations can be more effective than abstract symbols.
For example, Plass et al (2009) report on two experiments, both involving 80 to 90 students aged 16 to 18 in a large public high school in rural Texas. Nearly 90 percent of the students were of Hispanic descent, 40 percent were female, and they had not previously studied the topic addressed by the simulation—the behavior of a gas when heated. For the first experiment, participants were randomly assigned to one of two forms of the simulation, one of which incorporated only abstract symbols (e.g., numbers), while the other also incorporated icons—small pictures of flames representing temperature and weights representing pressure. After completing a questionnaire about prior chemistry experience and pretests of chemistry knowledge and self-efficacy, participants worked with the simulation for approximately 20 minutes. They then completed posttests of comprehension and transfer knowledge. When the authors found no significant difference in learning outcomes between the two groups, they hypothesized that it was because the learning task placed a low cognitive load (demand on working memory) on the students. For the second experiment, the investigators increased the simulation’s cognitive load by including a chart that displayed the effects
of changing the temperature or pressure of the gas. Comparing test results, the authors found significant positive differences in comprehension and self-efficacy for the group using the simulations with icons. Further analysis indicated that the added icons were especially beneficial for students with low prior knowledge of chemistry.
Other research suggests that representations that are too realistic may impede learners’ ability to transfer their understanding to another domain. Son and Goldstone (2009) conducted a series of three experiments focusing on the scientific principle of competitive specialization. First, they compared intuitive descriptions with concrete (i.e., realistic) representations and found that intuitive descriptions led to enhanced domain-specific learning but also deterred transfer. Second, they alleviated the limited transfer by combining intuitive descriptions with idealized graphical elements. In the third experiment, they found that idealized graphics were more effective for learning and transfer than concrete graphics, even when unintuitive descriptions were applied to them. They concluded that idealized graphics enhance learning and transfer when compared with highly realistic graphics. In addition, research on the two-dimensional, cartoonlike Whyville game discussed above suggests that a high degree of realism is not always necessary to support science learning. Based on their review of research on education and training with games, Wilson et al. (2009) propose that as the degree of realism of the task in a game increases, psychomotor skill learning will also increase but then level off.
Finally, representation is related to the learning goals of the simulation or game. Clear learning goals can help designers focus on the perceptual salience of the information displayed. For example, in a simulation about harmonic motion, Parnafes (2007) noted that students typically tended to attend to the perceptually salient features of the simulation rather than the conceptually important features (features an expert would attend to). This study suggests that, when designing simulations, it is important that the salient features of the simulation are ones that will be most productive in terms of the targeted learning goals.
Narrative, sometimes called fantasy, is an extremely important feature of games. It engages learners, allows them to interact with the game without fear of real-life consequences, and makes them feel immersed in the game (Wilson et al., 2009). Without a strong narrative, a game designed for informal use may not attract players and a game designed for classroom use will not generate excitement, interest, or enthusiasm for science learning. Barab, Arici, and Jackson (2005) found, based on their iterative design process in creating and modifying Quest Atlantis, that a strong narrative was one ele-
ment supporting engagement. In a further study of Quest Atlantis, Barab et al. (2007) observed that students saw an erosion diagram as part of the narrative, rather than an abstract representation of the scientific process of erosion. The authors suggest that too much narrative might hinder learning of formal scientific concepts, principles, and methods, making it difficult for students to distinguish these concepts from the particular situation in the game. Thus, game developers must carefully balance context with content.
The narrative in games designed for science learning often presents players with a question, problem, or mission that requires information to respond. Wilson et al. (2009) refer to this type of narrative as “mystery” and propose that learner motivation is positively related to the level of mystery in a game.
An extensive body of research supports the view that providing learners with feedback enhances learning, and this also appears to be the case when using simulations and games. For example, the “reflection prompts” in ThinkerTools encouraged students to reflect on their own thinking, which in turn led to gains in both science process skills and conceptual understanding (White and Frederikson, 1998). Rieber, Tzeng, and Tribble (2004) found that students given graphical feedback during a simulation on laws of motion with short explanations far outperformed those given only textual information.
Moreno and Mayer (2000, 2004) conducted a series of studies to investigate the impact of design principles applied to computer games on student retention of science content and on problem-solving transfer questions. In one of these studies, undergraduate university students played a computer game about environmental science that included personalized instructional content, delivered as narrated speech by a pedagogical agent. Students who heard personalized content outperformed students who received neutral content. In another study, Moreno and Mayer (2005) compared using the pedagogical agent to give only corrective feedback (communicating to the learner whether she or he is right or wrong) with using it to give explanatory feedback (learners were told whether or not they were correct and were also given an explanation of why the answer was right or wrong). They found that providing explanatory feedback increased retention and transfer of the targeted concepts.
Nelson (2007) conducted a River City study in which he explored the impact of embedded guidance messages on student understanding of real-world science inquiry processes and knowledge. He found that increased viewing of guidance messages was associated with significantly higher score gains on a test focusing on knowledge of disease transmission.
Clark et al. (2009) identified the degree of user control as a dimension of simulations. However, the research reviewed above suggests that user control is an important feature of games as well. The optimal degree of user control in a given simulation or game is related to the science learning goal or goals targeted. For example, the limited degree of control provided to users of PhET simulations appears to be well aligned with the goals of these simulations—to increase conceptual understanding of specific science topics.
If the goal of a simulation or game is to increase science process skills and understanding, the research suggests that the degree of user control must be carefully balanced. On one hand, providing some autonomy to design and carry out virtual experiments appears to engage and motivate users of River City and Quest Atlantis. On the other hand, students often become confused when allowed to engage in open-ended inquiry—whether in a school science laboratory or in a virtual inquiry environment (Mayer, 2004; Moreno and Mayer, 2005). Providing students with guidance along with some control—such as the feedback from a pedagogical agent described above (Moreno and Mayer, 2005)—appears to enhance learning of science processes as well as science content.
Plass, Homer, and Hayward (2009), based on their review of the research, identify manipulation of content as a design principle for effective simulations, proposing that, “learning from visualizations is improved when learners are able to manipulate the content of a dynamic visualization compared to when they are not able to do so” (p. 49). Among other studies supporting this principle is a comparative study of two forms of a chemistry simulation—one that allowed the user to manipulate the content (e.g., the temperature and pressure of a gas) and one that allowed the user to only control pacing (Plass et al., 2007). Study participants who interacted with the simulation that allowed content manipulation demonstrated larger learning gains than those who were only allowed to control pacing.
Wilson et al.(2009), in their review of the research on gaming, report that allowing learners to navigate through a computer program based on their personal preferences leads to more positive attitudes and higher cognitive outcomes (Vogel et al., 2006). They also found that game players value control at all levels, from simply picking out a wardrobe or specific facial features for their avatars to determining strategies in game play. The authors propose that increasing the amount of control given to learners using games will positively affect skill-based learning.
Individual Learner Differences
The studies discussed above suggest that differences among individuals influence how they respond to, and learn from, simulations and games. For example, students with lower science achievement, as measured by a pretest, experienced greater gains in inquiry and content knowledge after using ThinkerTools (White and Frederiksen, 1998). Plass et al. (2009) found that adding icons that represented temperature and pressure concretely (as opposed to only abstract symbols) improved understanding of gas laws, especially among learners with low prior knowledge of the topic. These findings suggest that it is important to consider the target audience when designing a simulation or game and also to include adaptive features that modify the pace and type of information, based on user responses.
LIMITATIONS OF THE RESEARCH
The preceding discussion reveals many gaps and weaknesses in the body of research on the use of simulations and games for science learning. Although both simulations and games have been used for training and education for over three decades, they have not been studied systematically (Clark et al., 2009). Rapid changes in technology and delivery platforms result in changing definitions of what constitutes a game or a simulation, making it difficult to focus the research. Another problem is that researchers do not always describe the context for the interaction with the simulation or game, including other instructional support that might be provided in a classroom setting or informally by peers, making it difficult to separate out the unique contribution of the simulation or game. In addition, researchers sometimes fail to examine or report important variables related to student abilities and attitudes, such as previous science knowledge and previous experience with simulations or games. Another limitation is that studies have usually involved small groups of students with little diversity, making it difficult to generalize the results to the large, diverse population of U.S. science students.
The studies of games and simulations reviewed in this chapter unevenly address the methodological challenge associated with how to model outcomes that are by their very nature “nested” (students within classrooms or recitation sections, classrooms within schools or universities). The authors of several studies randomly assigned classrooms to different treatments (e.g., different versions of a simulation) or to treatment and control conditions, but analyzed and reported on data from individual students. These studies must be interpreted with caution, as the analysis of student-level data may lead to findings of statistically significant effects that are not warranted.2
The questions researchers have asked about the effectiveness of simulations and games for learning, as well as the methods they have used, reflect a wide range of theoretical perspectives on how people learn. For example, the theoretical perspectives of neuroscientists studying how playing action video games affects visual response times are quite different from the theoretical perspectives underlying studies of how interactions with simulations affect understanding of science concepts. Reflecting these diverse perspectives, investigators have used a range of different research methods to measure the learning outcomes of simulations or games. The wide distribution of the published research evidence across journals in a variety of different disciplines makes it difficult to build on and extend a coherent base of research across studies and over time.
Another problem is that researchers studying games and simulations have not given enough attention to the adequacy of the instruments used to measure student outcomes (Quellmalz, Timms, and Schneider, 2009). Assessments are often designed to measure conceptual understanding alone, rather than other learning goals, and generally rely on paper and pencil tests, rather than taking advantage of digital technology to embed assessments in simulations or games (see Chapter 5). As a result, there is only limited evidence related to many of the five learning goals.
The research on games is particularly limited. Game designers often study potential users’ reactions to and experience of a game to gauge consumer acceptance, but they rarely conduct formal research on science learning. Another challenge is that games are often designed for informal learning by self-selected users. Because of these challenges, only a few scholarly studies have been conducted. O’Neil, Wainess, and Baker (2005) searched three databases for studies of the effectiveness of games for learning and training published over a 15-year period and also conducted a hand search of journals for the year 2004-2005. Among the several thousand articles about games, the authors were able to identify only 19 articles that had been published in peer-reviewed journals and provided empirical information on the effectiveness of games. Although studies have documented the effectiveness of particular games to support learning among specific populations, it is unclear whether, or to what extent, the study findings can be generalized to other populations of learners (Hays, 2005).
All of these challenges make it difficult to build a coherent base of evidence that could demonstrate the effectiveness of simulations and games and inform future design improvements. Experts do not agree on the best directions for future research and development to support science learning. The field needs a process that will allow research evidence to accumulate across the variety of simulations and games and in the face of the constant innovation that characterizes them.
Science learning is a complex process involving multiple learning goals. A simulation or game can be designed to advance one or more science learning goals.
Conclusion: Simulations and games have potential to advance multiple science learning goals, including motivation to learn science, conceptual understanding, science process skills, understanding of the nature of science, scientific discourse and argumentation, and identification with science and science learning.
There is promising evidence that simulations enhance conceptual understanding, but effectiveness in conveying science concepts requires good design, testing, and proper scaffolding of the learning experience itself.
Conclusion: Most studies of simulations have focused on conceptual understanding, providing promising evidence that simulations can advance this science learning goal. There is moderate evidence that simulations motivate students’ interest in science and science learning. Less evidence is available about whether simulations support development of science process skills and other science learning goals.
The emerging body of evidence about the effectiveness of games in supporting science learning is much smaller and weaker than the body of evidence about the effectiveness of simulations. Research on a few examples suggests that games can motivate interest in science and enhance conceptual understanding, but overall it is inconclusive.
Conclusion: Evidence for the effectiveness of games for supporting science learning is emerging, but is currently inconclusive. To date, the research base is very limited.
The available research suggests that differences among individual learners influence how they respond to, and learn from, simulations and games. Some studies of simulations have found that students with lower prior knowledge experienced greater gains in targeted learning goals than students with more prior knowledge related to these goals. Differences across gender and race in young people’s use of commercial games could potentially influence their motivation to use games for science learning; however, a few studies of games have demonstrated gains in science learning across students of different genders, races, English language ability, and socioeconomic status.
Conclusion: Emerging evidence indicates that different individuals and groups of learners respond differently to features of games and simulations.
Although the research evidence related to science learning through interaction with simulations is stronger and deeper than that related to games, the overall research base is thin. Development of simulations and games has outpaced research and development of assessment of their learning outcomes, limiting the amount of evidence related to other learning goals beyond conceptual understanding.
Conclusion: The many gaps and weaknesses in the body of research on the use of simulations and games for science learning make it difficult to build a coherent base of evidence that could demonstrate their effectiveness and inform future improvements. The field needs a process that will allow research evidence to accumulate across the variety of simulations and games and in the face of the constant innovation that characterizes them.