4
Authentic Experiences for Computing: Reviewing the Impact
The charge of this report was to investigate the promise of authentic learning experiences for computing in developing interest and competencies that would support K–12 learners’ future endeavors in computing. This chapter focuses on the outcomes identified in Chapter 3—knowledge/skills, interest, motivation, self-efficacy, belonging, engagement, and persistence—and reviews the evidence that authentic experiences can support them. The committee draws heavily on its review of the literature, which sought to identify studies, interventions, or programs that incorporate authentic approaches. Broadly construed, that is, the focus was on programs that incorporate hands-on, problem- or project-based approaches (see Appendix A for a discussion of the committee’s search strategy and studies identified). Whenever possible, we highlight studies that specifically examined the impacts of these experiences on learners who have been traditionally underrepresented in computing based on gender, race, ethnicity, or perceived ability.
It is important to acknowledge that the interventions and programs reviewed in this chapter and throughout the report cannot be identified as professionally or personally authentic as defined by the committee. This is especially true given that personally authentic experiences require a perception by the learner of being personally relevant, and it cannot be assumed that this would be uniform for all learners in an experience. However, it is valuable to understand what is currently known about the impact of authentic experiences in computing more broadly, given current attention to these approaches.
Many of the studies reviewed are still in exploratory or early stages of the research. Studies of this type help establish initial connections between
authentic experiences and outcomes of interest, but they support inferences about associations among the factors, not inferences about causation. In other words, the findings may show that there is a connection between the experience and changes in the outcome, but they do not provide insights about ways that the experience might cause changes in the outcome measure. Moreover, many of the studies do not measure change. Nevertheless, preliminary conclusions can be drawn from the well-designed studies.
What follows are sections summarizing the evidence for cognitive (knowledge and skills), affective (interest, identity/belonging, motivation, and self-efficacy), and behavioral (engagement, persistence, and retention) outcomes, respectively. It should be noted that the studies reviewed may not capture the full range of settings, including home and online communities. Moreover, within the research and evaluation literature, there is a tendency to focus on individual programs or individual curricula in ways that do not reflect an ecosystemic view of outcomes. As such, there are limitations to the generalizability.
COGNITIVE OUTCOMES
In this section we focus on two cognitive outcomes: knowledge development and skill development. Knowledge outcomes measure what is generally thought of as the theoretical or practical understanding of a subject or field. Knowledge development involves the accumulation of new knowledge as well as the integration of knowledge into conceptual frameworks. The acquisition of knowledge is important because it increases the speed and accuracy with which people can recall and retrieve knowledge and complete tasks (National Academies of Sciences, Engineering, and Medicine [NASEM], 2018b, p. 90). The integration of knowledge into mental structures is the basis for expertise in domains (NASEM, 2018b, p. 85).
Skill outcomes measure skills—the development of proficiency in the application of knowledge through training and/or practice. Skills can be further categorized as technical, such as computer programming skills, or professional and interpersonal, such as critical thinking, creativity, problem-solving, flexible/adaptability in thinking, and collaboration. Many of the studies reviewed presented changes in knowledge or skills or a combination of both.
Knowledge
In this section, the committee considers studies that examine how and to what extent authentic experiences in computing support development and improvement of knowledge outcomes. The interventions and programs studied used different platforms, and different types of technology (e.g.,
game design activities within online platforms, unplugged computer activities utilizing classroom materials, computer-aided design [CAD] activities, and specialized IT camps). Knowledge outcomes measured included awareness of computing-related fields, definitional knowledge, and other domain-specific content knowledge. Whenever it was difficult to discern whether an outcome involved knowing how to do something versus the practice of doing something (a skill), the discussion was moved to the section on skill outcomes.
Current understanding of how computing knowledge develops is hampered by the lack of consensus about the components of computational thinking and of validated assessments of computing concepts. As pointed out by Tran (2018), although some recent studies have started to assess computing, there continues to be a lack of research. Few studies sought to assess learner growth in knowledge of computing and awareness of the career opportunities afforded to those with computing skills. The studies examined were largely exploratory and did not seek to assess the efficacy or effectiveness of the intervention using experimental studies. While the reviewed studies point to various ways to increase knowledge and awareness through engaging learners in computing practices on and off the computer, caution is needed when linking knowledge and awareness to preparation for computing fields. That is, knowledge of computing concepts and awareness of career opportunities are likely not sufficient to prepare learners for computing careers.
Three studies reported gains in knowledge outcomes associated with participation in authentic experience for computing. The gains were evident in learners’ knowledge of definitions of key computing concepts (e.g., computational thinking, algorithm, sequence, branching, iteration, variable, abstraction, clarity, correctness, and efficiency, data collection, data analysis, data representation, problem decomposition, parallelization). Two of these studies integrated unplugged computer activities into the regular school day classroom (Folk et al., 2015; Ouyang, Hayden, and Remold, 2018).
For example, Ouyang, Hayden, and Remold (2018) describe the “Quest CT” intervention that aimed at helping upper elementary school (fifth and sixth grade) teachers integrate computational thinking concepts through unplugged computing activities within science lessons. Following an intensive summer professional development experience, teachers co-developed then field tested unplugged, integrated computational thinking lessons with learners in a summer camp. The following academic year teachers implemented the activities in their regular school day classrooms. There were 327 matched pre-post surveys administered: approximately half of the learners in the study were women, 34 percent were English learners, and roughly 34 percent were significantly behind grade level in reading. Gains from pre- to post-test were statistically significant for both the assess-
ment as a whole and the sub-scores in two categories: awareness of computational thinking (i.e., given a scenario, identifying the computational thinking process being used) and computational terms (e.g., data collection, data analysis, data representation, problem decomposition, parallelization, and algorithms and procedures). Even though statistically significant gains were found for all learner groups analyzed, boys showed slightly higher scores at both pre- and post-test as well as larger gains between pre- and post-test.
In another study (Jenson, Black, and de Castell, 2018), a game design intervention was used to increase students’ programming knowledge. Participants’ previous programming knowledge was a factor that produced differential outcomes—multiple regression analysis showed that pre-test programming knowledge accounted for 25 percent of the variance in post-test scores.
Studies also examined learners’ awareness of skills needed for participation in STEM fields and awareness of the fields themselves. High school learners’ (n = 95) perceptions of the need for creativity and problem-solving skills in STEM fields, particularly engineering, increased somewhat after a 2-week summer camp featuring experiences with modeling in 3D CAD software and printing 3D artifacts (Bicer et al., 2017). There were no statistically significant differences between changes among learners with different ethnic backgrounds (Asian, Black, Hispanic, White, and Native American) or between men and women.
Similarly, results of the SPIRIT program (Harriger, Magana, and Lovan, 2012), a week-long, residential, summer camp for 75 high school learners at a Midwestern university, indicated increased awareness of careers in computing. The goal of the program was to educate learners about career opportunities for people with IT skills. The intervention included hands-on experiences using the Alice storytelling environment and learning about the work experiences of IT professionals. Learners completed a survey 10 months after attending the program. Findings from learners’ self-reporting showed that the program raised learners’ awareness of computing opportunities.
Denner et al. (2012) studied 59 middle school girls who participated in a voluntary after-school computer game–making club over a period of 14 months, including 3 weeks during the summer. The club was focused on “programming, documenting and understanding software, and designing for usability” (2012, p. 242). Each of the girls created between two and five games, working in pairs for 1 hour of coding time. The rest of the time in the club was spent on other tasks such as writing in an online science journal and taking field trips to local colleges and IT schools. The primary results from the study were that game making can contribute to understanding computational concepts; and further, for novices to coding and game making, “extensive” instructional support is necessary.
Skills
In this section, the committee considers evidence on the extent to which authentic experiences support skill outcomes related to computing. In reviewing this literature, an outcome was considered a skill if it pertained to the practice of doing something rather than knowing how something was done. The distinction between computing skills and computational thinking skills is not consistent across the studies reviewed; some authors considered computing skills as a component of computational thinking whereas others view computing and computational thinking as independent topics.
Elementary-age Learners
Kazakoff and Bers (2012) examined computing skills in 58 kindergarten learners (ages 4.5–6.5) who participated in 20 hours of instruction during the regular school day using a robot programming system. The system allows learners to use interlocking wooden blocks or corresponding onscreen representations and to go back and forth between the two. In the quasi-experimental study, the experimental group showed greater positive change from pre-test to post-test in a picture sequencing assessment. Sequencing is considered a component of computational thinking.
Moore et al. (2020) studied young learners’ representation and translation between representations (including computational representations) as a precursor to algorithmic reasoning. They reported that learners in grades K–2 engaged with and moved among multiple representations as they managed the cognitive demands of various computational tasks. Through a series of case studies, they found that learners were able to translate among representations by constructing intermediary representations such as gestures and placing objects in the environment to represent current state, an early form of simulation, to manage cognitive load.
Allsop (2019) investigated the ways computational thinking could be evaluated in a classroom environment for 30 primary school learners (ages 10–11) in London in the context of a computer game design and development intervention over a period of 8 months. Outcomes of interest included computer programming, computational concepts, metacognitive practices, and learning behaviors. Participants developed metacognitive skills related to computational thinking. These included planning and choosing appropriate strategies for a solution; monitoring their work and thinking about how to improve it; and evaluating their work in terms of checking for errors. Additionally, participants were able to use programming constructs such as sequences, loops, parallelism, conditionals, operators, and event handling in making their games.
Adolescents
Fronza, El Ioini, and Corral (2015) described the design and implementation of a short course during summer school to teach computational thinking to 11th and 12th grade learners (n = 19). Participants designed and developed mobile apps using AppInventor. Use of computational thinking skills (data collection and analysis, data representation, problem decomposition, abstraction, algorithms and procedures, automation, and simulation and parallelization) was assessed through an analysis of the quality and functionality of the learners’ final projects. The results suggest that learners, regardless of their background, can exercise the computational thinking skills to walk through the process of identifying a problem to solve in the form of a functional product.
In a study of high school learners, participating in two summer academies with computational thinking enhanced maker activities. Yin et al. (2019) examined changes in computational skills and dispositions. The Assessing Computational Thinking Maker Activities (ACTMA) project developed maker activities and formative assessments that promote physics and engineering learning, and computational thinking skills and dispositions in makerspaces. The maker activities included electronic circuits, e-textiles, makey-makey, breadboards, and arduino microprocessors. The effectiveness of the activities was evaluated by comparing the pre- and post-test scores across four computational thinking skill dimensions (i.e., abstraction, decomposition, algorithmic thinking, and pattern generalization) and across the five activity types. A self-reporting portion of the survey asked learners to report on their frequency of using computational thinking skills and to provide their assessment of their computational thinking skills. The assessment also asked learners to perform activity-related tasks such as diagramming of circuits and supplying missing code snippets to program a blinking pattern on an Arduino. The results suggest that learners’ scores improved significantly on all the measures (i.e., computational thinking-integrated achievement test, frequency of using computational thinking, self-rated maker activity knowledge, and self-rated computational thinking skills).
Pohner and Hennecke (2018) researched learners’ learning of problem-solving in the context of robotics. In an exploratory case study, they analyzed the use of problem-solving strategies of teams participating in the World Robot Olympiad, an international robotics competition. Participating teams used the LEGO robotics set to build and program a robot to complete a novel task specified by the competition committee. Typically, the learner teams completed their projects over the course of 4 months. The aim of their research was to determine how successful and less successful teams differ in their problem-solving strategies. Their problem-solving process model encompassed four aspects: conceptual aspects describing issues
with the general design of the robot (hardware, theoretical); algorithmic aspects describing issues with the development of algorithms or solution approaches of the given problem (software, theoretical); construction aspects describing issues with the construction of the robot (hardware, practical); and implementation aspects describing issues with the implementation within a programming language (software, practical). The two teams that were the basis for the case studies achieved different results. Each team’s diaries were analyzed using descriptive statistics and qualitative content analysis. The successful team invested more time in the theoretical aspects (conceptual and algorithmic), capped off construction at week 15, and reserved the duration of the time for the implementation aspects, whereas the less successful team interwove construction and implementation until nearly the end of the 28-week period. In addition to evidence of problem-solving in the robotics domain, teams completed projects that required algorithm development and programming.
Several studies described outcomes associated with the development of skills related to computer science such as programming, debugging, and code commenting skills. Denault, Kienzle, and Vybihal (2008) described a 5-day computing summer camp for high school learners (grades 10 and 11) held on a university campus. The goal of the intervention was to motivate the learners to consider computer science as their future field of study. Computer game development was the context for learners to exercise problem-solving skills. Through code analysis and analysis of responses to an evaluation questionnaire, the researchers concluded that the camp had a positive impact on learning. They reported a steady growth in the code complexity over the first 4 days and restructuring and commenting of code on day five. An increase in the number of if statements on the fifth day was seen as evidence that learners made improvements to their decision trees.
Evidence of skills outcomes in authentic programs was seen in interventions of different types: game design and development, app development, makerspaces, robotics, computer aided design, and augmented reality in the context of camps, competitions, and afterschool clubs. Outcomes in the computational thinking skills category included using computational thinking, recognizing computational thinking, and identifying problems to be solved through computational thinking, whereas outcomes in the computing skills category included computer programming, debugging, and commenting; analyzing technical requirements; and designing for interaction. Meta-cognitive skills such as planning and choosing appropriate strategies for a solution; monitoring one’s work and thinking about how to improve it; and evaluating one’s work were also reported. Several studies focused on outcomes related to the development of computational thinking skills at different ages suggest that these skills can be nurtured in school-age learners
in age-appropriate ways. The evidence base is limited—few programs attempted to assess learner skills development and those that did were not utilizing common assessments and common definitions of computational thinking skills.
Summary
Overall, the evidence of outcomes in knowledge and skill development is emergent. Most studies we reviewed were exploratory and took a design-based research approach to refining curricula and/or pedagogy to enhance learner engagement or were early pilots of such interventions. Few studies applied rigorous approaches to evaluating the efficacy of effectiveness of interventions on learners’ learning.
AFFECTIVE OUTCOMES
Individuals’ decisions about subject areas to study and careers to pursue are influenced by a host of affective and contextual factors (Dou and Gibbs, 2013). For example, early STEM interest appears to be a predictor of later STEM interest and eventual choices of careers in STEM. Beyer (2014) documents the connection between interest, self-efficacy, and subsequent college-level computing course-taking. Similarly, Denner et al. (2014) showed relationships between affective factors and course-taking at the community college level.
This section reviews the findings from the literature on affective attributes hypothesized to be important outcomes of authentic experiences for computing, specifically interest, identity/belonging, self-efficacy, and motivation. The discussion of each outcome addresses the measurement issues as well as how participation in authentic experiences in computing affect these outcomes. It is important to note that most of the studies reviewed had sample sizes that ranged from 10 to 100 students; however, a few self-efficacy studies noted larger sample sizes.
Interest
In the studies reviewed, interest was typically measured through the use of self-report surveys that asked participants direct questions about their perceptions of the experiences and their inclinations to pursue future computing activities. Questions asking participants to agree/disagree with statements such as, “I could see myself pursuing a career in computing or computer science” are typical. Sometimes, data are collected from observers or through more open-ended mediums such as journal writing and feedback from camp instructors.
All experiences, programs, and activities in these studies took place in out-of-school settings. The most common setting was in summer camps that lasted between 5 days and 2 weeks. In some instances, the summer camp experience was combined with after-school sessions for 21 weeks or combined with Saturday sessions for about 2 years (i.e., Ladeji-Osias, Partlow, and Dillon, 2018). The duration of the authentic experiences in computing were highly variable. In some cases, experiences included bi-weekly 2-hour sessions for 4 months; five Saturday sessions for 3 hours/session; 1-day workshops; and 30–45-minute sessions. This variability makes it difficult to evaluate program effects.
Many of the studies reviewed reported no significant change in interest from pre- to post-activity (Bugallo, Kelly, and Ha, 2015; Denault, Kienzle, and Vybihal, 2008; Krayem et al., 2019). However, in some cases, it is possible that no change in interest is related to a ceiling effect (Bugallo, Kelly, and Ha, 2015); that is, participants already had a high degree of interest and even if participants enjoyed the activity, there is no way to be able to measure this possible change.
Use of wearable technologies and e-textiles is one approach researchers are exploring for engaging learners in computing and increasing their interest (Buechley and Eisenberg, 2008; Buechley et al., 2008; Merkouris, Chorianopoulos, and Kameas, 2017; Qiu et al., 2013). For example, Lau et al. (2009) examined outcomes for 25 middle school learners in a workshop-style summer camp activity that lasted 5 days and covered electronic circuit theory, t-shirt circuit design, integrated circuits, and programming. The overall task for participants was to make their own interactive garment, wear it, and present how they built it to their peers. Feedback from camp instructors as well as results of a pre-/post-survey of participants indicate that although learners did not develop interest in learning about electrical theory, they did develop interest in computing. In particular, participants became excited about the wearables, which led them to further experiment on their own, and the learners indicated the intention to participate in future computing courses.
Importantly, a few studies focused explicitly on girls and learners from minoritized racial/ethnic groups (Gardner-McCune et al., 2013; Jageila et al., 2018; Ladeji-Osias, Partlow, and Dillon, 2018). However, across the studies, the findings were mixed.
In a study of 22 girls (ages 13–18) who participated in five 3-hour workshops across consecutive Saturdays, Jageila et al. (2018) did not observe significant changes between pre- and post-program responses about participants’ interest in a career in computing. This may be due to participants’ high levels of interest at the start of the program.
Gardner-McCune et al. (2013) reported on a program aimed to engage 9th–12th grade African American and Latina learners in research and design
activities (85% of participants were girls in these underrepresented groups). Sixty-four learners were enrolled in the program at some point; however, not all of them participated in all phases of the program. The program was divided into three 5–7-week phases during which participants were taken through a series of activities to help them learn how computers work and process information and how to develop proposals for new applications or devices. They learned about development environments and creation of innovative devices and applications. Learners exhibited and presented their work through showcases, public competitions, and research poster sessions at technical conferences. While data are not available for all of the participants in the program, the available data indicate that slightly over half of the participants have majored in or intend to pursue a STEM or computing major.
Ladeji-Osias, Partlow, and Dillon (2018) reported on a program for middle-school-aged African American boys, which involved a 4-week summer program and 10 Saturday sessions during the academic year. The long-term goals of this project were for participants to create products using 3-D modeling software and printers, develop software and embedded applications, enhance computational thinking skills, and pursue related entrepreneurial ventures. Data were collected through a participant survey and observations by an external team. The results from the survey revealed an increase in participants’ responses to the question “I understand how to use software to create an app” as well as interest in pursuing “a career in app development.” However, only 30 out of 38 participants completed the final survey.
Identity/Sense of Belonging
As highlighted in Chapter 2, development of a computing identity is related to a sense of belonging to the field (Dempsey et al., 2015; Rodriguez and Lehman, 2018) and influences interest and persistence (Brickhouse, Lowery, and Schultz, 2000; Carlone and Johnson, 2007; Rodriguez and Lehman, 2018; Seyranian et al., 2018; Vincent-Ruz and Schunn, 2018). Interest, motivation, and participation in computing may ebb over time, especially if learners develop a sense that they do not belong in the field, whether STEM in general or computing in particular. In this section, we talk about the outcome commonly referred to as STEM identity or belonging, which is how a person views oneself in terms of being a mathematician, scientist, engineer, or perhaps a computer programmer.
Identity development can involve multiple components including change in perception of who should pursue STEM fields and change in perception of an individuals’ own fit in STEM. Studies that examine identity as an outcome used a variety of data collection strategies including observations, online activities, video and audio recordings, semi-structured
interviews, and researcher reflections. Sample sizes for the studies ranged from 17 to 93 learners.
Ahn et al. (2014) conducted a longitudinal study on an after-school program for middle school students called Sci-Dentity.1 The program utilized science storytelling and news media to engage learners in STEM ideas and examined learners’ development of personal STEM identity over an 18-month time period. Some of the activities related to computing and influence computing-related outcomes. Using case-study methodologies (e.g., video recordings of each session that they participated in, observations notes, audio recordings of conversations with these learners, artifacts produced by these learners, and researcher weekly meeting reflections), the researchers mapped the identity trajectories for the two Black boys who participated. One of the boys expressed a desire to become a “game designer or scientist.” He disclosed that he knew of particular game design companies in his home state that he aspired to work for and had also done independent, online research to identify colleges that had game design degree programs. The other boy did not get the same benefit from Sci-Dentity. He said he participated in the program because it gave him a connection to his friends, and shared that he probably will not go to college.
Two studies focused on the Digital Youth Divas project, an out-of-school program that seeks to engage young girls (predominantly Black and Latina learners) in design-based engineering and computing activities. This program includes four components: (1) a curriculum clustered into three project units—e-fashion, e-paper, and e-dance that combines computational and digital artifacts to produce creative artifacts; (2) video narratives featuring racially diverse women characters that parallel the project activities in the units (e-fashion, e-paper, and e-dance); (3) a private online social learning network used to share and showcase their work and interact with adult mentors who pretend to be these fictional narrative characters; and (4) distributed online and offline mentorship—diverse in-person mentors encourage participation and completion, and online mentors offer feedback and advise the girls on the next steps.
One study of the results of this program included 29 girls who participated. Through a pre- and post-program survey that measured access, interest, experience, and perception, the researchers found that the participating girls identified stereotypical descriptions of those who should pursue STEM fields (people who are good in math and science). At the post-survey, these stereotypes declined as three times more girls ranked people who are artistic in the top three as people who can pursue STEM careers. The
___________________
1 The research described the program as being implemented in a large, urban school district where approximately 90 percent of students come from minority groups and nearly 77 percent qualified for free and reduced meal (FARM) programs.
study reported there was no change in girls’ perception of their own place in STEM, however.
In another iteration of this work, Pinkard et al. (2017) worked with 17 Latina and Black girls at two elementary schools serving low-income populations, where they examined the connection between the co-designed narrative materials created and identity development. Through qualitative analysis of in-person observations of the design sessions, summaries of the girls’ online activities, and semi-structured interviews with the participating girls, the researchers found evidence that narratives motivated girls to continue with challenging STEM activities and STEM identity development. The details presented through the characters in the narratives created opportunities for the participants to reveal their stereotypes and awareness of limiting gender, racial, and intersectional and multiple identities in the narratives’ storylines.
Choudhury, Lopes, and Arthur (2010) reported findings from experiences with an IT Career Camp developed in partnership with several businesses and corporations. The camp was designed for high school learners with the goal of helping them view IT/information systems work as exciting and creative. Each day of the camp included a visit to the businesses and corporations where learners took part in a hands-on activity to solve a problem involving IT, as well as a competition between teams to solve such problems. The learners also worked on a service-learning project. Teachers and counselors served as advisors during the camp. Through pre- and post-camp surveys and interviews, the researchers found that the learners’ perceptions of IT professionals changed significantly. Many learners began the camp with a very stereotyped, unfavorable image of IT professionals (i.e., “[they] work in a ‘back closet’ and are ‘very pale, with glasses’”), and were reluctant to share with their friends that they were attending an IT camp. At the end of the camp, their opinions toward IT professionals were more positive (i.e., “I thought IT was something of a nerd thing, but my opinion has changed.”). However, the camps did not significantly influence participants’ decisions to pursue an IT career or IT major.
Motivation
Self-determination theory posits that there are two main types of motivation: intrinsic (motivated by internal factors) and extrinsic (motivated by external factors or influences). This report focuses on studies that measure intrinsic motivation (Ryan and Deci, 2000; 2017), as researchers have stressed the necessity of cultivating intrinsic motivation to counter ongoing challenges of novice programming (Pinkard et al., 2017).
Change in motivation can be captured in multiple ways, including a desire to learn more about computing or an intention to take computing
courses in the future or pursue a career in computing (or cybersecurity). A few different types of methods were used for data collection: self-reported surveys administered pre- and post-treatment; a survey administered only post-treatment; and triangulation of data from observations, semi-structured interviews, focus groups, and field notes. Researchers’ findings on change in motivation in each study varied, ranging from no change in motivation (such as in Bugallo, Kelly, and Ha, 2015) to an increase in motivation (such as in Jin et al., 2018; Klopfer, Yoon, and Rivas, 2004; Lau et al., 2009; Scott and White, 2013).
Bugallo, Kelly, and Ha (2015) developed a 2-week summer camp for high school learners that provided rigorous instruction and hands-on engineering tasks designed to solve everyday problems. Some of the engineering tasks included building a line-following robot and fabricating a fiber optic voice link. Results of a pre-/post-survey of 38 participants in the camp showed no significant change in learners’ career motivation in electrical and computer engineering.
Jin et al. (2018) report on a 1-week GenCyber summer camp that involved 181 high school learners. Slightly over half of participants (51.3%) were Black and Latinx, and there were about twice as many men as women. The camp is modeled on game-based learning and hands-on lab experience. Participants engaged in four different modules:
- A social engineering and information security module to raise general awareness of social engineering scams;
- A secure online behavior game that allowed learners to handle email messages, text messages, weblinks, and phone calls appropriately, using various computing devices such as school computers, mobile phones, laptop computers, and networked game consoles;
- A Cyber Defense Tower Game that allowed learners to protect their virtual computer server from the different cyber-attacks;
- A computerized version of a physical card game about cybersecurity. The computer-based card game is a single-player version of the physical one.
Each GenCyber camp day consisted of four 90-minute sessions with two 15-minute group discussions. On a post-camp survey completed on the last day of camp, 154 participants indicated that game-based learning for cybersecurity motivated them to pursue higher education and careers in the field of cybersecurity.
Scott and White (2013) reported on a culturally responsive multimedia program called COMPUGIRLS that holds sessions after school and during the summer. For this study, 41 Black and Latina girls, aged 13–18,
from high-needs districts participated.2 COMPUGIRLS utilizes multimedia activities as a means of developing computational thinking, enhancing socio-technical analytical skills, and providing a dynamic, learning environment that nurtures the development of a positive self-concept. Through the course of the project, the girls created a digital product using various tools and a research paper, which included research questions, peer-reviewed references, primary data, analysis, and implications of their findings.
Qualitative data were collected in the form of field notes from observation, focus groups, and interviews. Two primary themes emerged from the qualitative data: (1) girls were empowered by the challenge of learning and mastering the technology; and (2) manipulation of the technology and engagement in the learning experiences became a form of self-expression and exploration, which encourages the enactment of a social justice agenda and a way to inform their community and peers.
Self-Efficacy
As described by Bandura (1997), self-efficacy comprises an individual’s belief in their ability to perform and successfully complete a task. Self-efficacy is important because it can promote interest and motivation and shape career aspirations. It has been shown to correlate with academic achievement (Aivaloglou and Hermans, 2019; Dunn and Kennedy, 2019). Researchers have used various instruments to measure self-efficacy in K–12 STEM education; across studies the results seem to strongly suggest that self-efficacy is related to learner persistence in STEM pathways (Brown et al., 2016; Concannon and Barrow, 2010).
Change in self-efficacy was captured in ways that were specific to the context of the program studies and the specific activities: in electronics, circuitry, coding, and programming self-efficacy; in performing robotics and cybersecurity-related tasks; and comfort in using newer technology. Data were collected via self-reported pre- and post-treatment surveys; cybersecurity engagement and self-efficacy survey; weekly learner journals; and observation. Several of the studies reported increases in learner self-efficacy following participation in a variety of hands-on, potentially authentic, computing experiences (Amo et al., 2019; Barker et al., 2018; Jagiela et al., 2018; Nugent et al., 2016; 2019; Stapleton et al., 2019). The results for gender differences
___________________
2 As Scott and White (2013) describe, “In the contextualized aggregate, these districts represent the typical characteristics of an urban school. Except for one of the high school districts, the other two host students who for the large part qualify for free or reduced lunch (82–90%), disproportionately serve racial and ethnic minority students (African American 10%, Latino 62%, White 19%, and Native American 3%), and have less than 50% of high school students passing statewide assessment” (pp. 662–663).
were mixed (Loksa et al., 2016), and differential impacts on learners from underrepresented racial and ethnic groups were rarely examined.
Qiu and colleagues (2013) investigated the impact of a 12-hour project-based computational textiles curriculum on learners’ technological self-efficacy. The researchers were particularly interested in this approach as a means to diversify the computing community by drawing upon non-traditional applications that might interest a broader group of learners. Computing concepts and practices targeted included abstraction and modularity, computer architecture, variables, control flow, functions, data structures, debugging, and iteration. Evaluation results indicated a positive impact on students’ technological self-efficacy as well as interest in programming and electronics.
Nugent et al. (2019) examined the WearTec intervention, an electronic textile curriculum delivered to upper elementary learners, primarily in formal classroom settings. The sample size was large, with 800 learners participating in lessons and activities utilizing LilyPad Arduino programmable microcontrollers along with conductive thread, LED lights, motors, and switches to create a textile-based product of their design. Overall, participants’ self-efficacy increased over time. Interestingly, although e-textile interventions are commonly described as appealing particularly to girls, boys had significantly higher self-efficacy following the WearTec intervention, even after controlling for pre-treatment scores and despite no gender differences in knowledge outcomes. Learners from minoritized groups had similar increases in self-efficacy as compared to their White peers, although they made up only approximately one-third of the participants.
Nugent et al. (2016) examined how multiple versions of their robotics program across 8 years—delivered through summer camps, academic year clubs, and robotics competitions—supported middle school learners in STEM learning and motivation. They reported results from 6 years of data collected from 1,825 campers, 3 years of data from 458 competition participants, and 2 years from 126 club participants. Participants were from 23 states in the United States, 70 percent were boys and 30 percent were girls. Participants completed a survey that measured four constructs:
- task value: perceived value of science, mathematics, and robotics;
- self-efficacy: confidence in performing robotics tasks;
- workplace skills: teamwork and problem-solving; and
- career orientation: interest in STEM careers.
The researchers also included a quasi-experimental study that compared results from the summer camp to a control group composed of learners identified by regional educational service units as those with interest in technology and robotics.
The researchers highlight two results. First, the camp data showed more positive results in the increase in interest in engineering. There were increases in learner interest in engineering careers in 2 of the 3 years of competition data, but not in science, technology, or mathematics. The clubs did not show any increases in learner interest in pursuing STEM careers. Participation in robotics camps, clubs, and competitions tended to increase learner self confidence in performing robotics tasks. Participants showed gains in self-efficacy as they accumulated experience in writing programs to control their robot’s actions effectively.
Summary
Overall, the evidence for affective outcomes appears to be mixed, despite showing promise in some areas. With respect to interest, most of the studies were still in the exploratory/early stage phase and suggested little change. The same is true for motivation with most of the studies exploring themes that may contribute to enhancing motivation through STEM experiences. The research for identity and self-efficacy was more promising. Findings suggest that prolonged exposure to STEM experiences lead to significant increases in self-efficacy.
BEHAVIORAL OUTCOMES
There is widespread belief that a necessary condition for expanding participation in careers and majors in computing is broadening exposure to and engagement with computing (Folk et al., 2015). As the reasoning goes, engaging in computational thinking—in the contexts of technology use, computing, or more general STEM applications—can help build the skills needed to understand and work effectively with computers. Adding direct, hands-on experience can further build these skills and can also foster interest and comfort with computing. These experiences, in turn, can then increase the likelihood of pursuing—and persisting in—computing studies and careers.
The literature describes a wide range of strategies for ensuring that K–12 learners are exposed to opportunities to engage in computing. In some cases, the strategies entail use of technology and other times not. They generally target learners in the middle and high school age range, and have been designed for a variety of settings, ranging from traditional classrooms to small-group workshop-like settings. Most of the studies reviewed were exploratory or in early stages with a subset including evaluations of impact. As such, very few studies have taken the extra step of examining the effects of these strategies on improving engagement in computing; there is even less evidence on strategies that promote persistence. These studies are discussed below, organized by outcome focus.
Engagement
Participating in authentic activities in STEM, and computing more specifically, is believed to be helpful in fostering continued engagement in the activities themselves (see Chapter 2). In turn, this greater engagement, coupled with the skill-building focus of the activities may encourage learners to pursue STEM careers, including computing and computing-related career tracks. Whereas there is a robust literature describing the nature and logic of the myriad strategies designed to increase engagement in STEM activities and/or course taking, there is a dearth of research on the effectiveness of authentic experiences for computing in promoting engagement in computing specifically.
Several exploratory studies examined descriptively the level of learners’ engagement in various types of programs but did not attempt to assess whether participation in a given program affected the likelihood or level of engagement relative to what it would have been absent the program (Folk et al., 2015; Gardeli and Vosinakis, 2019; Wanzer et al., 2020; Weston, Dubow, and Kaminsky, 2019). For example, Folk et al. (2015) focused on a program designed to infuse middle and high school science classes with exercises designed to build computational thinking skills applicable in a wide range of settings. This program, Discover Science through Computational Thinking, did not entail the use of technology. Rather, it involved enhancements to instructional modules that required learners to develop algorithms or follow explicit logic trees to solve problems. The study, which focused on 16 middle and high school teachers and their learners, used pre- and post-tests of learners and qualitative feedback from teachers, and learners to judge the impact of the program on learner engagement and learning. The evaluators concluded that the strategies were successful in engaging the learners in the computational thinking exercises. However, they also noted that learners often needed prompting to engage with the task and frequently needed support to complete the tasks.
Two studies explored descriptively the effectiveness of game-based strategies for promoting engagement in computing and computational thinking. One of these, ARQuest, is a mobile augmented reality game for middle-school learners. ARQuest blends game play and game design activities with real-time feedback intended to build computational thinking skills (Gardeli and Vosinakis, 2019). The study focused on 26 learners ages 9 and 10 across two schools. Based on observations of the game play during a single session (25 minutes for three groups of learners and 1 hour for the fourth group), the evaluators judged that the game successfully engaged learners meaningfully in computational thinking. They did not examine whether the game led to higher levels of engagement in computational thinking or whether participants were more likely to enroll in computing courses or majors later on.
The other study of a game-based learning strategy examined the use of the mobile game, Meleon (Klein, 2013). This game was designed as a structured mobile game environment intended to promote computational thinking and skill-building by inviting learners to play the game on their own using their local environment as the environmental inputs. The evaluation of Meleon is based on the experiences of 15 10- to 15-year-olds in a youth play club. The evaluators observed learners playing the game for an average of about 20 minutes each. Based on observations and semi-structured interviews with learners, the author concluded that learners were highly “immersed” in the game and able to reflect on their actions during play. However, like the study of ARQuest, there was no comparison group and no evaluation of the impacts of Meleon on more distal outcomes.
Two of the studies identified looked at the impact of the program on engagement in computing activities. Amo et al. (2019) conducted a randomized controlled trial of a 60-minute hands-on workshop in networking (“cyber detectives”) for learners in middle school and from low-income families. The study found no growth in cyber awareness as a result of their participation in the exhibit (n = 79).
The other impact evaluation study (Kloper, Yoon, and Rivas, 2004) examined whether there were differences in levels of engagement and skills development of learners who played games—Virus and Live Long and Prosper—when delivered via Palm Pilot devices supported simulations versus using wearable Tag-based simulations. Tag-based simulations involved participants wearing a small computer that allows them to become agents in the simulation, whereas Palms involve participants using a personal digital assistant. In a randomized controlled trial that included 188 11- to 16-year-olds in three schools, the evaluators found no difference in learner engagement between those using the Palm- versus the Tag-supported simulations. The study did not include a no-treatment control group and, thus, by design, did not estimate impacts on engagement per se.
Persistence and Retention
To meet the workforce demands of the future as forecasted by the Bureau of Labor Statistics (2020), it is critical to increase computing literacy among the population, as well as to have greater numbers of individuals prepared for computing careers. Moreover, Dee and Gershenson (2017) highlight that it is important to promote equitable access to computing careers among underrepresented groups. There are too few learners exiting high schools with the skills required to pursue computing majors in college, and many who have the skills lack the interest and/or self-confidence to enter computing majors. This suggests the importance of finding ways to increase the likelihood that more of those who gain exposure and have
predispositions to engage in computing majors and careers persist in preparing for them.
A recent observational study explored predictors of the decisions of high school girls with interest in computing to act on that interest by persisting in computing majors during college (Weston, Dubow, and Kaminsky, 2019). This study surveyed learners who had demonstrated an interest in computing by registering on the Aspirations in Computing Award website when they were in high school. Using data from a 3-year follow-up survey, the study showed that the strongest predictor of persistence was having taken an AP computing course in high school. Notably, participating in tech-related workshops, internships, or after-school programs was negatively associated with persistence in computing majors.
The other study identified in the literature search examined the impacts of a particular web-based learning platform for teaching beginning programming, EarSketch, on intentions of high school learners to persist in computing (Wanzer et al., 2020). The program was developed to foster persistence in computing among diverse population groups (i.e., learners not identified as White or Asian race/ethncity) through a variety of applications, including brief workshop settings, summer camps, and even in full-semester courses. Two of the Wanzer et al. (2020) studies focused on high school learners who had taken a computing class that incorporated EarSketch (DRK studies of a 2015–2016 and a 2016–2017 cohort). Both studies had a treatment and a control group and measured impacts by comparing the difference between learners’ self-reported intentions to persist in computing before and after the EarSketch experience. Both treatment groups had higher average gains in their reported intentions to persist in computing than their control-group counterparts. However, in neither case were the gains large or significantly higher for the treatment group.
Summary
Overall, the studies reviewed with respect to engagement suggest that the findings were generally mixed and/or insignificant. Only one study (Folk et al., 2015) reported positive impacts on long-term learner engagement; the others only showed short-term impacts (Gardeli and Vosinakis, 2019; Klein, 2013) or none at all. Many of these studies did not include control groups or measure more long-term impacts. For retention, the findings suggest that engaging in computing courses (such as AP computer science) or computing experiences may lead to persisting in computing, although more research is needed to further interrogate these findings.
SUMMARY
In this chapter, the committee reviewed the literature on the relationships between authentic experiences for computing and desired outcomes, such as improvements in knowledge and skills about computing, changes in interest in computing, and motivation to pursue future activities in computing. The results were limited and, in some cases, suggested positive impact. However, many of the studies were exploratory and were not designed to establish causal conclusions about the relationships between the interventions and changes in the outcomes. Additionally, unlike the evidence described in Chapter 3 that discussed connections across settings and time (see section Learning Over Time and Across Experiences: An Ecosystems Approach), these examples mainly examine programs in isolation. Nonetheless, they provide a foundation for future studies.
This discussion of research on the outcomes of authentic experiences in computing lead to several insights. There is substantial variability in the settings and measures making standardization in terms of measures and interventions difficult. For example, measures of cognitive outcomes (knowledge and skills) were typically designed to measure specific learning outcomes directly related to a given curriculum, programmatic activity, or course. Most of these were not standardized measures or based on large-scale assessments (at the state, national, or international level), rather they were developed by the program designers. Affective outcomes involved measuring engagement, interest, self-efficacy, and the development of an identity as a computational thinker; outcomes that can be difficult or impractical to measure. These limitations in measurement likely contribute to the lack of significant findings.
Despite the limitations in the research base, the past decade has seen an increase in research and development activities related to computing. Future research includes developing programs to expose learners to authentic experiences for computing as well as better articulation between the goals and objectives for the programs and design mechanisms to evaluate the extent to which they have been reached. This alignment is needed to better understand how the program leads to the intended outcomes. Overall, more research is needed.
