Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors (2021)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Appendix A Search Strategy and Data Coding The committee sought to assemble a set of studies that represented the extent of available evidence. The review focused on literature and programs centered around three categories of outcomes: (1) affective (such as interest, identity/belonging, motivation, and self-efficacy); (2) cognitive (such as knowledge and skills); and (3) behavioral (such as engagement, persistence, and retention). A search was conducted through Scopus requesting studies from the past two decades (2000–2020) and limited to English. Table A-1 shows the syntax used to identify studies. The search parameters were set to be broad and inclusive, which yielded a large number of studies (N = 2974). The list was culled down by removing studies that did not meet committee-determined standards for quality and relevance. Specifically, studies were classified based on the following attributes: goal, intervention type, study type, findings reported, outcome category, STEM field, methods, setting, authentic, and the age/grade range. To determine the type of study, the committee used the common guidelines for education research and development (Earle et al., 2013). Box A-1 briefly documents the guide and summarizes the different study types. From this analysis, 309 studies were identified as potentially relevant and requiring additional review. The committee then reviewed the abstracts for each of the 309 studies. Table A-2 illustrates the systematic culling down of studies that were reviewed at each stage whereas Table A-3 highlights the ones described in the chapter on outcomes. In Table A-2, studies with measurable impact estimates are those studies that can be characterized as Impact and Effectiveness Research based on Box A-1. Prepublication Copy, Uncorrected Proofs A-1 

BOX A-1 Types of Research and Their Purposes Foundational Research: Studies of this type seek to test, develop, or refine theories of teaching or learning and may develop innovations in methodologies and/or technologies that will influence and inform research and development in different contexts. Exploratory/Early-stage Research: Studies of this type investigate approaches to education problems to establish the basis for design and development of new interventions or strategies, and/or to provide evidence for whether an established intervention or strategy is ready to be tested in an efficacy study. Studies in this genre should establish initial connections to outcomes of interest (usually correlational rather than causal) and support the development of a well- explicated theory of action that can inform the development, modification, or evaluation of an intervention or strategy. Design and Development Research: Studies of this type draw on existing theory and evidence to design and iteratively develop interventions or strategies, including testing individual components to provide feedback in the development process. Typically, this research involves four components: (1) development of a solution based on a well-specified theory of action appropriate to a well-defined end user; (2) creation of measures to assess the implementation of the solution(s); (3) collection of data on the feasibility of implementing the solution(s) in typical delivery settings by intended users; and (4) conducting a pilot study to examine the promise of generating the intended outcomes. Impact and Effectiveness Research: Studies of this type assess how well a program, policy, or practice works. It uses comparisons of individuals who do and do not experience the intervention to judge the impacts. In some cases, impact evaluations are designed to determine effectiveness under ideal conditions (Efficacy Research). Impact evaluations also may test the effectiveness of a strategy or intervention under circumstances that would typically prevail in the target context (Effectiveness Research) and, in other cases, it may be to test the scalability of the intervention across a broad spectrum of populations and settings (Scale-up Research) or the replicability of the studies in similar or different contexts (Replication Research). SOURCE: Adapted from Earle et al. (2013, pp. 1–53). Prepublication Copy, Uncorrected Proofs A-2 

TABLE A-1 Search Syntax for Locating Studies Search Syntax Strategies and Yield Criteria and Syntax 1 2 3 4 5 6 7 8 9 Computing focus (STEM X X X STEM OR science OR technology OR engineering OR mathematic* OR X X X X X X computing OR “computer science” OR “computer program*”) AND Personalized learning focus (“hands-on” OR “design based” OR “maker ed” OR “maker space” OR “place based” OR authentic OR “technology education” OR “project X X X X X X X X X based” OR “problem based” OR “activity design” OR “maker movement*” OR “problem solving” OR “game design”) AND Setting (“in school” OR “out of school” OR formal OR informal OR librar* OR museum* OR “community setting*” OR YMCA OR “boy scout*” OR X X X X X X X X X “girl scout*” OR club* OR “summer camp”) AND Target youth (“K-12” OR “elementary school*” OR “middle school*” OR “high school*” OR child* OR youth* OR adolescent* OR teenager* OR X X X “young student*”) (student* AND NOT {college* OR undergraduate* OR graduate* OR X X X university}) AND Focal outcomes (engagement OR {time on task} OR “future course*” OR “future class*” OR “future club*” OR persistence OR “choice major” OR “curriculum X X X choice*” OR “class choice” OR “career choice”)) (interest OR awareness OR attitude*)) X X X (knowledge OR skill* OR disposition* OR network*)) X X X AND Recency PUBYEAR AFT 2000 X X X X X X X X X Citations returned = 2,974 172 74 154 382 179 331 655 225 802 a Focal outcome were aligned with commonly stated program goals. Prepublication Copy, Uncorrected Proofs A-3 

TABLE A-2 Search Summary Abstracts Studies Studies Studies with Retrieved & Retrieved & Referenced in Credible Impact Outcome Domain Reviewed Reviewed the Text Estimate Affective 107 76 24 3 Cognitive 149 102 37 1 Behavioral 53 40 16 3 Total 309 218 77 7 Prepublication Copy, Uncorrected Proofs A-4 

TABLE A-3 Evaluation Studies Identified Through the Literature Search Author (Date) Study Type Intervention/ Practice Setting Grade/ Age Group Outcomes for Impact Estimates Affective Outcomes Barker et al. (2018) Exploratory Wearable technologies After-school, on site Elementary; Self-efficacy Middle school Bugallo and Kelly Exploratory Participatory STEM Summer program/ High school Identity; Self-efficacy (2014) activities camp Bugallo et al. (2015) Exploratory Participatory STEM Summer program/ High school Interest; Motivation; activities camp Self-efficacy Choudhury et al. (2010) Exploratory Participatory STEM Summer program/ High school Identity activities camp Denault et al. (2008) Exploratory Participatory CS activities Summer program/ High school Interest camp Doerschuk et al. (2013) Exploratory Participatory CS activities Summer program/ High school Interest camp Evans and Schares Exploratory Maker spaces/ activities University K–12 Interest (2017) Gardner-McCune et al. Exploratory Participatory CS activities University High school Interest (2013) Harriger et al. (2012) Exploratory Participatory CS activities Summer program/ High school Interest camp Jagiela et al. (2018) Exploratory Participatory STEAM University High school; Interest; Self-efficacy activities Middle school Krayem et al. (2019) Exploratory Participatory STEM University High school Interest activities Ladeji-Osias et al. Exploratory Participatory STEM Summer & Saturday Middle school Engagement; interest (2018) activities Program Monterastelli et al. Exploratory Participatory STEM University High school Interest (2008) activities Nugent et al. (2016) Exploratory Participatory STEM Camps, Clubs & Middle school Self-efficacy activities Competitions Prepublication Copy, Uncorrected Proofs A-5 

Nugent et al. (2019) Exploratory Wearable technologies Classroom— Elementary school Self-efficacy; programming traditional & after (K–5) design & knowledge school Stapleton et al. (2019) Exploratory Participatory STEM University; Summer Middle school Self-efficacy activities camp Ahn et al. (2014) Design & Participatory STEM After-school, on site Middle school Identity Development activities Erete et al. (2016) Design & Participatory STEM Community—other Middle school Identity Development activities Jin et al. (2018) Design & Participatory CS activities Summer program/ High school Knowledge & Skills; Motivation Development camp Lau et al. (2009) Design & Wearable technologies Summer program/ Middle school Knowledge & Skills; Motivation; Development camp Interest Pinkard et al. (2017) Design & Participatory STEM Multiple OST Elementary school Identity Development activities (K–5) Scott and White (2013) Design & Participatory CS activities Multiple OST High school Motivation Development Klopfer et al. (2004) Impact & Participatory STEM Classroom— Middle school; Motivation Effectiveness activities traditional High school Loksa et al. (2016) Impact & Participatory CS activities Summer camp High school Self-efficacy Effectiveness Cognitive Outcomes Bicer et al. (2017) Exploratory Maker spaces & activities Summer program High school Perception of skills & creativity Brooks and Sjöberg Exploratory Games Research Lab Elementary school Interactions with technology, (2019) setting (K–5) design decision-making Bugallo et al. (2015) Exploratory Participatory STEM Summer program High school Engineering knowledge activities Dasgupta et al. (2019) Exploratory Maker spaces & activities Classroom— Middle school Technology & science traditional knowledge; experimental strategies & graph interpretation Prepublication Copy, Uncorrected Proofs A-6 

de Paula et al. (2018) Exploratory Participatory STEM After School Middle school Computational & game activities Program—at a development skills school Denault et al. (2008) Exploratory Participatory STEM Summer program / High school Computer programming activities camp Folk et al. (2015) Exploratory Unplugged STEM Classroom— K–12 CT concepts & terms; activities traditional (across grade bands) recognition of CT Freina et al. (2019) Exploratory Games Classroom— Elementary school Programming & documentation traditional (K–5) skills Gardeli and Vosinakis Exploratory Mobile apps Classroom— Elementary school Problem solving, algorithmic (2019) traditional (K–5) concepts & game design Nugent et al. (2013) Exploratory Participatory STEM After school, on-site Middle school Computer programming, activities engineering & design, GPS & science, math, robotics & workplace skills Nugent et al. (2016) Exploratory Participatory STEM Camps, clubs & Middle school Computer programming, activities competitions engineering & design, GPS & science, math, robotics & workplace skills Nugent et al. (2019) Exploratory Wearable technologies Classrooms— Elementary school Knowledge of circuity & traditional (K–5) programming; engineering design Ouyang et al. (2018) Exploratory Participatory CS Activities Classroom— Elementary school CT Concepts traditional (K–5) Sharma and Ali (2018) Exploratory Participatory STEM Classroom— High school Cartesian coordinate system; Activities traditional applied geometry Weitze (2017) Exploratory Making games / sims Classroom— High school Disciplinary content; game traditional design & making Jenson et al. (2018) Design & Course Classroom— Middle school Computer programming; Development traditional academic achievement Prepublication Copy, Uncorrected Proofs A-7 

Yin et al. (2019) Design & Maker Spaces/Activities Summer academy Middle & High school Physics & engineering, CT Development definition & CT skills Garneli et al. (2015) Impact—Efficacy Games ns Middle school Problem solving, programming & use of CS constructs Behavioral Outcomes Amo et al. (2019)— Exploratory Hands-on cyber security Summer camp 13–17-year-olds Cyber security engagement & study 2 workshops self-efficacy Freudenthal et al. Exploratory Media-propelled CT High school math & High school students Enrollment in CT courses, (2011) college calculus First year college students engagement in problem-solving, class math course enrollment& college grades Gardeli and Vosinakis Exploratory Game: Mobile augmented Primary school Age 9–10 Enjoyment & collaboration (2019) reality classrooms Klein (2013) Exploratory Casual mobile game Play Club 10–15-year-olds Level of emersion & reflection Simpson et al. (2017) Exploratory Computer-supported Summer camp Rising high school Positions of power collaborative learning freshmen Vickery (2014) Exploratory Web design workshop Public library 8–16-year-olds Sustained enjoyment Folk et al. (2015) Exploratory Deliberate instruction in K–12 STEM classes Middle & High school Recognition & definition of CT & algorithm design instructors & students. algorithms; engagement in CT (unplugged) Weston et al. (2019) Exploratory AP and computing courses High school High school students Persistence in computing post high school & technology majors Charlton and Poslad Exploratory Maker events: Shareable Community setting Age 14–15 Engagement in design and (2016) (Descriptive) wearables development & product application Mystakidis et al. (2014) Development Immersive multi-media Library Grades 2–6 Interest in books & positive learning experience mentality toward reading Amo et al. (2019)— Impact & Cyber-networking Museum Middle school General problem solving, study 1 Effectiveness workshop (low-income) science engagement, cyber & (RCT) digital awareness Prepublication Copy, Uncorrected Proofs A-8 

Klopfer et al. (2004) Impact & Palm versus wearable Science classes Study 1: 14–16-year-olds Engagement in collaborative Effectiveness supported participatory Study 2: 11–13-year-olds problem-solving, motivation to (RCT) simulations participate in problem-solving & self-reported learning Wanzer et al. (2020) Impact & Web-based introductory Workshops; Middle school; Intent to persist in computing Effectiveness computing experience summer camps, High school traditional classes NOTE: ns means not specified. For each outcome category, the articles are organized by study type and then alphabetized. Prepublication Copy, Uncorrected Proofs A-9 

Prepublication Copy, Uncorrected Proofs A-10