4

How Do Summer Programs Influence Outcomes for Children and Youth?

Whereas in Chapter 2 we documented the range of activities children and youth participate in during the summer months, and in Chapter 3 we described the effect that this season has on the development of children and youth, in this chapter we describe what is known from the research literature regarding how participation in summer programs affects young people’s outcomes and the factors that enhance program effectiveness (see Box 4-1). The chapter also includes information about international examples of summer programs.

We note that this chapter is not inclusive of all the activities that children and youth participate in over the summer months, for example unstructured activities and summer programs that have not been rigorously studied. As a result, it does not capture all the benefits that children and youth can derive from positive summer activities, such as the opportunities to explore interests, practice new skills, and practice independence and self-regulation during free time. Instead, this chapter focuses on the effectiveness of structured programs that intend to build certain skills and competencies for children and youth.

CLASSIFICATION OF PROGRAM EFFECTIVENESS EVIDENCE

The earlier chapters drew on a range of research to describe what children and youth do during the summer and how summer affects their development. Much of that discussion is descriptive, but as we turn to examining program effectiveness it is important to determine how confident we are that outcomes measured by the program are the result of the program itself rather than other factors, such as the passage of time or pre-existing differences between participants and those who did not participate. In deciding how to classify research evidence of program effectiveness, the committee was guided by a desire to be helpful to policy makers and practitioners looking to adopt evidence-based practices and to use a scheme that would fit the available evidence. The committee considered a number of classification and rating schemes. We found the U.S. Preventive Services Task Force framework particularly appealing, as it encompasses two important

dimensions: certainty of benefit (with high, moderate, and low levels of certainty) and magnitude of net impact (substantial, moderate, small, zero, and negative). However, due to the variety of outcomes studied, limitations of the available evidence base, and variability regarding what constitutes a small or large effect size depending on measure and age level,¹ the committee concluded that we could not accurately classify the magnitude of

___________________

¹ The committee did not feel we could rely on effect sizes to distinguish the magnitude of benefits. We know that the magnitude of an effect size is influenced by a number of factors, including the type of assessment used, grade level and subject, and type of study conducted. For instance, annual spring-to-spring gains on broad standardized assessments vary by subject and grade level, from as large as 1.52 in reading between spring of kindergarten and spring of first grade, to as small as 0.01 in mathematics from spring of 11th grade to spring of 12th grade (Lipsey et al., 2012). Further, for many measures in the studies we identified, we have no way to benchmark effect sizes.

the benefit for all outcomes beyond three categories: positive, zero, and negative.

We also considered the evidence framework used in the Every Student Succeeds Act (ESSA), which outlines four tiers of evidence depending on the certainty that a program will create positive child or youth outcomes based on the strength of the research design. The lowest level of ESSA evidence is based on a program having a theory of action, developed in part from the research base, which is inappropriate for the committee’s focus on outcomes. Also, the ESSA framework does not differentiate between well-conducted randomized controlled trials (RCTs) conducted with small numbers of children or youth or at only one site, on the one hand, and studies that had weaker comparison groups, on the other—a distinction the committee felt was important, particularly since some of the interventions studied are not intended to be offered at large scale. Further, given the lack of experimental literature, the committee wanted to include pre/post study designs and to discuss important information about child, youth, and family experiences to provide guidance regarding implementation.

After considering multiple options, the committee determined we would classify quantitative outcomes evidence into three categories that signal our certainty that the program created the measured effect and would do so in other contexts:

1. Conclusive evidence. There is strong certainty that the program created positive outcomes for many children and youth that could be replicated in other settings. This evidence comes from a well-conducted rigorous experimental design to identify for causal inference—an RCT that has been conducted with many children or youth (more than 350) across more than one site. In an RCT, participants are randomly assigned to either receive the intervention or participate in a comparison group that does not receive the intervention, providing a robust counterfactual.
2. Moderate evidence. There is reasonable certainty that the program created positive outcomes for children and youth. Evidence in this category comes mainly from studies with well-conducted quasi-experimental designs. These designs approximate experimental research by identifying a comparison group that is similar to the program participants’ observed pre-intervention characteristics (e.g., test scores, grade, race, gender). Alternatively, the evidence may come from RCTs where the generalizability of the findings are uncertain due to having a small number of children and youth in the study or due to being conducted at only one site.
3. Suggestive evidence. There is evidence of a relationship between program participation and youth outcomes; however, certainty

In a few cases, we provide qualitative research evidence to document the perspective of children, youth, and families about their program experiences.

The RAND Corporation recently published an ESSA evidence review of summer programming (McCombs et al., 2019). The committee drew heavily on that review to identify the applicable outcomes literature, although our classification of evidence differs from the one used in that review. In addition, committee members identified additional pre/post studies and qualitative literature, including systematic reviews, to include. Because we did not conduct a comprehensive literature review of the pre/post studies, there are studies with suggestive findings that are not cited below. We also note that while the pre/post studies cited examine how participants changed over time, they do not include a comparison group (to observe those that did not participate in the program), and this makes it very difficult to draw conclusions about cause and effect.

EVIDENCE FOR THE EFFECTIVENESS OF SUMMER PROGRAMS

In this section, we detail the evidence for the effectiveness of summer programs by the four categories of outcomes the committee was tasked to investigate: safety, risk-taking, and pro- and anti-social behavior; physical and mental health; social and emotional development; and academic learning and enrichment. Within each of these outcome domains, we discuss the types of programs that show evidence of effectiveness and provide information on the agents that typically offer this type of program.

There is evidence of effectiveness for many different types of programs, but the outcomes research base is not representative of all the types of summer programming available and it does not represent the full range of populations served or all types of agents offering programs. For instance, the majority of identified outcomes studies examined academic summer programs targeted to youth who are low-income or performing below grade level. Even within the academic programs, we found far more evaluations of reading than of mathematics, writing, or science. Other types of popular programs, including sports camps and religious camps, have not been rigorously studied. Also, the literature does not investigate the benefits of

the most common pattern of participation in summer programs, namely, participating in multiple, short programs over multiple summers.

Few summer programs have been studied through trials replicating them in different contexts. Many of the studies we identified are one-off program evaluations, and many contain little information regarding implementation, making exact replication difficult. Further, many of the RCTs include small numbers of children and youth because the programs themselves are intentionally small, while many studies of programs that do include larger populations are not randomized trials. As a result, we found few studies demonstrating conclusive evidence of benefits. Instead, we found more moderate and suggestive evidence.

At the same time, we found no evidence of significant negative outcomes from the reviewed studies. This may stem either from the low probability of harm coming from summer programming or from a bias in publication toward positive findings. It is important to note that while the majority of studies found evidence of effectiveness, studies did not typically find that programs were effective in producing all measured outcomes. Across the studies we identified, about a third of the outcomes measured were positive and statistically significant.

Safety, Risk-Taking, and Pro- and Anti-Social Behaviors

Safety and Supervision

Although no rigorous outcomes studies examined whether summer programs improve physical safety or supervision, the committee believes such effects are both important and likely benefits of many types of summer programs. This is partly because in the afterschool literature conclusive evidence does exist for effects on supervision. RCT of afterschool programs find that children and youth assigned to participate in these programs were more likely to be supervised by adults, and elementary school participants reported feeling safer compared with youth not assigned to these programs. Increased adult supervision also meant that children were less likely to be cared for by older siblings and had less time for unsupervised activities with peers (Dynarski et al., 2003, 2004; Gottfredson et al., 2010; James-Burdumy et al., 2005). Appropriate adult supervision promotes personal safety and reduces participation in risky behaviors (Gottfredson et al., 2010). It is reasonable to think these same benefits accrue to participants in summer programs as well.

Risky Behaviors

Summer can be used as a time to provide programs and interventions intended to reduce risky behaviors, such as alcohol use and unsafe sex. There is moderate evidence that programs can achieve these outcomes (see Table 4-1). For instance, a summer program designed by Johns Hopkins University professors, in collaboration with local community partners, which included a targeted intervention to increase safe sex practices among youth, yielded moderate evidence that it improved youth’s confidence in condom use and their ability to respond safely to potentially risky sexual situations (Tingey et al., 2015). Similarly, a study of an intervention program intended to reduce risky behaviors among girls in foster care transitioning to middle school found moderate evidence that the program reduced substance use and delinquency the next school year (Kim and Leve, 2011).

Crime

As described in Chapter 2, many city governments offer youth employment programs. One of the goals of this type of program is to reduce the probability that youth will be the victims or perpetrators of crime during the summer, as these risks increase during the summer months. An RCT of Chicago’s youth employment programs for adolescents at risk of crime involvement found moderate evidence that the programs decreased violent crime arrests, though they had no effect on school enrollment, property crime arrests, or drug crime arrests (Heller, 2013).

Physical and Mental Health

Physical Health

Summer programs can be designed to improve the physical health and well-being of children and youth, and we have suggestive evidence they can achieve this goal. One study found that overweight children and youth in a residential physical health summer program who attended for at least 29 days had short-term weight loss, improved blood pressure, and increased aerobic fitness (Gately et al., 2005). Another pilot study of a 7-week summer program for migrant children that included a healthy eating and exercise curriculum, found that BMI improved for the participants, with greater benefits accruing to children who attended regularly (Kilanowski and Gordon, 2015).

There is suggestive evidence that structured programs, such as summer school, may prevent weight gain and help maintain physical fitness over the summer due to the structured environment itself, with its restricted access to unhealthy food and time set aside for exercise (Park and Lee, 2015). There is also descriptive evidence that both day camps and overnight camps establish conditions for physical activity that would support physical health. For instance, Hickerson and Henderson (2014) measured the number of steps youth took during a typical day camp experience and a typical overnight camp experience; they found that the number of steps nearly met or exceeded the guidelines for youth physical activity set by the U.S. Department of Health and Human Services. Similarly, Brazendale and colleagues (2017) found that the majority (70% to 80%) of day camp attendees were meeting the recommended levels of moderate to vigorous physical activity during their camp time.

As noted earlier, many communities offer free summer meal programs, because summer is a time of greater food insecurity for low-income children and youth who rely on free and reduced-price lunches during the school year. While there are no studies examining the effectiveness of such programs in promoting physical health, a recent study indicates that there may be an unmet need for these programs. Only 15 percent (3 million students) of the 20 million children who receive free and reduced-price lunches during the school year consistently received free summer lunches during the summer (Anderson et al., 2018).

Mental Health

The committee found no evidence that summer detrimentally effects the mental health of children and youth. Indeed, summer is a time when children and youth with mental health needs can participate in targeted programs that provide them with support.

We found moderate evidence (see Table 4-2) that these targeted programs can be effective in achieving some improved mental health outcomes. The majority of the studies providing this evidence are RCT of small populations of children and youth; at the same time, these programs are not intended to be offered at scale to large numbers of children at one time.

For example, a week-long program for girls with separation anxiety disorder—which provided cognitive behavioral therapy, repeated practice in separating from parents and peers, and opportunities for social interaction—produced moderate evidence of decreasing separation anxiety and increasing global functioning (Santucci and Ehrenreich-May, 2003). A 4-week

program providing self-advocacy training among students with learning disabilities reduced depression among participants (Stevens, 2005).

Social and Emotional Development

Summer is a time all agents can use to advance various aspects of children’s and youth’s social and emotional skills. We found moderate evidence (see Table 4-3) that programs designed to meet the specialized needs of specific groups of children and youth produced social and emotional benefits.

For instance, two studies examined programs that focused on increasing self-advocacy among students with disabilities. The shorter of these, a 1-week program, was found to improve youth self-advocacy skills (Grenwelge and Zhang, 2012), and the longer of these, a 4-week program, improved participants’ attributional style (i.e., how one explains the cause of events) and self-esteem (Stevens, 2005). Similarly, we find moderate evidence from a 3-week Canadian therapeutic summer program for children and youth with learning disabilities coupled with social, emotional, or behavioral issues, namely that the program reduced social isolation and improved self-esteem into the next school year (Michalski et al., 2003).

Programs designed for children and youth with social difficulties show moderate evidence of effectiveness in improving social skills (Hektner et al., 2017) and the ability to seek friendship help (Foley-Nicpon et al., 2017). All the specialized programs that were rigorously studied demonstrated moderate or suggestive evidence of effectiveness. It may be that the efficacy of these programs is tied to their intense targeting of populations identified as being in need of such programs. However, as is the case with other types of programs, none of the programs that measured multiple outcomes was effective in producing all the outcomes measured.

One study of a voluntary pre-kindergarten (pre-K) academic program for low-income children found moderate evidence that participation in the program aided students with social aspects of their transition to school (Berlin et al., 2011). Another study, which examined the effects of residential camp for overweight children and youth, found suggestive evidence that in addition to reducing participants’ weight it resulted in improving their self-esteem (Gately et al., 2005).

There is substantial suggestive evidence that medical camps, such as camps for cancer patients and children with diabetes, can provide social and emotional benefits to participants. For instance, in research reviews of studies of summer camps for cancer patients, authors identify suggestive evidence that the camps improved participants’ self-concept, friendship, empathy, quality of life, and emotional well-being (Martiniuk et al., 2014; Packman et al., 2005). For instance, children at a camp for cancer patients described gaining more confidence in social settings and increased engagement with peers:

TABLE 4-3 Social and Emotional Development: Evidence for Summer Program Effectiveness

In a literature review of camps for burn victims, lasting 2 days to 1 week, authors find qualitative evidence suggesting that burn camp can decrease camper isolation, improve self-esteem, and promote coping and social skills, though the quantitative pre/post studies had mixed findings. (Maslow and Lobato, 2010).

Although they are very popular, summer camps (both day and overnight) are typically not formally evaluated. The American Camp Association (2017) finds that 93 percent of overnight camps target the development of youth social skills, but only 25 percent of them measure social skill outcomes. Pre/post studies suggest that camps offered to all children and youth can be a context for positive, personal development for youth (Garst et al., 2011), especially when programming is intentionally designed to do so (Bialeschki et al., 2007). Participation in camps has led to improved social and emotional competencies, including teamwork and public speaking (Povilaitis and Tamminen, 2018), leadership development (Garton et al., 2007; Thurber et al., 2007), reading self-efficacy (Garst and Ozier, 2015), independence (Allen et al., 2006), and values (Collado et al., 2013).

Academic Learning

Academic learning was the most rigorously studied area of outcomes in the literature we identified. We find evidence of effectiveness for several types of summer academic learning programs, including voluntary programs, mandatory programs, and at-home programs (see Table 4-4). Many of these types of programs have been examined by multiple studies. When this is the case, Table 4-4 reports the highest level of evidence provided by the studies and also indicates, with an asterisk, where there were mixed results from different studies (some having positive findings and others null findings). We note that not all types of academic learning programs have been rigorously studied. For instance, we found no studies of extended school-year programs, likely because school districts are mandated to provide these services to meet the educational needs of students with special needs. Most of the rigorously studied academic learning programs were programs that targeted children and youth from low-income families in an effort to address differential summer learning loss or targeted students performing below grade level to provide academic remediation. We also find evidence of spill-over academic effects in an employment program and an arts program designed for youth in correctional education. Below we

TABLE 4-4 Academic Learning: Evidence for Program Effectiveness

^aAn asterisk is applied wherever the table reports the highest level of evidence provided by the studies and also indicates that there were mixed results from different studies (some having positive findings and others null findings).

describe the research evidence for academic learning outcomes by type of summer program.

Mandatory Academic Programs

Almost all school districts provide summer programs for high school students who have failed to pass a course, and some require elementary or middle school students who are performing far below grade level and are at risk of being retained in grade to attend summer programming. For these programs, participation and successful completion are mandatory to allow these students to move on to the next grade or to the next course. Studies have found moderate certainty of benefits achieved from mandatory summer school programs that provide reading and math instruction to elementary school students, specifically in improving reading (Matsudaira, 2008) and in mathematics achievement (Mariano and Martorell, 2013; Matusdaira, 2008).

While credit recovery programs are prevalent, we identified only two rigorous evaluations of these programs. One study found suggestive evidence that a high school credit recovery program for English learners increased the number of English language arts courses taken by 12th grade, but it found no evidence that the program improved test scores in English language arts, raised the number of math or science courses taken, or improved on-time graduation rates (Johnson, 2017). Another study tested the efficacy of an online Algebra I course as compared with an in-person Algebra I course. This last study found moderate evidence of the effectiveness of in-person courses relative to online courses, as students taking in-person courses had higher algebra assessment scores, grades, and credit recovery rates than online students. However, the mode of the course did not differentially affect longer-term math performance (Heppen et al., 2016).

Voluntary Academic Programs

Some schools, districts, universities, and community partners offer voluntary academic summer programming to children and youth with the intent of improving students’ success in school, most often in reading and mathematics. These programs usually target students performing below grade level or low-income students considered at greater risk of academic loss during the summer months.

The content and structure of studied programs vary from one another in the following ways:

• By grade level: Some programs targeted certain grade levels (e.g., early elementary, middle grades), while others spanned multiple grade levels (e.g., grades 1–7).

• By duration: Some programs were offered for half-days, although most studied programs ran for a full day. Programs ran anywhere from 3 weeks to 6 weeks over the summer.
• By content: Some programs focused on one academic subject, others spanned multiple subjects, and many (but not all) included nonacademic enrichment activities (e.g., arts or sports).
• By curriculum: Programs tended to have a set curriculum; however, each program offered a different curriculum, and some studies did not describe the curriculum that was used.

Given the variability in programs, study design, and measures, it is unsurprising that in this category some programs demonstrated academic benefits and others did not. We interpret the combined evidence from these studies to suggest that voluntary summer learning programs have the potential to benefit children and youth but are not guaranteed to do so.

There is suggestive evidence that a voluntary pre-K program improved participants’ early literacy skills (Edmonds et al., 2009). However, another study of this type of program found no benefits for early literacy or mathematics (Story, 2008).

Studies of voluntary reading programs targeted to students performing below grade level or low-income students provide moderate evidence of benefits in reading achievement (Borman et al., 2009; Schacter and Jo, 2005; Zvoch and Stevens, 2013), while other studied programs provide suggestive evidence of benefits (Cleary, 2002; Luftig, 2003; Waters, 2004). However, a number of studies of reading programs found no benefits for participating children and youth (Dynia et al., 2015; Story, 2008). None of the reading programs that were shorter in duration (e.g., half-day, 3-week programs) was found to be effective.

We found both moderate and suggestive evidence of effective voluntary mathematics programs for middle and high school students (Bowens and Warren, 2016; Stewart, 2017). Additionally, a study of a voluntary STEM program provides suggestive evidence that the middle school youth who participated had stronger attendance in the following school year in one cohort, although those findings were not replicated in other cohorts (Mac Iver and Mac Iver, 2015).

We find evidence that multi-subject voluntary programs can benefit children and youth. There is suggestive evidence that such programs yield benefits in reading and/or mathematics achievement (Betts et al., 2005; Chaplin and Capizzano, 2006; Concentric Research and Evaluation, 2018), although some studies found no evidence of effectiveness (Bakle, 2010). A longitudinal, multidistrict RCT was conducted to evaluate 5-week voluntary learning programs that included academic instruction and enrichment activities. It found conclusive evidence of short-term benefits in mathemat-

ics when children returned to school in the fall, although the initial gains the treatment group experienced in the fall did not persist at the same levels in the spring. The study also provides suggestive evidence that highlights the importance of strong attendance: among students with high attendance, mathematics gains seen in the fall were also found in the spring, when state assessments were done, and among students who attended a second summer those with high attendance outperformed control group students in mathematics and reading in the fall and spring. The quality of instruction (focused on clear instruction, on-task behavior, and teachers’ ensuring that all students understood the material) was also correlated with language arts achievement (Augustine et al., 2016).

At-Home Programs

At-home learning programs are compelling options for policy makers and funders because they tend to be lower in cost than in-person programs for helping students gain or maintain academic skills over the summer. We found conclusive and moderate evidence of benefits from this type of program. Only one program focused on mathematics, a 9-week program for middle school students that thematically linked to the Boston Red Sox baseball team and covered key mathematical concepts from the prior school year. This program produced suggestive evidence of benefits in mathematics achievement (Nelson, 2014).

The other type of at-home learning programs studied were reading programs offered to elementary school students. These programs, which provide children with high-interest books at their own reading level, have been provided both through book fairs by sending books directly home to families and through book mobiles that visit rural communities. We have conclusive and moderate evidence of benefits from this type or program, although an equal number of studies of at-home reading programs show no evidence of effectiveness. Effective at-home reading programs for elementary students often occurred over multiple summers (Allington et al., 2010; Stein, 2017) and/or were scaffolded by teachers prior to or during the summer program (Kim and White, 2008; Kim et al., 2016; Melosh, 2003). For instance, in one program students participated in a school-year book fair and voluntary summer reading for three summers before the program showed an effect on their state reading scores (Allington et al., 2010). For the Project Reads program, there is evidence of effectiveness for the version of the program in which scaffolded instruction was provided by teachers prior to the summer (Kim and White, 2008; Kim et al., 2016); programs without this scaffolding have not been associated with positive outcomes (Kim, 2006, 2007).

Non-Academic Programs

In addition to providing benefits to youth safety, we find conclusive evidence that New York City’s summer youth employment program, which targeted low-income youth, resulted in improved youth engagement with school and participation in and performance on academic assessments (Leos-Urbel et al., 2003; Schwartz et al., 2014). There is also suggestive evidence that youth in correctional education who participated in a 3-week summer theater program had higher overall GPAs than nonparticipants (Coronado, 2000)

RESEARCH-BASED BEST PRACTICES

In sum, the research base demonstrates that summer programming can measurably benefit youth across multiple domains, although it is not guaranteed to do so. Because the overall evidence base is fragmented and not representative of all programs available to children and youth over the summer, it is particularly important to understand what factors increase the quality and effectiveness of summer programs. In the following, we draw out key themes that emerge from the literature we reviewed and the testimony heard by the committee.

Intentional Design to Meet Student Needs and Desired Outcomes

As discussed earlier, programs designed to meet specific students’ needs and that link their content to desired student outcomes appear to be particularly successful. For instance, programs designed to address the social and emotional needs of children and youth with disabilities demonstrated moderate or suggestive evidence of effectiveness. This suggests that the efficacy of these programs might be tied to the intentional targeting of the program to a population that has been identified as in need of such a program (McCombs et al., 2019). Similarly, meta-analyses of out-of-school time programs have found that programs designed to enhance students’ social and emotional skills were successful in doing so, while programs that were not intentionally addressing these skills tended to be ineffective in improving these skills (Durlak et al., 2011).

Based on testimony provided to the committee, we identified cultural responsiveness as a key component of intentional programming. Programs that are not responsive to students’ cultural values, beliefs, and backgrounds are, at a minimum, unlikely to attract and retain youth, and at worst could do harm by inflicting offensive beliefs or actions on students.

In the context of summer, and concerning out-of-school time programs broadly, cultural responsiveness requires considering both the staff practices

that directly influence youth experiences (e.g., behavior management strategies, expectations for youth, and interaction styles) and the organizational structures (e.g., program communications, written rules and policies) that both directly and indirectly influence youth access to and experiences in programs (Simpkins et al., 2017). For example, program structures should connect to content that is relevant to youth’s lives in culturally meaningful ways, and in their interactions with youth staff should support opportunities to explore youth’s cultural identity. These approaches are relevant for summer experiences as well.

Sustained Attendance

Research also clearly demonstrates that students need to attend summer learning programs to benefit from them (Augustine et al., 2016; Borman et al., 2005; Borman and Dowling, 2006; Kilanowski and Gordon, 2015; McCombs et al., 2009). For instance, in a study of an academic voluntary summer learning programs, the authors found that students who attended the summer 2013 program for at least 20 days benefited in mathematics in the fall of 2013 relative to comparison-group students, and those effects persisted through spring 2014. After the summer 2014 program, high attenders outperformed the comparison group in both mathematics and language arts in the fall and spring. These benefits were also demonstrated on state academic assessments in spring (Augustine et al., 2016).

Sufficient Duration

Summer programs should be of sufficient duration to meet the goals established for the program. Duration seems particularly important for academic programs. None of the programs we reviewed that lasted 3 weeks or less resulted in benefits for children and youth. For voluntary academic programs, recent research suggests programs should last at least 5 weeks so that they can provide sufficient content to demonstrably improve student achievement (Augustine et al., 2016). There are also indications that greater benefits accrue across multiple summers of participation (Augustine et al., 2016); for example, a study of Horizons National, a multisummer program, found academic benefits that accrued after multiple summers of participation (Concentric Research and Evaluation, 2018).

INTERNATIONAL EVIDENCE

The committee was also charged with understanding whether there are lessons and examples from other nations that may have implications for the United States. To respond to this charge, the committee commissioned a

paper on international summertime experiences. A review of the published literature highlighted international examples of summertime programming and offered some potentially promising practices and lessons for the United States (see Box 4-2) (Pulizzi, 2019).

Several limitations were noted, however, in the review of this global summertime literature, limitations that also apply to findings from the United States. The international programs that were evaluated were often stand-alone activities and lacked follow-on support (Davies et al., forthcoming). The evaluation time frames typically were short and did not

extend beyond a single summer, resulting in uncertainty about longer-term outcomes and sustained changes. In addition, the measurement of international program outcomes was frequently pre/post intervention after a single summer experience, and program evaluation designs were generally not structured to account for other factors that could affect the measured outcomes, reducing confidence in causal attributions.

CONCLUSIONS

Based on our review of the outcomes and the best-practice literature, the committee reached a number of conclusions that have implications for policy and practice.

REFERENCES

Concentric Research and Evaluation. (2018). Examining the Long-Term Effects of the Horizons National Student Enrichment Program on Student Academic Outcomes. Available: http://www.horizonskids.org/Customer-Content/horizonsnewcanaan/news/PDFs/Horizons_National_Retrospective_Study_Report_November_2018.pdf

Coronado, A. A. (2000). The Effects of a Summer Performing Arts Program on At-Risk Adolescents. Dissertation, University of Southern California. University Microfilms International (UMI) No. 9955025.

Davies, S., Aurini, J., and Milne, E. (forthcoming). Summer learning: Classic, new and future research. Chapter for Section IV, Case Studies of Extended Education. In S. Hoon Bae, J. L. Mahoney, S. Maschke, and L. Stecher (Eds.), International Developments in Research on Extended Education: Perspectives on Extracurricular Activities, After-School Programs, and All-Day Schools. Opladen, Berlin, and Toronto: Barbara Budrich.

Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., and Schellinger, K. B. (2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82(1), 405–432.

Dynarski, M., Moore, M., Mullens, J., Gleason, P., James-Burdumy, S., Rosenberg, L., Masnfield, W., Heaviside, S., Levy, D., Pistorino, C. and Silva, T., 2003. When Schools Stay Open Late: The National Evaluation of the 21st-Century Community Learning Centers Program. First Year Findings. Washington, DC: U.S. Department of Education.

Dynarski, M., James-Burdumy, S., Moore, M., Rosenberg, L., Deke, J., and Mansfield, W. (2004). When Schools Stay Open Late: The National Evaluation of the 21st Century Community Learning Centers Program—New Findings. Washington, DC: U.S. Department of Education.

Dynia, J. M., Piasta, S. B., Justice, L. M., and Columbus Metropolitan Library. (2015). Impact of library-based summer reading clubs on primary-grade children’s literacy activities and achievement. The Library Quarterly, 85(4), 386–405.

Edmonds, E., O’Donoghue, C., Spano, S. and Algozzine, R. F., 2009. Learning when school is out. The Journal of Educational Research, 102(3), 213–222.

Foley-Nicpon, M., Assouline, S. G., Kivlighan, D. M., Fosenburg, S., Cederberg, C. and Nanji, M. (2017). The effects of a social and talent development intervention for high ability youth with social skill difficulties. High Ability Studies, 28(1), 73–92.

Garton, M. S., Miltenberger, M., and Pruett, B. (2007). Does 4-H Camp influence life skill and leadership development? Journal of Extension, 45(4), Article 4FEA4. Available: http://www.joe.org/joe/2007august/a4.php.

Garst, B. A., and Ozier, L. W. (2015). Enhancing youth outcomes and organizational practices through a camp-based reading program. Journal of Experiential Education, 38(4), 324–338.

Garst, B. A., Browne, L. P., and Bialeschki, M. D. (2011). Youth development and the camp experience. New Directions for Youth Development, 2011(130), 73–87.

Gately, P. J., Cooke, C. B., Barth, J. H., Bewick, B. M., Radley, D. and Hill, A. J. (2005). Children’s residential weight-loss programs can work: A prospective cohort study of short-term outcomes for overweight and obese children. Pediatrics, 116(1), 73–77.

Gillard, A., and Watts, C. E. (2013). Program features and developmental experiences at a camp for youth with cancer. Children and Youth Services Review, 35(5), 890–898.

Gottfredson, D., Cross, A. B., Wilson, D., Rorie, M. and Connell, N. (2010). Effects of participation in after-school programs for middle school students: A randomized trial. Journal of Research on Educational Effectiveness, 3(3), 282–313.

Grenwelge, C., and Zhang, D. (2012). The effects of the Texas Youth Leadership Forum summer training on the self-advocacy abilities of high school students with disabilities. Journal of Disability Policy Studies, 24(3), 158–169.

Hektner, J. M., Brennan, A. L., and August, G. J. (2017). Incorporating well-adjusted peers in a conduct problems prevention program: Evaluation of acceptability, fidelity, and safety of implementation. School Mental Health, 9(1), 66–77.

Heller, S. B. (2013). Experimental Evidence on Improving the Human Capital of Disadvantaged Youth. Dissertation, The University of Chicago. University Microfilms International (UMI) No. 3568385.

Heppen, J. B., Sorensen, N., Allensworth, E., Walters, K., Rickles, J., Taylor, S. S., and Michelman, V. (2017). The struggle to pass algebra: Online vs. face-to-face credit recovery for at-risk urban students. Journal of Research on Educational Effectiveness, 10(2), 272–296.

Hickerson, B., and Henderson, K. A. (2014). Opportunities for youth physical activity promotion: An examination of youth summer camps. Journal of Physical Activity and Health, 11(1), 199–205. doi:10.1123/jpah.2011-0263.

James-Burdumy, S., Dynarski, M., Moore, M., Deke, J., Mansfield, W., Pistorino, C. and Warner, E. (2005). When Schools Stay Open Late: The National Evaluation of the 21st Century Community Learning Centers Program. Final Report. Washington, DC: U.S. Department of Education.

Johnson, A. (2017) Causal Impact of Summer Credit Recovery on High School English Language Learner Outcomes. Stanford, CA: Stanford Center for Education Policy Analysis.

Kilanowski, J. F., and Gordon, N. H. (2015). Making a difference in migrant summer school: Testing a healthy weight intervention. Public Health Nursing, 32(5), 421–429.

Kim, H. K., and Leve, L. D. (2011). Substance use and delinquency among middle school girls in foster care: A three-year follow-up of a randomized controlled trial. Journal of Consulting and Clinical Psychology, 79(6), 740.

Kim, J. S. (2006). Effects of a voluntary summer reading intervention on reading achievement: Results from a randomized field trial. Educational Evaluation and Policy Analysis, 28(4), 335–355.

_____. (2007). The effects of a voluntary summer reading intervention on reading activities and reading achievement. Journal of Educational Psychology, 99(3), 505.

Kim, J. S., and White, T. G. (2008). Scaffolding voluntary summer reading for children in grades 3 to 5: An experimental study. Scientific Studies of Reading, 12(1), 1–23.

Kim, J. S., Guryan, J., White, T. G., Quinn, D. M., Capotosto, L., and Kingston, H. C. (2016). Delayed effects of a low-cost and large-scale summer reading intervention on elementary school children’s reading comprehension. Journal of Research on Educational Effectiveness, 9(supp. 1), 1–22.

Leos-Urbel, J., Schwartz, A. E., Weinstein, M., and Weitzman, B. (2003). More Than a Paycheck? The Impact of Summer Youth Employment on Students’ Educational Engagement and Success. Policy Brief no. 02-12. New York: New York University, Institute for Education and Social Policy.

Lipsey, M. W., Puzio, K., Yun, C., Hebert, M. A., Steinka-Fry, K., Cole, M. W., Roberts, M., Anthony, K. S., and Busick, M. D. (2012). Translating the Statistical Representation of the Effects of Education Interventions into More Readily Interpretable Forms. NCSER 2013-3000. Washington, DC: National Center for Special Education Research, Institute of Education Sciences, U.S. Department of Education.

Luftig, R. L. (2003). When a little bit means a lot: The effects of a short-term reading program on economically disadvantaged elementary schoolers. Literacy Research and Instruction, 42(4), 1–13.

Mac Iver, M. A., and Mac Iver, D. J. (2015). The Baltimore City Schools Middle School STEM Summer Program with VEX Robotics. Baltimore Education Research Consortium.

Mariano, L. T., and Martorell, P. (2013). The academic effects of summer instruction and retention in New York City. Educational Evaluation and Policy Analysis, 35(1), 96–117.

Martiniuk, A., Silva, M., Amylon, M., and Barr, R. (2014). Camp programs for children with cancer and their families: Review of research progress over the past decade. Pediatric Blood & Cancer, 61(5), 778–787.

Maslow, G. R., and Lobato, D. (2010). Summer camps for children with burn injuries: A literature review. Journal of Burn Care & Research, 31(5), September-October, 740–749. doi: 10.1097/BCR.0b013e3181eebec4.

Matsudaira, J. D. (2008). Mandatory summer school and student achievement. Journal of Econometrics, 142(2), 829–850.

McCombs, J. S., Kirby, S. N., and Mariano, L. T. (eds.). (2009). Ending Social Promotion Without Leaving Children Behind: The Case of New York City. RAND Corporation.

McCombs, J. S., Augustine, C., Unlu, F., Ziol-Guest, K., Naftel, S., Gomez, C. Marsh, T., Akinniranye, G., Todd, I. (2019). Investing in Successful Summer Programs: A Review of Evidence Under the Every Student Succeeds Act. RAND Corporation.

Melosh, G. M. (2003). Summer Books!—Stemming Reading Loss in High-Poverty Second Graders Through Access to Appropriate Books During the Summer Vacation Break. Dissertation, University of Florida.

Michalski, J. H., Mishna, F., Worthington, C., and Cummings, R. (2003). A multi-method impact evaluation of a therapeutic summer camp program. Child and Adolescent Social Work Journal, 20(1), 53–76.

Nelson, B. W. (2014). A Summative Evaluation of a Middle School Summer Math Program. Dissertation, Walden University. University Microfilms International (UMI) No. 3614119.

Packman, W., Greenhalgh, J., Chesterman, B., Shaffer, T., Fine, J., Van Zutphen, K., Golan, R., and Amylon, M. D. (2005). Siblings of pediatric cancer patients: The quantitative and qualitative nature of quality of life. Journal of Psychosocial Oncology, 23(1), 87–108.

Park, K. S., and Lee, M. G. (2015). Effects of summer school participation and psychosocial outcomes on changes in body composition and physical fitness during summer break. Journal of Exercise Nutrition and Biochemistry, 19, 81–90. doi: 10.5717/jenb.2015.15052005.

Povilaitis, V., and Tamminen, K. A. (2018). Delivering positive youth development at a residential summer sport camp. Journal of Adolescent Research, 33(4), 470–495.

Pulizzi, S. (2019). Paper Commissioned by the Committee on Summertime Experiences, and Child and Adolescent Education, Health and Safety. Washington, DC: National Academies of Sciences, Engineering, and Medicine.

Santucci, L. C., and Ehrenreich-May, J. (2013). A randomized controlled trial of the Child Anxiety Multi-Day Program (CAMP) for separation anxiety disorder. Child Psychiatry & Human Development, 44(3), 439–451.

Schacter, J., and Jo, B. (2005). Learning when school is not in session: A reading summer day-camp intervention to improve the achievement of exiting First-Grade students who are economically disadvantaged. Journal of Research in Reading, 28(2), 158–169.

Schwartz, A. E., Leos-Urbel, J., Silander, M., and Wiswall, M. Making Summer Matter: The Impact of Youth Employment on Academic Performance. Working Paper 03-14. New York: New York University, Institute for Education and Social Policy.

Simpkins, S. D., Riggs, N. R., Ngo, B., Vest Ettekal, A., and Okamoto, D. (2017). Designing culturally responsive organized after-school activities. Journal of Adolescent Research, 32(1), 11–36.

Stein, M. L. (2017). Supporting the summer reading of urban youth: An evaluation of the Baltimore SummerREADS Program. Education and Urban Society, 49(1), 29–52.

Stevens, B. E. (2005). Just Do It: The Impact of a Summer School Self-Advocacy Program on Depression, Self-Esteem, and Attributional Style in Learning Disabled Adolescents. The Claremont Graduate University.

Stewart, K. L. (2017). Closing the Summer Learning Gap in the Content Area of Mathematics. Dissertation, Grand Canyon University. ProQuest LLC, ProQuest No. 10275169.

Story, I. N. (2008). Pre-Kindergarten Summer School: An Intervention for Kindergarten Readiness. Dissertation, University of Wyoming. University Microfilms International (UMI) No. 3320743.

Thurber, C. A., Scanlin, M. M., Scheuler, L., and Henderson, K. A. (2007). Youth development outcomes of the camp experience: Evidence for multidimensional growth. Journal of Youth and Adolescence, 36(3), 241–254.

Tingey, L., Mullany, B., Chambers, R., Hastings, R., Lee, A., Parker, A., Barlow, A. and Rompalo, A. (2015). Respecting the circle of life: One year outcomes from a randomized controlled comparison of an HIV risk reduction intervention for American Indian adolescents. AIDS Care, 27(9), 1087–1097.

Zvoch, K., and Stevens, J. J. (2011). Summer school and summer learning: An examination of the short- and longer-term changes in student literacy. Early Education & Development, 22(4), 649–675.

_____. (2013). Summer school effects in a randomized field trial. Early Childhood Research Quarterly, 28(1), 24–32.

This page intentionally left blank.