National Academies Press: OpenBook

Community Programs to Promote Youth Development (2002)

Chapter: 6 Lessons from Experimental Evaluations

« Previous: 5 The Landscape of Community Programs for Youth
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

CHAPTER 6
Lessons from Experimental Evaluations

Findings from nonexperimental and small scale quasi-experimental studies of community programs for youth provided the foundation for the descriptions in Chapter 5 of programs considered to be effective by participants, evaluators, observers, and qualitative researchers. In this chapter we summarize the committee’s review of findings from experimental and large-scale quasi-experimental evaluations of community programs for youth. The committee examined several meta-analyses and review articles that summarize the findings from many different studies of community programs for youth and considered in more detail three specific program evaluations.

This examination had four primary objectives. First, we were interested in whether there is evidence from the most rigorous evaluations available that community programs for youth make a difference in the lives of their participants. Second, we wanted to know whether the programs that had received these types of rigorous evaluations included either the program features outlined in Chapter 4 as potentially important components of community programs for youth or the personal and social assets outlined in Chapter 3 as potentially important targets for these pro-

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

grains. Third, when this information was available, we wanted to know whether the program features hypothesized to be important in Chapter 4 were indeed important in both facilitating positive youth development and preventing the emergence of serious problems. Finally, as a lead-in to the next chapter, we also wanted to assess the quality of these evaluations. We include in this chapter comments on whether the evaluations selected for inclusion in this chapter meet the most rigorous methods of evaluation—described in Chapter 7—that can be used with community programs for youth.

REVIEWS AND META-ANALYSES OF EVALUATIONS

The committee reviewed seven reputable reviews and meta-analyses of prevention and promotion programs for youth from the fields of mental health, violence prevention, teenage pregnancy prevention, and youth development.1 We considered reviews that included both programs for youth with a primary focus on prevention and programs explicitly focused on a youth development framework since, as explained previously, the committee found the distinction between prevention and positive youth development not very clear in practice. Programs with prevention goals, for example, often included components common to programs with youth development goals. Details of the programs included in the meta-analyses reviewed are summarized in Table 6–1.

As noted, the primary objective of reviewing these meta-analyses was to provide our best assessment of the extent to which community programs have beneficial effects for youth. Our secondary objective was to determine the extent to which these programs incorporate the features of positive developmental settings developed in Part I and whether there is evidence of links between these program features and positive youth outcomes. Since none of the programs were designed with these specific assets and features in mind, we often had to make assumptions about likely features from the descriptions of the programs included in the various reviews. Through this review, we also identified the strengths and limitations of these evaluations.

1  

We included reviews and meta-analyses published between 1997 and 2000 that appeared in either published, professionally reviewed journals, or selected government documents. In the latter category, only government documents using rigorous methods of review were included in order to reduce the potential biases associated with reviews conducted by groups with high stakes in positive or negative conclusions.

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

We begin with meta-analyses focused nominally on mental health promotion and mental illness/problem behavior prevention programs because these have the most rigorous evaluations. There are several reasons for this: first, there is a long tradition of both mental health promotion and mental illness/problem behavior prevention work; consequently, these programs have had time to develop and to have comprehensive rigorous evaluations completed. Second, many of these programs are being conducted in school buildings during the official school day. These two characteristics make it much easier to conduct a successful randomized trial design evaluation because the participants are drawn from a “captive” population. Conducting such evaluations in community programs is much more difficult precisely because participation is voluntary. We say more about this later in this chapter and in Chapter 7. We then move onto the meta-analyses and summative reviews of largely out-of-school time community programs.

Mental Health Programs
Durlak and Wells

Durlak and Wells (1997, 1998) conducted two meta-analytical reviews of primary and secondary prevention mental health programs operating prior to 1992 for youth under the age of 19. (Primary prevention programs intervene with normal populations to prevent problems from developing. Secondary prevention programs target individuals already at risk or exhibiting problems). Both meta-analyses included only programs with a control group of some type (e.g., a group that is not participating in the program or a group that is on a waiting list to participate in the program). Randomized designs were used in 61 percent of the primary prevention program evaluations and 71 percent of the secondary prevention program evaluations. The primary prevention review (Durlak and Wells, 1997) included 177 programs; the secondary prevention review (Durlak and Wells, 1998) included 130 evaluations. Finally, most of these programs took place in schools (a likely reason that these studies had such low attrition rates of young people leaving the program before completion and final testing).

In general, the results suggest that preventive mental health programs can be effective across a variety of psychological outcome measures for periods of up to two years following the intervention exposure. They are particularly effective at increasing such personal competencies as

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

TABLE 6–1 Community Programs to Promote Adolescent Development

Program Name/Age, Grade Evaluated

Program Description

Outcomes

Personal and Social Assets

Features of Positive Developmental Settings

Across Agesa,b

Grade: 6th

Location: Family, school, community

Sessions: Mentoring (2 hrs/wk), community service (1 hr every other week), social problems solving (26 1 hr sessions) over school year

Content: mentoring by older adults, classroom-based life skills training, community service activities, workshops for parents

↑ Positive attitudes toward school, the future, older people, knowledge of elders, and community service

↓ School absence

Intellectual

psychological

emotional

social

Emotional and instrumental support; belonging; positive social norms; opportunity for efficacy; opportunities for skill building; integration with family, schools, and community

Adolescent Transitions

Projecta,d

Age: 10–14

Grade: 6th–8th

Location: Family, school, community

Sessions: 12 over 18 hrs

Content: Youth self-regulation skills training (teen focus group), parent management skills training (parent focus group), consultant to improve parent-youth communication (teen/parent focus group)

↑ Social learning

↓ Negative engagement with family, conflict, negative family events, youth aggression (increased school behavior problems for teen focus group at one year follow-up)

Psychological

emotional

social

Emotional and instrumental support; belonging; positive social norms; opportunities for skill building; integration with family, schools, and community

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

Bicultural Competence

Skillsa

Age: 11–12

Location: Community

Sessions: 10

Content: Skills training to promote competence and positive identity based on bicultural fluency

↑ Self-control, assertiveness, healthy coping, substance abuse knowledge

↓ Alcohol, tobacco, and other drug use

Psychological

emotional

physical

Belonging; positive social norms; opportunities for skill building

Big Brothers

Big Sistersa,b,c,d

Age: 10–16

Location: Community

Sessions: 9–12 hrs/mon for one year

Content: Activities with mentor

↑ Grade point average, parental trust

↓ Hitting behavior, likelihood of initiating alcohol and drug use, skipping school, lying to parents

Intellectual

psychological

emotional

social

physical

Emotional and instrumental support; belonging; positive social norms; physical and psychological safety; opportunity for efficacy; opportunities for skill building

Brainpower Program f

Age: 10–12 (indicated)

Location: School

Sessions: 12 lessons (60–90 minutes each)

Content: Social competency: focus on improving the accuracy of children’s perceptions and interpretations of others’ actions

↓ Aggressive behavior following the intervention

Social

Belonging; opportunities for skill building

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

Program Name/Age, Grade Evaluated

Program Description

Outcomes

Personal and Social Assets

Features of Positive Developmental Settings

Bullying Prevention

Programc,d

Grade: 4th–7th Norway; 5th–8th U.S. equivalent (universal)

Location: School

Sessions: 9–12 hrs/mon for one year

Content: 32-page booklet included information on the scope, cause, and effects of school bullying and detailed suggestions for reducing and preventing it. Abbreviated bullying info to families with school-age children. A 25-minute video with vignettes of bullying situations. Students completed a brief questionnaire related to bullying to increase awareness and promote discussion of the problem

↓ 50 percent or more in bully/victim problems for boys and girls across all grades (4–9), with more marked effects after 2 years than after 1 year

(sleeper effects)

Psychological

emotional

social

Positive social norms; physical and psychological safety; opportunities for skill building

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

The Child Development

Projecta

Age: 11–12

Grade: 3rd–6th

Location: Family and school

Sessions: Integrated curriculum over school year

Content: Cooperative learning, reading and language arts, developmental discipline, school community building, home activities

↑ Peer social acceptance

↓ Alcohol and tobacco use, loneliness and social anxiety; marijuana use, carrying weapons, vehicle theft (high implementation subgroup)

Intellectual

psychological

emotional

social

physical

Belonging; positive social norms; opportunities for skill building; integration with family, schools, and community

Coping and Support

Trainingd

Grade: 9th–12th

(indicated)

Location: School

Sessions: 12-sessions

Content: Group life skills training

↑ Self-control, problem-solving abilities, and perceived family support

↓ Suicide risk behaviors and anger problems

Intellectual

psychological

emotional

social

physical

Emotional and instrumental support; belonging; positive social norms; physical and psychological safety

Coping with Stress

Coursed

Grade: 9th–10th

(indicated)

Location: After school

Sessions: 15, 45 min each

Content: Cognitive intervention to encourage adaptive coping (cognitive restructuring, identify and challenge negative or irrational thoughts)

No significant differences at the end of the intervention, but ↓ cases of “major depressive disorder” or dysthymia (milder depressive disorder) at 12 months postintervention .

Intellectual

psychological

emotional

Opportunities for skill building

Counselor-CARE d

Grade: 9th–12th

(indicated)

Location: School

Sessions: 1, 3.5–4 hrs

Content: Brief assessment and resource identification program using computer program and counselor

↑ Self-esteem

↓ Suicide risk behaviors and anger problems, reports of depression

Psychological

emotional

Emotional and instrumental support

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

Program Name/Age, Grade Evaluated

Program Description

Outcomes

Personal and Social Assets

Features of Positive Developmental Settings

Creating Lasting

Connectionsa,b

Age: 12–14

Location: Family, church, community

Sessions: Youth (15 hrs); parents (55 hrs); volunteer service (18 hrs); follow-up and consultation support (1 yr)

Content: Church community mobilization, parent and youth strategies to promote communication and self-management skills, follow-up case management service

↑ Youth use of community services, related action tendencies, perceived helpfulness

↓ Onset of substance abuse delayed as parents changed their substance use beliefs and knowledge

Psychological

emotional

social

physical

Emotional and instrumental support; belonging; positive social norms; opportunities for skill building; integration with family, schools, and community

ENABLe

Age: 12–13

Grade: 7th–8th

Location: School classroom, community organization

Sessions: 5, 1 hr each

Content: Social influence theory; help youth understand social and peer pressures to have sex, develop and apply resistance skills, emphasis on postponing sexual involvement.

↑ Pregnancies in teen-led groups, otherwise no effects ↓

Psychological

emotional

social

physical

Positive social norms; opportunities for skill building; integration with family, schools, and community

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

Friendly PEERsuasionb

Age: 11–14

(indicated)

Location: Girls, Inc.

Sessions: 14, 1 hr each

Content: Hands-on, interactive activities teach about the short- and long-term effects of substance abuse; healthy ways to manage stress; how to recognize media and peer pressure to use drugs; skills to make responsible decisions about drug use. As peer leaders, youth plan and implement substance abuse prevention activities for 6- to 10-yr-olds

↓ Drinking and delayed onset of drinking

Intellectual

psychological

emotional

social

physical

Emotional and instrumental support; belonging; positive social norms; opportunity for efficacy; opportunities for skill building

Functional Family Therapyc

Age: 11–18

(indicated)

Location: Family

Sessions: 8–12 hrs, 26 hrs max.

Content: Behavioral systems family therapy; flexible delivery of service: 1–2 person teams to clients in-home, clinic, juvenile court, and at time of reentry from institutional placement

↓ Rates of offending, foster care or institutional placement reduced at least 25 percent

Intellectual

psychological

emotional

social

Positive social norms; opportunities for skill building; integration with family, schools, and community

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

Program Name/Age, Grade Evaluated

Program Description

Outcomes

Personal and Social Assets

Features of Positive Developmental Settings

Growing Healthya

Grade: 4th–7th

Location: School

Sessions: 43–56 lessons over 1 or 2 yrs

Content: Health competence promotion

↑ Positive knowledge and attitudes toward health, development and personal responsibility

↓ Smoking and intention to smoke

Psychological

emotional

physical

Positive social norms; physical and psychological safety

Improving Social Awareness-Social Problem Solving (ISA-SPS) Grade: 5th

Location: School

Sessions: 2 yrs

Content: Individual skill-building to promote social competence, decision making, group participation and social awareness; targets transition to middle school

↑ Coping with stressors related to middle school transition

↓ Teacher reports of problem behavior, psychopathology at six-year follow-up; boys: ↓ Alcohol use, violent behavior toward others, self-destructive/identity problems; girls: ↓ rates of cigarette smoking, chewing tobacco, and vandalism

Psychological

emotional

social

physical

Belonging; positive social norms; opportunities for skill building

Know Your Bodya

Grade: 4th–9th

Location: School

Sessions: 2 hrs/wk for 6 yrs

Content: Health promotion

↑ Healthy dietary patterns

↓ Smoking initiation

Physical

Positive social norms

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

Life Skills Traininga,c

Grade: 7th–9th

Location: School

Sessions: 2 sessions/wk for 15 wks (Y1); 10 booster sessions (Y2); 5 booster sessions (Y3)

Content: Competence promotion and resistance training

↑ Interpersonal skills, knowledge of smoking and substance abuse consequences

↓ Cigarette and marijuana smoking, alcohol intoxication and polydrug use

Intellectual

physical

Positive social norms; opportunities for skill building

Louisiana State

Youth Opportunitiesb

Age: 14–16

Location: School, community

Sessions: Summer, 8 weeks

Content: Live on campus, half-day in math and reading classes, half-day working at on-campus sites earning minimum wage, required to open saving account, other services provided: counseling, study skills training, health care, recreation, field trips and speakers

↑ Standardized math test scores, intention to stay in school, career maturity

↓ In reading skills, but less so than control group

Intellectual

psychological

emotional

social

Positive social norms; physical and psychological safety; opportunity for efficacy; opportunities for skill building

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

Program Name/Age, Grade Evaluated

Program Description

Outcomes

Personal and Social Assets

Features of Positive Developmental Settings

Midwestern Prevention

Projecta,b,c

Grade: 6th–7th

Location: Family, school, community

Sessions: School program (10 hrs), homework activities with parents (10 hrs)

Content: Parent education about parent-child communication skills, resistance skills training for youth, community organization, mass media coverage

↓ Monthly, weekly and heavy use of cigarettes, marijuana and alcohol

Intellectual

social

physical

Positive social norms; opportunities for skill building; integration with family, schools, and community

Multisystemic Therapy (MST) (indicated)

Location: Family, community

Sessions: 60 hrs over 4 months

Content: Family ecological systems approach; home-based services delivery

↓ Criminal activity, drug-related arrests, violent offenses and incarceration

Psychological

emotional

social

Emotional and instrumental support; belonging; positive social norms; physical and psychological safety

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

Penn Prevention Projecta

Age: 10–13

(selected)

Location: After school

Sessions: 1.5 hrs/week for 12 wks

Content: Cognitive behavioral; taught coping strategies to counteract cognitive distortions, specific focus on explanatory style

↑ Parents reported improvements in children’s home behavior at follow-up, but not posttreatment (sleeper effects)

↓ Depressive symptoms immediately posttreatment and at a 6-month follow-up period mediated by changes in the children’s explanatory styles

Intellectual

psychological

emotional

Positive social norms; opportunities for skill building

Positive Youth Development Program

Age: 11–14

Location: School

Sessions: 20

Content: Social/emotional cognitive skill-building; curriculum to promote general social competence and refusal skills for alcohol and drugs

↑ Coping skills, ability to generate alternative responses to hypothetical situations, teacher reports of social adjustment including conflict resolution with peers, impulse control, and popularity

No significant effects on measures related to drugs, cigarettes or wine, and only marginal effects related to alcohol

Intellectual

psychological

emotional

social

physical

Positive social norms; opportunities for skill building

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

Program Name/Age, Grade Evaluated

Program Description

Outcomes

Personal and Social Assets

Features of Positive Developmental Settings

Project ALERTa

Grade: 7th–8th

Location: School

Sessions: 10 (7th), 3 booster (8th)

Content: Competence promotion and refusal/ resistance training

↑ Self-efficacy, cigarette and marijuana knowledge and positive attitude, beliefs about immediate and negative social consequences of drugs

↓ Smoking cigarettes and marijuana, expectations of using

Intellectual

psychological

emotional

social

physical

Positive social norms; opportunities for skill building

Project Northlanda

Grade: 6th–8th

Location: Family, school, community

Sessions: Weekly activities and/or training over 3 yrs

Content: Youth skills and parent competence training, community organization

↑ Parent-youth communication, knowledge and attitudes for resisting peer influence, self-efficacy

↓ Alcohol use, cigarettes and marijuana for subgroups by previous risk level; alcohol for full sample

Intellectual

psychological

emotional

social

physical

Positive social norms; opportunities for skill building

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

Quantum Opportunitiesa,b

Grade: 9th–12th

Location: School, community, work

Sessions: Education-related activities (250 hrs), development activities (250hrs), service activities (250hrs) each year for 4 yrs

Content: Education activities, peer tutoring, community service activities, mentoring, life and family skills, incentives: hourly stipends and bonuses for completing program components

↑ High school graduation rates, college or postsecondary school attendance, honors and awards, positive attitudes and opinions about life and future, volunteer community service work

↓ Trouble with police, high school drop out, number of children

Intellectual

psychological

emotional

social

physical

Emotional and instrumental support; belonging; positive social norms; physical and psychological safety; opportunity for efficacy; opportunities for skill building; integration with family, schools, and community

Reducing the Riska,c

Grade: 10th

Location: Health education classes

Sessions: 15

Content: Cognitive-behavioral, teacher and peer role modeling, parent involvement, emphasis on avoiding unprotected sex either by abstinence or using protection

↑ Knowledge and communication with parents about contraception and abstinence, changes in normative beliefs; increased contraceptive use for females, lower-risk and sexually inexperienced at pretest

↓ Rates of initiation of intercourse

Intellectual

psychological

emotional

social

physical

Positive social norms; opportunities for skill building

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

Program Name/Age, Grade Evaluated

Program Description

Outcomes

Personal and Social Assets

Features of Positive Developmental Settings

Responding in Peaceful and Positive Waysa

Grade: 6th

Location: Family, school, community

Sessions: 25 over school yr

Content: Social/cognitive skill-building to promote nonviolent conflict resolution and positive communication; activities included team building and small group work, role playing, and relaxation techniques

↑ Decision-making knowledge and use of peer mediation (not found on student self-reports of behavior however)

↓ Weapon carrying, in-school suspensions

Intellectual

social

physical

Positive social norms; opportunities for skill building

School Transitional Environment Project

Grade: 6th–7th

(universal)

Location: School, family

Sessions: Transition to middle school yr

Content: Students placed in cohort, homeroom teacher becomes adviser to cohort and liaison among student, family, and school.

Homeroom teacher meets with other teachers to identify students needing counseling or support

↑ Better adjustment on measures of anxiety, depression, self-esteem and delinquent behavior; better teacher ratings of classroom behavioral adjustment; grades and attendance patterns

↓ Levels of school transitional stress

Social

Emotional and instrumental support; belonging; integration with schools

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

Social Competence Program for Young Adolescentsa

Grade: 5th–8th

Location: Family and school

Sessions: 16, 45 min each over 12 weeks; teacher and aide training, consultation and coaching

Content: Social competence promotion, family involvement

↑ Peer involvement, social acceptance, problem solving, use of conflict resolution strategies, positive solutions

↓ Aggressive and passive solutions

Psychological

emotional

social

physical

Positive social norms; opportunities for skill building

Summer Training and Education Program (STEP)b

Age: 14–15

Location: School, community

Sessions: 6–8 weeks during summer; half-time jobs (90 hours), half-day academic classes (90 hours), 2 mornings/wk in life skills training

Content: Employment and academic classes, life-skills training, part for classroom time as well as work time

↑ Reading and math test scores, knowledge tests of responsible social and sexual behavior (summer effects only; no effect for school year or long-term)

Intellectual

physical

Positive social norms; opportunity for efficacy; opportunities for skill building

Teen Incentives Programc

Grade: 9th

Location: After school program

Sessions: 14 weeks

Content: Small group sessions focused on self-esteem, decision making, communication, relationships, sexuality topics; career mentorship program with professional in health care (6wks); extensive role playing

↑ Contraceptive use by females inexperienced at pretest, consistent use by experienced males at pretest

↓ Initiation of intercourse for males

Intellectual

psychological

emotional

social

physical

Belonging; positive social norms; opportunity for efficacy; opportunities for skill building

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

Program Name/Age, Grade Evaluated

Program Description

Outcomes

Personal and Social Assets

Features of Positive Developmental Settings

Teen Outreach Program (TOP)a,b,c

Grade: 10th

Location: School and community

Sessions: (school year) 45 hrs volunteer service, weekly classroom discussions and activities

Content: Small group classroom discussions of values, decision making, communication skills, parenting, life options and volunteer experiences; volunteer service in school or community

↓ School failure and suspension, school suspension and teen pregnancy

Intellectual

psychological

emotional

social

physical

Emotional and instrumental support; belonging; positive social norms; physical and psychological safety; opportunity for efficacy; opportunities for skill building

Valued Youth Partnershipa

Age: 12

Location: family, school, community

Sessions: 30 over school yr, 4 hrs of tutoring/week

Content: peer tutoring, stipends, leadership training, parent and business community involvement

↑ Reading grades, positive self-concept, positive attitudes toward school

↓ School dropout rates

Intellectual

psychological

emotional

social

Positive norms; opportunity for efficacy; opportunities for skill building

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

Woodrock Youth Development Projecta,b

Age: 6–14

Location: Family, school, community

Sessions: Weekly classes and activities, daily mentoring, home visits and contacts

Content: Social competence promotion, life skills, human relations classes to develop resiliency skills, peer tutors, homework assistance, extracurricular activities (weekend retreats, after-school clubs, crisis intervention, summer program), parent training and involvement

↑ Positive race relations, school attendance

↓ Drug use for past year (younger subgroup) and past month (older and younger subgroups)

Wrong direction outcome on attitudes toward drug use in older subgroup

Intellectual

psychological

emotional

social

Emotional and instrumental support; belonging; positive social norms; opportunities for skill building; integration with family, schools, and community

aCatalano et al., 1999.

bRoth and Brooks-Gunn, 2000.

cElliot and Tolan, 1999.

dGreenberg et al..

eKirby, 1997.

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

assertiveness, communication skills, feelings of self-confidence, and skill performance, assets consistent with the committee’s framework. More specifically, the meta-analysis of the primary prevention programs (Durlak and Wells, 1997) revealed that most programs significantly reduced problems (e.g., anxiety, behavior problems, and depressive symptoms) and increased competencies.

Durlak and Wells (1997, 1998) also compared the effects of environment-centered versus person-centered approaches. Person-centered programs worked directly with individuals using techniques based on social learning theory (either modeling appropriate behaviors or reinforcing appropriate behaviors) and other direct instructional approaches focused on educational and interpersonal problem solving. Environment-centered programs tried to change either the home setting (through parent education about child development and changing parental attitudes and childrearing techniques when appropriate) or the school setting (often through teacher training in interactive instructional techniques and classroom management skills). Parent education was often used as a general descriptor, but all of the person-centered programs were effective for some outcomes, particularly for increasing competencies. Of the environment-centered approaches, only school-based programs were effective, and these were more effective at increasing competencies than reducing problems. Most of the school-based environment-centered programs focused on changing the psychological and social aspects of the classroom environment through increasing either interactive instructional techniques or effective classroom management techniques.

In their secondary prevention meta-analysis, Durlak and Wells (1998) compared the effectiveness of behavioral, cognitive-behavioral, and nonbehavioral treatments.2 In general, all three treatments were effective at both problem reduction and competency enhancement. In addition, cognitive-behavior treatment was more effective than either behavioral treatment or nonbehavioral treatment at reducing mental health problems, and the behavioral treatments were more effective than the nonbehavioral treatment at improving children’s competencies.

2  

Reinforcement, modeling, and desensitization procedures were the behavioral treatments used in these interventions. The cognitive-behavior treatments emphasized self-instructional training and other ways of using cognitive processes to modify behavior. The nonbehavioral treatments mostly used classic talking analytic techniques (the classic situation in which a client talks with a therapist about his or her problems or concerns) for older children and activity-oriented play therapy for younger children.

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

Because these programs are based on well-established clinical theories, they have a strong theory of change.3 Little information about these theories, however, was included in the information provided by the programs, and very little information was provided about the quality of the services actually provided. Such information is critical if these programs are to be implemented in other settings and programs by staff members who are not clinical psychologists or social workers. Certainly the primary prevention efforts should be replicable in after school community-based programs. But more information will be needed about the specifics of the programs before this is possible.

Greenberg and Colleagues

Greenberg and colleagues (1999) reviewed 130 universal, selective, or indicated mental illness prevention programs for children and adolescents. The universal prevention programs targeted whole populations of youth. The selective programs targeted individuals or subgroups identified as at risk for developing mental disorders. The indicated prevention programs targeted individuals identified as having early warning signs of mental disorders but not yet meeting diagnostic criteria. Out of 130 total programs, researchers selected 34 using the following criteria: a randomized experimental design or quasi-experimental design with a comparison group; pretest and posttest measures; a written manual specifying the theory and procedures used in the program intervention; a clearly defined sample, with adequate information about their behavior and social characteristics; and some evidence of positive mental health outcomes. Follow-up assessments were preferred but not required. This review focused less on the personal and social assets identified in the committee’s framework and more on the features of the programs themselves and on prevention of problem behaviors. In general, the researchers concluded:

  1. Multiyear preventive programs produce longer-lasting effects than short-term programs.

3  

“Theory of change” is described as a clear, substantive theory that explicitly states the process by which change should occur. Ideally, this theory should be the driving force in program development and should guide the decision of what to measure and how. See Chapter 7 for elaboration.

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
  1. Preventive program interventions should be directed at risk and protective factors4 rather than at categorical problem behaviors, such as school dropout, delinquency, substance abuse, and teenage pregnancy.

  2. The few studies with follow-up data suggest possible “sleeper effects” in program benefits (that is, some programs showed a greater effect at a time somewhat distant from the end of the treatment than immediately following the intervention). If this is true, programs that do not collect follow-up data may underestimate the impact of their program.

  3. Prevention programs targeting multiple domains (i.e., individual, school, and community) are more effective than programs targeting one domain.

We looked more closely at whether these conclusions held true for the 10 programs targeting 10- to 18-year-olds (see Table 6–1 for details). Do short-term programs produce time-limited benefits? This was difficult to determine for this subset of programs because half of the eight short-term program evaluations (less than one year) did not do follow-up testing. Of the three programs that did follow-up testing, two found mixed results and one found positive follow-up effects. These three programs are summarized below.

The Adolescent Transition Project randomly assigned participants to one of four groups: teen focus, parent focus, teen-parent focus, and a self-directed control group. The teen focus group received training in self-regulation development along with skill development exercises. The parent focus curriculum provided training for parents in parent management skills in areas such as setting appropriate limits and problem solving with teens. The combined parent-teen group used peer consultants to open discussions between teens and parents on issues of self-regulation, parent-child interaction, and communication patterns and life skills. In the comparison group, the teens were provided with self-directed instructional materials focused on self-regulation and life skills development. After 12 weekly 90-minute sessions, the parents and adolescents in all three treatment groups reported significantly less home-based problem behavior than at the pretest; in contrast, however, there was an in-

4  

Risk factors, such as academic failure or poverty, are variables associated with a greater likelihood of negative or undesirable outcomes. Protective factors, such as warm, nurturing adult relationships, and the personal and social assets outlined in Chapter 3, reduce the likelihood of problem behavior in the presence of risk.

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

crease in teacher reports of school problem behavior for the adolescents in the teen-focused treatment group at the one year follow-up. No explanations were provided for these patterns of results or for why school problem behavior may have increased in the teen-focused treatment group.

In the Coping with Stress Course, a subset of targeted adolescents with elevated self-reported depressive symptomatology were randomly assigned to 15 group clinical counseling sessions of 45 minutes each. The “treated” adolescents’ reports of depressive symptomatology declined more that the control adolescents from pretest to posttest, but this difference had disappeared by the 12-month follow-up. In contrast, however, there were fewer cases of major depressive disorder and dysthymia (a milder form of depression) in the treatment group than in the control group at the follow-up testing.

Only the Penn Prevention Project found unambiguous evidence of both follow-up and sleeper effects. This selected intervention program targeted children ages 10 to 13 at risk due to elevated depressive symptoms or family conflict. The randomly selected intervention groups met once each week for 12 weeks after school for 1.5 hours. Postintervention data showed fewer depressive symptoms, better classroom behavior, and reduced likelihood of attributing negative events to stable enduring causes for the treatment group compared with the “untreated” control group. While levels of both the experimental and control groups’ depressive symptomatology increased over the 24-month follow-up period in the Penn Prevention Project, this increase was larger in the control group than in the treatment group. Also, even though there were no significant differences at the posttest, parents of the intervention group reported more improvements in the children’s home behavior at the 24-month follow-up than parents of the control group. Effects were maintained at the 12-, 18-, and 24-month follow-ups. Interestingly, although this program is considered short-term, it was the only program of the six in this review focused on adolescents with follow-up data suggesting a possible sleeper effect in program benefits. This calls into question the claim that long-term interventions are needed for long-lasting effects.

What is most interesting about the Penn Prevention Project is that it has been implemented in a variety of in-school and out-of-school settings, including community-based organizations. It has also been evaluated repeatedly and has had good success at replication. The program consists primarily of training in self-management skills and social interpretative skills. The adolescents are taught new ways to cope with inter-

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

personal conflict as well as new ways to interpret other people’s behaviors so that they do not misinterpret social interactions as more hostile or demeaning than they actually are. These skills in turn are assumed to help the adolescent cope better with interpersonal interactions likely to reinforce depression and conflict. The Penn Prevention Project has extensive program materials and training manuals to help organizations implement the program based on its principles (Shatté and Reivich, no date).

Do multiyear programs foster more enduring benefits than short-term program interventions? Four programs involving adolescents come close to meeting the multiyear criteria, but only two, the Improving Social Awareness-Social Problem Solving (ISA-SPS) and the Bullying Prevention Program, actually extended beyond one year. The ISA-SPS two-year program targeted the transition to middle school by focusing on changing the school culture. The social problem-solving curriculum in ISA-SPS has three phases. The readiness phase emphasized teaching self-control, group participation, and social awareness. In the instructional phase, teachers teach social decision-making and problem-solving strategies. In the final application phase, teachers extend the curriculum to real-life conflicts in which the students are helped to use problem-solving skills to solve their conflicts through modeling, direct instruction, and group discussion. In the evaluation reviewed, positive gains in the youth’s interpersonal behaviors (see Table 6–1 for details) were maintained for six years after exposure to the intervention.

The Bullying Prevention Program, part of a national campaign against bullying in Norway, lasted two years. It is a school-based universal prevention program focused on reducing bullying problems by increasing awareness and knowledge of the problem, involving teachers and parents together with adolescents in discussing these problems, establishing clear rules against bullying behavior, and providing support and protection for bullying victims. In the evaluation of this program reviewed by Greenberg, there were decreases in bullying for the “treated” adolescents. The positive effects of this program were maintained at least through the 20-month follow-up assessment.

The other two programs, Big Brothers Big Sisters and the School Transition Environmental Project, were 9 to 12 months in length. Mentors in the Big Brothers Big Sisters program (described in more detail later in this chapter) met with their assigned child three times per month for three to four hours for at least one year. Treatment effects were found across multiple outcome measures ranging from hitting behavior

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

to academic performance (see Table 6–1 for details). Follow-up data have not been collected.

The School Transitional Environmental Project focused on changing the school social climate to assist in the transition to middle school in large school systems with many feeder schools. Homeroom teachers were assigned to smaller cohorts to function as a liaison and guidance counselor. The effects of the program intervention were positive (see Table 6–1 for details), but no follow-up data were collected, so we know nothing about the stability of the effects over time.

So, do the programs targeting adolescents, warrant the same conclusions reached by Greenberg and colleagues for the full array of programs included in the review? First, do multiyear preventive programs produce longer-term effects than short-term program interventions? Many positive long-term effects were found in the two programs lasting longer than one year; however, with only two programs as examples, it would be unfair to conclude that multiyear programs contribute to longer-lasting effects than short-term programs, especially given the evidence of sustained effects from two of the short-term programs discussed earlier.

Second, are programs targeting multiple levels and settings (i.e., individual, school, community) more effective than programs targeting one domain? The programs most relevant to our age group give mixed support for this conclusion. On one hand, the eight programs concentrating on individual change achieved positive postintervention effects. The two programs with follow-up data, the Penn Prevention Project and the Coping with Stress Course, also showed evidence of sustained effects at follow-up assessments. Thus, programs targeting the individual can be effective. However, it is quite likely that these programs also indirectly create changes in other domains, such as the family or school. On the other hand, the four programs targeting multiple levels also yielded positive postintervention effects, and the three programs with follow-up data showed long-term effectiveness as well.

In summary, from the 12 programs targeting youth ages 10 to 18, we can draw several conclusions. Short-term programs can be just as effective as programs lasting nine months to two years. Program interventions concentrating on individual change can be as effective as those focusing on multiple domains. One program, a selected intervention, demonstrated evidence of a sleeper effect. The possibility of sleeper effects underlines the importance of collecting follow-up data to ensure that effective programs are not being underestimated in their impact and to capture potential effects of programs yielding initial null results. Based

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

on these evaluations, it appears that mental disorder prevention programs can be designed to support positive psychological and emotional development. By and large, the effective programs used skill-building instruction and extensive opportunities to practice these new skills at the individual level, and both social support and social norm-building strategies at the school and program level. Future evaluation research in this domain should focus on the causal links between program interventions and outcomes and on the extent to which these programs can be implemented in community settings beyond the school and the school day. Some clearly were. Nonetheless, these results do suggest that life skill-building and positive norm-setting programs can be effective when done well.

Summary

By and large, the program interventions designed to prevent mental illness can be quite effective and, when assessed, appear to have long-term positive consequences. Programs based on clinical theories of behavior change and sound instructional practices are effective at both reducing problem behaviors and increasing a wide range of social and emotional competencies. In addition, interventions in the field of mental health promotion use high evaluation standards. All evaluations included in both reviews used control group comparisons, and the majority used random assignment. The high level of evaluation rigor obtained was understandably facilitated by the short-term nature of the programs, the integration of these programs into the school day, and the fact that program participation was more likely to be seen by participants as required rather than voluntary. Consequently, generalizabilty of the findings to nonschool settings is problematic. Nevertheless, there is sufficient evidence to conclude that programs such as the ones reviewed, which target the prevention of mental disorders, can be effective in both reducing problems and increasing social and emotional competencies. These programs are appropriate for consideration in many types of community-based programs and organizations.

Violence Prevention Programs

The Center for the Study and Prevention of Violence at the University of Colorado at Boulder published a series of Blueprints (Center for the Study and Prevention of Violence, 2000) describing program interventions effective in preventing violence. The Blueprints series was de-

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

veloped to give practical information to communities to help them select a program intervention that best matched their needs and resources. Of the 450 delinquency, drug, and violence prevention programs the center reviewed, 10 met most of the following criteria: an experimental design with random assignment or a strong quasi-experimental design; evidence of a statistically significant deterrent (or marginal deterrent) effect on delinquency, drug use, and/or violent behavior; at least one additional site replication with experimental design and demonstrated effects; and evidence that the deterrent effect was sustained for at least one year following treatment.

Of the 10 model programs, 6 included youth ages 10 to 18 in their target population: Big Brothers Big Sisters, Quantum Opportunities, the Midwestern Prevention Project, Life Skills Training, Multisystemic Therapy, and the Bullying Prevention Program. The first three are described in detail later in this chapter; the Bullying Prevention Program was described earlier. The remaining two—Life Skills Training and Multisystemic Therapy (see Table 6–1 for more details)—are described below.

The Life Skills Training program is a school-based universal prevention program designed to prevent drug use. The program intervention is a curriculum to teach general life skills and social resistance skills training.

The Multisystemic Therapy Program uses a family ecological systems approach to help serious violent or substance-abusing juvenile offenders. The therapy focuses on getting families involved in changing those aspects of the youth’s setting (i.e., peers, school, family, and community) that contribute to the problem behavior. The primary method is teaching effective parenting skills and helping parents overcome such barriers to effective parenting as drug abuse and lack of a social support network in the community.

In an effort to understand the source of change across the 10 violence prevention programs reviewed, Elliot and Tolan (1999) looked for evidence that change in risk or protective factors mediated change in violent behavior. However, the program evaluations either had not collected the necessary data to analyze the causal processes or had not reported on the analysis. Likewise, because the studies evaluated whole program packages rather than specific program components, it was not possible to determine exactly what worked in any given program. Nevertheless, these 10 programs do provide models of what can be done in communities to decrease rates of violence. Interestingly, most of the

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

programs involved the kinds of experiences and settings outlined in Chapter 4 as critical for promoting positive youth development. Teaching life skills and providing better adult social supports were common across all of the effective programs. Consequently, these programs are also good models for what can be done to promote positive development as well. Most actually gathered data relevant to this goal (see Table 6–1 for details) and were effective at promoting some aspects of positive development as well.

Teen Pregnancy Prevention Programs

Kirby (1998) reviewed evaluations of primary prevention programs designed to reduce sexual risk-taking and teen pregnancy. These programs met the following criteria: published in 1980 or later; experimental or quasi-experimental in design; a minimum sample size of 100 in combined experimental and control groups; targeted 12- to 18-year-olds; conducted in the United States or Canada; and measures of program impact on sexual or contraceptive behavior or pregnancy or birth rates.

Kirby divided programs into three groups based on whether they focused primarily on sexual antecedents (i.e., age, gender, pubertal timing), on nonsexual antecedents (i.e., poverty, parental education, parental support, drug and alcohol use), or on a combination of sexual and nonsexual antecedents. We focused on the latter two groups because of their fit with our youth development framework. Youth development programs were further categorized as service-learning, vocational education and employment, and other. All three categories focused on improving education and life options as the means to reduce pregnancy and birth rates.

Service-learning programs consisted of unpaid service time in the community as well as structured time for training, preparation, and reflection. The results for such programs were equivocal. On one hand, Teen Outreach Program participants reported lower rates of pregnancy and school failure than the controls during the school year in which they participated in the program intervention. Similarly, a health education curriculum combined with service learning was effective at reducing reported sexual activity. On the other hand, although a quasi-experimental evaluation of service-learning programs showed a short-term trend in reduced pregnancy rates, the result was not statistically significant and the trend disappeared one year later.

Vocational education and employment programs included academic

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

education and either vocational education or actual jobs. Again the results were equivocal. On one hand, the Youth Incentive Entitlement Pilot Projects was effective at reducing birth rates and increasing both teen employment and school enrollment. This program guaranteed part-time employment to youth during the school year and full-time employment during the summer if they remained in high school. On the other hand, four very rigorously evaluated employment and education programs were not effective in reducing pregnancies or birth rates. This evidence suggests that vocational and employment programs are not effective in decreasing pregnancy or birth rates.

In summary, the strongest evidence for program effectiveness using a youth development framework comes from the Teen Outreach Program. Why was this service-learning program effective while the others were not? We do not have an answer for this question because causal links in the interventions were not evaluated. Therefore, we do not know whether to attribute its success to the service-learning component or to other characteristics of the program, such as strong bonds with adults. Nevertheless, the program evaluations do offer promise that a youth development approach may provide opportunities that reduce the risk of teenage pregnancy.

Furthermore, Kirby summarized the strengths and limitations of each evaluation based on the following methodological principles: sampling, random assignment, sample size, long-term follow-up, measurement of behavior (sexual and contraceptive behaviors), statistical analyses, publication of results, replication, and independent evaluation. Kirby concluded that few studies abided by high standards for each principle, thus limiting generalizations about which specific programs or types of programs reduce sexual risk-taking and teen pregnancy.

Positive Youth Development Programs
Catalano and Colleagues

The most comprehensive review of positive youth development program evaluations was funded by the Department of Health and Human Services and compiled by Catalano, Berglund, Ryan, Lonczak, and Hawkins (1999). This report examined evaluations of positive youth development programs solicited from a wide variety of sources. The authors reviewed 77 program evaluations and chose 25 for their report (see Table 6–1 for details). Only research designs using a control or

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

strong comparison group with measures of youth behavioral outcomes were included. Programs had to have at least one significant effect.5 The programs selected had to include at least one of the following objectives identified by the authors as important for positive youth development: promotes bonding; fosters resilience; promotes social, emotional, cognitive, behavioral, and moral competence; fosters self-determination, spirituality, self-efficacy, a clear and positive identity, belief in the future, and/or prosocial norms; or provides recognition for positive behavior and/or opportunities for prosocial involvement. In general, these characteristics and strategies of effective positive youth development programs closely match the personal and social assets and features of positive developmental settings identified in Part I as important for healthy adolescent development.

Catalano et al. concluded that the most effective programs sought to strengthen social, emotional, cognitive, and behavioral competencies, self-efficacy, and family and community social norms for healthy social and individual behavior. To achieve these goals, the effective programs targeted a combination of social settings (family, school, church, community, and work). Of the social settings mentioned, 88 percent had a school component, 60 percent had a family component, and 48 percent had a community component.

Many of the positive youth development principles identified by Catalano et al. are closely linked to the assets and social settings identified by the committee. Opportunities to develop competence and self-efficacy, as well as the inclusion of prosocial norms (positive social norms in our language) were present in all 25 programs. Opportunities for prosocial involvement and recognition of positive behavior were provided in 88 percent of the programs. Social bonding was promoted in 76 percent of the programs. Explicit opportunities to develop the other five target assets (positive identity, self-determination, belief in the future, resiliency, and spirituality) were a focus for fewer than half of the programs.

5  

Although some may feel that the researchers erred in including only programs with significant results, the state of evaluation in this area is still such that it is virtually impossible to draw conclusions from nonsignificant results. Nonsignificant results could reflect poor program implementation, weak measures of outcomes, inadequate specification of what outcomes should be effected, program attrition, etc. All of these design problems make it very difficult to reach any conclusion about what does not work. Given this state of affairs, we agree with Catalano et al.’s decision not to include programs with nonsignificant effects in their review, and we have also chosen not to report on programs that do not work.

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

Catalano et al. (1999) concluded that two general program strategies were evident in the most effective programs: skill building and environmental/organizational change. A focus on social or cognitive behavioral skills occurred in 96 percent of the programs; decision-making and self-management skills in 73 percent; coping skills in 62 percent; and refusal-resistance skills in 50 percent. Effective environmental or organizational change programs focused on influencing teacher classroom practices and peer social norms.

This review of youth programs was very comprehensive. Catalano et al. concluded that positive youth development approaches could, when implemented well, increase positive youth behavior outcomes and decrease youth problem behaviors. In 19 of the programs reviewed, positive changes in youth behavior included significant improvements in interpersonal skills, quality of peer and adult relationships, self-control, problem solving, cognition, self-efficacy, commitment to schooling, and academic achievement. Of the programs reviewed, 24 showed significant reduction in problem behaviors, including drug and alcohol use, school misbehavior, aggressive behavior, violence, truancy, high-risk sexual behavior, and smoking.

However, Catalano et al. also noted that many of the evaluations themselves had limitations. Of the 25 programs meeting the inclusion criteria, only 64 percent used experimental designs with randomization and only half gathered any follow-up data. None included comprehensive information about the program, the implementation process, the youth development constructs being addressed, or the relation between the implementation information and outcomes. Also, assessment measures were rarely adequate to track positive youth development over time, and problem behaviors were measured much more frequently than positive behaviors. Consequently, much more research is needed before one can be confident about which programs and which aspects of these programs actually impact youth development. Nonetheless, the findings are promising enough that they can be used to inform future program design and to make decisions about which programs are ready for more rigorous evaluation.

Roth and Colleagues

Roth et al. (1998a) reviewed over 60 evaluations of prevention and intervention programs for adolescents that incorporated positive youth

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

development program objectives. Of the 60 programs reviewed, 15 were selected for inclusion in the review (see Table 6–1 for details). These 15 shared the following characteristics: a positive youth development focus, an experimental or quasi-experimental design, and a focus on youth not currently demonstrating problem behaviors. The programs were grouped into three categories (listed in order of how closely they matched a youth development framework): (a) programs focused on increasing positive behaviors and competencies (asset-focused programs); (b) programs focused on both reducing problem behaviors and increasing competencies and assets (problem-behavior and asset-focused programs); and (c) prevention programs focused on resistance skills.

The authors concluded that programs incorporating more elements of the youth development framework (a framework that seemed very consistent with the assets and features of settings developed in this report) showed more positive outcomes—that is, programs that focused on increasing more different assets and included more different types of program characteristics and opportunities. However, the evidence for this conclusion is weak, and the number of assets and features included in specific programs has not been systematically manipulated in such a way that such a conclusion is warranted at this point based on experimental and quasi-experimental evaluations. Like the Catalano et al. review, Roth and colleagues also concluded that the most effective programs included both caring adult-adolescent relationships and life skills development. In addition, the authors concluded that longer-term programs were more effective than short-term programs with limited goals. Again, however, the evidence for this is minimal.

This review stands out because it included a framework to categorize youth development programs; it focused on community-based (rather than school-based) programs; and it insisted on rigorous standards of evaluation before concluding that a program was effective. Thus it provides quite comprehensive evidence that community programs focused on youth development can be effective.

Unfortunately these evaluations still did not reveal much about either the long-term effects or the generalizability of these programs. There was also no insight about which specific aspects of the programs were most effective for any particular outcome or population group. Nonetheless, they do provide useful information about promising programs that should be considered for replication and for more intensive evaluation.

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Hattie and Colleagues

Hattie and colleagues (1997) performed a meta-analysis of the effects of adventure programs drawn from extensive database searches. These searches yielded 96 evaluations of adventure programs meeting the inclusion criteria for the review. The authors used sample size, the presence of control groups, methodological descriptions, and instrument quality to rate the evaluations as low, medium, or high in quality. Nine studies were excluded based on the four authors’ agreement on their poor quality and because their means were so different from the other studies. School-based outdoor education program evaluations were also excluded because of a typically shorter duration, nonchallenging experiences (relative to other adventure programs), and evaluation results that differed from the more challenging programs.

Although adventure education programs do not usually take place in the communities from which the youth are being recruited and the evaluations of these programs suffer from the same methodological problems encountered in the previous reviews, we include the findings from this review because effective adventure programs can suggest potential program components important for healthy development. Adventure programs also typically include features of positive developmental settings outlined in Part I. Common features of adventure programs include wilderness or backcountry settings, groups of less than 16, mentally and physically challenging experiences, intense social interactions often relating to group problem solving and decision making, trained nondirective leaders, and an average duration of two to four weeks.

Hattie et al. concluded that short-term positive effects were maintained and sometimes even increased over time. More specifically, the authors concluded that adventure programs have significant effects on the kinds of psychological, emotional, and cognitive assets discussed in Chapter 3—such as self-control, confidence in one’s abilities to be effective, good decision making, improved school achievement, leadership, independence, assertiveness, emotional stability, social comparison, time management, and flexibility.

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
General Conclusions
Positive Development Framework
  • Two program components emerged as important across all reviews: service learning and mentoring. Service learning was a particularly crucial program component for youth ages 15 to 18.

  • The vast majority of programs reviewed were school-based and included young people of all ages. Whether such programs can be successfully implemented in a more voluntary setting (e.g., during nonschool hours and in out-of-school settings) is not clear.

  • There is little overlap between the measures collected in most of the evaluations summarized in these reports and the framework for positive adolescent development outlined in Part I. In part this reflects the fact that these studies were not designed to assess the features in this framework. But it might also suggest a scarcity of well-validated measures of personal and social assets and features of positive developmental settings (see Chapter 8 for more discussion of this issue). In part it also likely reflects the disconnection between research and practice traditions and points to the need for more collaborative efforts involving the research, practice, and policy communities in both designing and evaluating community programs for youth.

  • Most evaluation studies show modest short-term effects; some show longer-term effects, but follow-up data are rare and, when collected, rarely cover a period of more than two years after the program end. In general, follow-up studies tend to show smaller effects over time.

The Quality of Evaluation
  • Most of the evaluations conducted in school-based programs included in this review met high technical standards for assessing effectiveness for individual children. However, these programs focused more on preventing negative outcomes than on fostering positive development.

  • Evaluations of more voluntary community-based programs are still less rigorous than the evaluations of school-based prevention programs. Possible reasons for this discrepancy are discussed in Chapter 7.

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
  • Few evaluations are comprehensive. That is, they do not have: an analysis of program theory, quality information about implementation, quality information about effectiveness, quality analysis of why effects did or did not come about, an analysis of the replicability of the program, or an analysis of how important the program is and why.

THREE MODEL PROGRAM EVALUATIONS

Three programs—the Teen Outreach Program, Quantum Opportunities, and Big Brothers Big Sisters—stand out among the programs included in the meta-analyses and reviews just summarized for several reasons. First, these programs are consistent with the youth development framework laid out in Part I. Second, these programs illustrate that high-quality experimental evaluation can be done with community programs for youth. These three programs also reflect different program delivery mechanisms. Big Brothers Big Sisters and Quantum Opportunities Program are offered by community agencies, while the Teen Outreach Program has community components but is managed by schools. We now describe these in more detail because there is sufficient evidence that they can be used a promising models for the design of new youth programs.

Teen Outreach Program

Evaluators:

J.P.Allen et al. (1997)

Level of intervention:

Multilevel (community, school)

Target population:

9th–12th grade students

Evaluation research design:

Experimental with random assignment

Sample:

695 (342 experimental, 353 control)

Attrition analysis:

Dropped out of program prior to evaluation: 5.3 percent experimental; 8.4 percent control

Cost:

$500–700/student/academic year for class of 18–25 students

The Teen Outreach Program is a widely replicated program that involves both a school-based discussion curriculum focused on life skills, parent-adolescent communication, and future life planning and an intensive volunteer service experience. Although it was designed to prevent

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

adolescent problem behaviors by promoting healthy social development in high school students, it focuses very little attention directly on sexual behavior and reproduction. The evaluation focused on three problem behaviors: teenage pregnancy, school suspension, and school failure. The program was designed to help adolescents understand and evaluate their future life options and develop life skills and autonomy in a setting that creates strong social ties to adult mentors. To accomplish these goals, three components were designed: supervised community service, classroom-based discussions of service experiences, and classroom-based discussion and activities related to social-developmental tasks of adolescence. The cost of offering the program to a class of 18 to 25 for an academic year is $500 to 700 per student, when both facilitator and site coordinator time is included in the cost. When staff time is provided as an in-kind contribution by a school and a community service organization, the direct costs of the program drop to under $100 per student.

Each adolescent participated in an intensive volunteer activity (drawn from a wide range of such opportunities) coordinated by trained staff working in cooperation with local community organizations. The activity was chosen by the participant with the help of trained staff who assisted by matching the individual’s interests with community needs. Examples of activities included working as aides in hospitals and nursing homes, participating in walkathons, and peer tutoring. The program sites were required to provide a minimum of 20 hours per year of volunteer service for each participant. Participants averaged 45.8 hours of volunteer service during their 9 months of involvement.

Classroom discussions occurred at least once weekly throughout the academic year; trained facilitators, usually teachers or guidance personnel, led them. The Teen Outreach Curriculum created the framework for such activities as structured discussions, group exercises, role plays, guest speakers, and informational presentations. Discussions were designed to help students prepare for, and learn from, their service experiences by dealing with such topics as lack of self-confidence, social skills, assertiveness, and self-discipline. Discussions also focused on such developmental topics as values clarification, managing family relationships, and handling close relationships.

The program focused very little explicit attention on the three target problem behavior outcomes. For example, less than 15 percent of the “official” curriculum dealt with sexuality, and even these materials were often not used, either because they overlapped with similar information being offered in the school or were in conflict with community values.

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Evaluation Design and Results

Because the program had showed promise in previous nonexperimental studies (Allen et al., 1990; Philliber and Allen, 1998) with high school students, program developers submitted the program to a rigorous evaluation. In this evaluation, 695 high school students in 25 schools nationwide were randomly assigned to either the intervention or a control group at either the student level or the classroom level. Student-level assignment was used when there were more students wanting the class than there were slots. In this case, students were randomly assigned to get the class or be assigned to a wait list for the following year. Classroom assignment was used when the school preferred to include the program in already existing courses. In this case, it was randomly decided which course would be involved in the program and which would not; students did not select the class knowing it was going to be involved in the program. Baseline data were gathered at the start of the program; posttest data were gathered nine months later at the end of the program. Data were collected during the 1991–1995 school years. (A subsequent additional evaluation has been completed but the results are not yet available). After controlling for initial experimental and control group nonequivalence on the outcome measures, fewer youth in the experimental group were suspended, failed courses, or became pregnant than students in the control groups.

Strengths and Limitations

Some aspects of the evaluation of Teen Outreach Program are exemplary. Initial nonexperimental evaluations of the program (Allen et al., 1990, 1994) indicated that it was very promising. The next logical step for a nationally distributed well-developed program showing promise was to evaluate it using the most rigorous evaluation method—an experimental design with random assignment. Great care was taken to assess dosage. Two indicators of dosage were included in the analysis: the number of classroom sessions attended and the number of hours of volunteer service. There were no significant dosage effects for either indicator on the prediction of pregnancy or academic suspension. However, a significant dosage effect was found for the relation between the number of volunteer hours and course failure: students who performed more volunteer service were at lower risk for course failure during the program. Consequently, the positive effects produced by the evaluation

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

using 25 different sites gives a high degree of confidence that the program effects are generalizable.

However, neither the program design nor the evaluation was explicitly guided by an elaborated theory of change. In addition, only a limited range of outcomes was assessed, and little information was gathered about the quality of the implementation. Finally, no follow-up data were collected to assess how sustained the effects were.

In conclusion, the essentials of the program are well based in research, but the evaluation did not have a well-articulated theory of change. The evaluation research showed that the program reduced teenage pregnancies, school failure, and academic suspensions. It is especially effective with adolescents who need the program the most—at-risk youth. While no single study is beyond criticism, the technical quality of evaluation research on this program has steadily improved with each study, and no common bias runs throughout the entire program of studies. So replication and generalizability are not a problem. Dosage effects were included in the evaluation analyses. There is some evidence that more frequent participation in the program relates to outcomes, although there are selection issues. No fine-grained analyses were done to determine which program components were responsible for the effect.

Quantum Opportunities Program

Evaluators:

Hahn, Leavitt, and Aaron (1994)

Level of intervention:

Multi-level (community, school, work)

Target population:

9th–12th grade students receiving public assistance

Evaluation research design:

Experimental with random selection and assignment

Sample:

50 students (25 intervention, 25 control) at 5 sites

Attrition analysis:

88 of the 100 program participants; 82 of 100 control participants were interviewed in the follow-up evaluation.

Cost:

$10,600/student for 4 years; $2,650/ student/year

The Quantum Opportunities Program was designed to serve very poor adolescents living in high-risk neighborhoods by providing educa-

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

tion, community service, and development activities, as well as financial incentives, from 9th grade through high school graduation. Adolescents from families receiving public assistance entered the program in the 9th grade and continued for four years through high school.

It is a community-based, year-round, multiyear, and multilevel youth development program. Each participant was eligible to receive the following experiences each year:

  • 250 hours of education activities, such as participating in computer-assisted instruction, peer tutoring, etc., with the goal of enhancing basic academic skills;

  • 250 hours of developmental activities, including participating in cultural enrichment and personal development, acquiring life/ family skills, planning for college or advanced technical/vocational training, and job preparation; and

  • 250 hours of service activities, such as participating in community service projects, helping with public events, and working as a volunteer in various agencies.

Financial incentives were built into the project design. Staff and agencies received payments based on student participation numbers. Participants themselves received small hourly stipends ($1 increasing to $1.33) for participation, a $100 bonus, and $100 for college or training funds after completing 100 hours of programming.

Evaluation Design and Results

Participants were randomly selected from lists of 8th grade students from families receiving public assistance. The students were then recruited and randomly assigned to the experimental or control group. At the end of 9th grade, test scores declined for the intervention group and there were no significant differences in education expectations between the intervention and the control group. After two years—the end of 10th grade—academic and functional skill scores increased and exceeded control group participants in five areas. At the end of senior year, performance in all 11 academic and functional skill areas was greater for the intervention group than the control group. About 6 months after scheduled high school graduation, the intervention group showed benefits from the program in several areas.

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Strengths and Limitations

Like the Teen Outreach Program evaluation, this evaluation was excellent in many ways. However, it also had limitations. First, a substantive theory of change was not explicitly stated and mediating causal processes were not investigated. Nonetheless, the program was designed with some attention to theory and explicit cause and effect relations: for example, it emphasized the importance of community through the service component; it taught responsibility through the program requirements; it provided opportunity through the rich set of offerings; it taught investment skills through earned stipends and college savings; it stressed the importance of meaningful adult relationships throughout the high school years; and finally, it recognized the importance of continuity across time by creating a four-year program. Each of these program design components is compatible with the committee’s list of features of positive developmental settings.

The multiple components of the program produced multiple positive effects using a strong pre- and postexperimental design. Youth were not only randomly assigned to the program intervention and control groups, but they were also randomly selected prior to program recruitment. The positive effects of this study are particularly impressive considering the frequent difficulties in recruitment and retention of youth ages 15 to 18. The program was well implemented according to evidence of participation and retention rates, although there was no specific information regarding the quality of program activities. The theory behind the program was not well articulated in the evaluation design. Consequently, most of the goals proposed in the program design phase were not evaluated.

In summary, this program seems very promising and is grounded in research on the importance of service learning and mentoring. In general, the program effects look positive; however, there is no explicit theory of change. Consequently, we know little about why and for whom it worked. Like the Teen Outreach Program, the program offers a service-learning component for young people having difficulties in school. The program also has a mentoring component that the staff thinks is critical for positive results. The evaluation study was relatively small (100 youth from 4 cities in the intervention). Finally, the program is quite expensive. The extent to which the payments to the youth are essential has not been assessed.

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

Big Brothers Big Sisters

Evaluators:

Grossman and Tierney, 1998

Level of intervention:

Community and individual

Target population:

10- to 16-year-olds

Evaluation research design:

Experimental design with random selection and assignment

Sample:

1,138 (571 treatment, 567 control)

Attrition analysis:

487 of the 571 treatment participants; 472 of the 567 control participants

Cost:

Average of $1,543/match/year

Big Brothers Big Sisters is a community-based mentoring program that matches an adult volunteer, known as a Big Brother or a Big Sister, to a child, known as a Little Brother or Little Sister, with the expectation that a caring and supportive relationship will develop. The match is well supported by mentor training and ongoing supervision and monitoring of the mentor relationship by a professional staff member. Ideally, the matched pair spends three to five hours per week together over the course of a year or longer. The activity goals are identified in the initial interview held with the parent or guardian and the child. The top priority is for the matched pair to develop a relationship that is mutually satisfying and provides contact on a regular basis. The goals established for a specific match are developed into an individualized case plan, which is updated by the case manager as progress is made and circumstances change over time. The case manager is responsible for maintaining contact with all parties in the match relationship.

In order to ensure effective matches between volunteers and youth and to monitor program quality, the professional staff screens all applicants, youth, and their families. Orientations are conducted with youth and volunteers, and training is provided for volunteers. Staff supervises matches between youth and volunteers by contacting all parties within two weeks of the initial match and then having monthly telephone contact with the volunteer for the first year of the program. The staff also contacts the youth directly at least four times in the first year.

Evaluation Design and Results

An experimental design using random assignment was used to evaluate the Big Brothers Big Sisters program at eight program sites: Phoenix,

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

Wichita, Minneapolis, Rochester, Columbus, Philadelphia, Houston, and San Antonio. Youth ages 10 to 16 who came to the agencies during the study period, October 1991 to February 1993, were randomly assigned to either the mentoring or the control group. Baseline data were collected, treatment subjects were matched with a mentor when possible, and control subjects were placed on a waiting list for 18 months. Pre-and posttreatment data consisted of self-assessment questionnaires on family background information and outcome variables. The evaluation sample consisted of 959 10- to 16-year-olds (487 treatment, 472 controls) from single-parent households. At pretest, 69 percent were between the ages of 11 and 13; 43.3 percent lived in households receiving public assistance; 27.1 percent were experiencing physical, emotional, or sexual abuse; 23 percent were minority girls, 15 percent were white girls, 34 percent were minority boys, and 28 percent were white boys.

Matches were found for 378 (78 percent) of the 487 members of the treatment group; 109 of the treatment group were not matched. The evaluators suggested three main reasons for the failure to match the 109 treatment youth: (1) 33 became ineligible for the program because their parent remarried, the youth became too old, or the residence of the youth changed; (2) 31 because the youth no longer wanted to participate; and (3) 21 because the staff couldn’t find a suitable volunteer mentor. The other 24 were not matched for a variety of reasons such as parent or youth not following through on the intake process.

Youth in the treatment group (including both those who received a mentor and those who did not) were less likely to initiate drug and alcohol use than the waiting list youth; they also reported hitting others less often. Their grade point average was higher, and they attended school more often. Finally they also reported better parental relationships and more parental trust.

Strengths and Limitations

The Big Brothers Big Sisters program is a very promising program based on the multiple benefits to youth of being mentored by nonfamilial adults. The random assignment evaluation design meets high standards. One major weakness, however, is the total reliance on self-reporting by the youth for the outcome measures. Another is the failure to include implementation or process analysis. As a result, we really know little about why the average-level effects are obtained and why the program may have worked very well for some youth and not at all for others. A

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

good qualitative evaluation component would have provided preliminary answers to these questions, and such an evaluation is now under way. A third weakness is the failure to gather follow-up data. Thus we know nothing about the length of positive impact. Finally, because Big Brothers Big Sisters does not have an explicit theory of change—the analysis of implementation quality does not go beyond descriptions of the number of contacts between mentor and young person and beyond their reports of satisfaction with the relationship. And with regard to this measure, many pairs did not meet as often as recommended by the program. A much more detailed study of implementation is now under way.

Three other aspects of the evaluation need to be noted. First, the treatment group included some youth who were assigned to receive the program intervention but were never matched; thus the analyses may underestimate the effect size. Second, outcome effects were reported using relative rather than absolute percentages, thereby possibly inflating effect sizes.

In summary, Big Brothers Big Sisters is a very promising community program for youth. The program is grounded in the research showing that positive relationships with nonfamilial adults support positive development. The researchers included all youth assigned to the treatment when they estimated the effects of the program. This procedure is likely to underestimate the impact of the program on those who actually received the services. Nonetheless, the program yielded significant positive benefits on multiple fronts. The evaluation was exemplary in some ways and limited in others. A strong methodological design with random assignment was used to evaluate the program. However, one could be more confident about the results if there were long-term follow-up data, the analysis included site-level as well as individual-level information, and more was known about the characteristics of each of the eight sites.

CONCLUSIONS ABOUT PROGRAM FEATURES

The committee was very interested in understanding why programs are effective and particularly how and why they are effective in promoting adolescent development. The quality of information in most studies about the particular features of effectiveness, however, was minimal. Furthermore, these studies were not necessarily designed to collect information about those aspects of adolescent development outlined in this report. However, we read the descriptions of these studies carefully for

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

clues as to why they might have worked and how they were related to the elements of the development framework we hypothesized as being important at the beginning of this report. Each program, in fact, included several components that were consistent with this framework. We learned that many programs can effectively promote healthy development, although we learned much less about why. Without such information, we cannot say very much about which particular components or combination of components were responsible for the programs’ effectiveness. We therefore cannot draw firm conclusions about which and how many of the features of settings in Chapter 4 were critical for the success of these programs. Nevertheless, in this section we attempt to compare these programs with our framework.

It is important to keep in mind that we imposed our list of setting features onto programs that were not designed with this particular framework in mind; we therefore had to rely on our interpretation of the program descriptions and other information provided in the evaluation materials to infer whether a program offered a particular feature or not.

Supportive Relationships

Supportive relationships were a major component of most of the effective programs reviewed. Explicit mentoring activities are the most obvious examples of nurturing supportive relationships, but most of the programs provided youth with some form of regular supportive contact with nonfamilial adults. In many cases, mentoring was not explicitly stated as a program goal, but the adult-adolescent contact in these programs often took the form of mentoring. For example, Quantum Opportunities used an intensive case management approach to tailor their program to meet individual needs and circumstances. This approach required regular supportive interactions between the case manager and the adolescent. In addition, many of the programs officially designated mentoring as the central component of the program. The best example of this is Big Brothers Big Sisters. The positive effects of mentoring were clearly documented in the evaluation of this program. Follow-up was done to understand the factors contributing to supportive mentoring relationships. Through telephone interviews with 1,101 mentors in 98 mentoring programs and through youth focus groups and youth interview data (Herrera et al., 2000), the evaluators found that mentors with the closest and most supportive relationships reported more than 10

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

hours of contact per month, shared interests between the mentor and youth, and shared decision making about activities.

Opportunities to Belong

Healthy development is promoted by fostering a sense of belonging, an area which the Quantum Opportunities Program stressed. Instilling a sense of belonging was at the core of its mission as reflected in their motto, “Once in QOP, always in QOP.” If participants stopped attending activities and disappeared from the program, program staff tracked them down to find out what was wrong, to assure them they were still part of their QOP “family,” and to coax them back to the program. This attitude of refusing to give up on youth was considered by Hahn et al. (1994) to be a crucial component of the program. Although this outreach might have been seen as coercive, evidence from the student evaluations of the program suggest that this was not the case: for three of the four sites evaluated, 78–92 percent of the students thought the program was “very important” and 82–92 percent were “very satisfied” with the program. It is unlikely that students would give these responses if they felt coerced to attend or hassled by unwanted adult attention.

Positive Social Norms

Even when not explicitly stated as a program goal, all of these programs promoted positive social norms and discouraged norms related to the major problem behaviors of concern in these programs. Programs with a community service component—Quantum Opportunities and the Teen Outreach Program—best exemplify this feature. Youth participants were matched with an adult mentor and received classroom-based social problem-solving skills training and participated in community service activities with their mentors. The discussions in school were explicitly designed to support positive social norms among the students participating. In many cases, parents also participated in workshops, to enhance their ability to help their children maintain positive social norms and reject antisocial norms and pressures from peers to participate in problem behaviors. Programs that taught resistance skills also exemplify this feature.

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Support for Efficacy and Mattering

Providing opportunities for autonomy, taking responsibility, and challenge are likely to be especially important for older adolescents as they approach the transition to adulthood. Most of the programs reviewed in this chapter did not address opportunities for developing a sense of efficacy. More such programs were reviewed in Chapter 5. However, programs with a focus on skill acquisition, such as the Penn Prevention Program, do exemplify this characteristic because such learning opportunities, if implemented as discussed in Chapter 4, are the best settings in which to acquire a strong sense of personal efficacy. We talk more about this in the next section. Support for mattering was more clearly evident in the effective programs reviewed in this chapter. This feature was most evident in programs that provided opportunities to participate in community service. Two of the model programs, Quantum Opportunities and the Teen Outreach Program, were successful in involving high school students in community service, life and family skills training, and support in planning for college and jobs.

Opportunities for Skill Building

Opportunities for skill building were plentiful in the programs reviewed. An emphasis on social skills was a frequent program goal. Building skills in resisting peer influences to engage in a wide range of problem behaviors (such as unprotected sex, drinking and drug use, and illegal behaviors) was common in programs primarily focused on preventing such problems as drug and alcohol use, teen pregnancy, and HIV/AIDS (see program descriptions in Table 6–1 for ENABL, Life Skills Training, Project ALERT, Positive Youth Development Program). Also, many of the programs focusing specifically on the parent-youth relationship include activities to teach youth and parents better communication skills (e.g., the Adolescent Transitions Project, Creating Lasting Connections, Functional Family Therapy, Improving Social Awareness-Social Problem Solving, the Midwestern Prevention Project, the Social Competence Program for Young Adolescents, the Teen Outreach Program).

Other programs focus on individual cognitive, social, and emotional skill development. Some focus on such social and emotional skills as coping and self-regulation (Coping with Stress Course, Functional Family Therapy, the Penn Prevention Project). Others provided employment or economic management skills through part-time employment with

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

mentoring, encouragement for opening savings accounts, and financial incentives for participation (Quantum Opportunities, the Summer Training and Education Program, Louisiana State Youth Opportunities).

Integration of Family, School, and Community Efforts

Several programs excelled in integrating family, school, and community into their program design. The Midwestern Prevention Project includes a set of program activities to prevent adolescent drug use: (1) mass media programming with news clips, commercials, and talk show discussions on drug use incorporated with information on their program; (2) a school-based program teaching resistance and counteraction skills for drug use; and (3) parent education and organizing.

Project Northland also used a community-wide approach. Students received skills training to enhance their social competency in dealing with their parents, their peers, and the norms surrounding alcohol use. Parent education and involvement was also stressed through parent-student homework activities and through newsletters to parents containing educational information. Community-level changes in alcohol-related programs and policies were also targeted. Finally, the Valued Youth Partnership Program participants were given training in how to tutor and then engaged in tutoring of younger students for at least four hours per week. Parents were involved in school activities, and students were exposed to role models in the community through presentations and field trips.

Structure and Safety

These two characteristics of positive developmental settings from our list—appropriate structure and physical and psychological safety—received little mention in the program descriptions. Perhaps this omission reflects that fact that these characteristics are considered so basic that they are not worthy of explicit mention. One of the fundamental reasons for community programming for youth is to provide safe places for them to go. Wisdom would dictate then that programs must enable youth to feel and be psychologically and physically safe. To do this, programs need to have clear and consistent rules, expectations, and boundaries. Nevertheless, the absence of these features is likely to be a major reason why programs fail and why some programs produce negative results.

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

None of the programs included in this chapter dealt explicitly with the use of developmental appropriate levels of structure. Some of those reviewed in Chapter 5 did. These programs provided more opportunities for self-monitoring and active participation in rule making and enforcement as the youth matured.

SUMMARY

Our review of experimental and quasi-experimental evaluations of community programs for youth ages 10 to 18 leads to several general conclusions about these programs and the evaluation of these programs.

Community programs for youth, whether they are packaged in teen pregnancy prevention programs, mental health programs, or youth development programs, can facilitate positive outcomes for youth. This review of experimental and quasi-experimental evaluation of community programs for youth revealed that participation is associated with increases in such outcomes as motivation, academic performance, self-esteem, problem-solving abilities, positive health decisions, interpersonal skills, and parent-child relationships, as well as decreases in alcohol and tobacco use, depressive symptoms, weapon-carrying, and violent behavior. Although many of these studies were not designed around a positive youth development framework, many of the evaluations in fact included measures that reflected the personal and social assets and features of settings developed in this report.

Even with the most rigorous methods of evaluations, there is limited evidence that measures the impact of these experiences on the development of young people and therefore limited evidence on why program effects are or are not obtained from the evaluations reviewed. Some of these studies explicitly involved components consistent with the developmental framework outlined in Part I. However, without appropriate information, we cannot say very much about which particular components or combination of components were responsible for a specific program’s effectiveness. Consequently, research is needed to sharpen the conceptualization of features of community programs and to explore whether other key features should be added to the list. This work should focus on how different populations are affected by different program components and features (e.g., age, gender, socioeconomic status, ethnicity, community environment, developmental readiness, personality, sexual orientation, skill levels). It should also focus on how to incorporate these features into community programs and on how to maintain

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×

them once they are in place. Finally, such research should identify program strategies, resource needs, and approaches to staff training and retention that can cultivate and support the features of positive developmental settings in community programs for youth.

Also, it is our view that evaluations designed with some of the principles outlined in the next chapter will tell us much more about why particular programs work and for whom they are likely to be most effective. And new programs can be designed with some of the principles outlined in earlier chapters. If such programs are designed from the beginning with a well-articulated theory of change grounded in solid developmental theory and research, we will learn a lot more about what programs might do to facilitate positive youth development and to prevent problems from emerging,

Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
This page in the original is blank.
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 147
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 148
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 149
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 150
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 151
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 152
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 153
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 154
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 155
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 156
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 157
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 158
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 159
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 160
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 161
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 162
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 163
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 164
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 165
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 166
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 167
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 168
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 169
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 170
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 171
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 172
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 173
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 174
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 175
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 176
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 177
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 178
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 179
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 180
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 181
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 182
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 183
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 184
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 185
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 186
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 187
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 188
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 189
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 190
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 191
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 192
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 193
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 194
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 195
Suggested Citation:"6 Lessons from Experimental Evaluations." National Research Council and Institute of Medicine. 2002. Community Programs to Promote Youth Development. Washington, DC: The National Academies Press. doi: 10.17226/10022.
×
Page 196
Next: 7 Generating New Information »
Community Programs to Promote Youth Development Get This Book
×
Buy Paperback | $54.95 Buy Ebook | $43.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

After-school programs, scout groups, community service activities, religious youth groups, and other community-based activities have long been thought to play a key role in the lives of adolescents. But what do we know about the role of such programs for today's adolescents? How can we ensure that programs are designed to successfully meet young people's developmental needs and help them become healthy, happy, and productive adults?

Community Programs to Promote Youth Development explores these questions, focusing on essential elements of adolescent well-being and healthy development. It offers recommendations for policy, practice, and research to ensure that programs are well designed to meet young people's developmental needs.

The book also discusses the features of programs that can contribute to a successful transition from adolescence to adulthood. It examines what we know about the current landscape of youth development programs for America's youth, as well as how these programs are meeting their diverse needs.

Recognizing the importance of adolescence as a period of transition to adulthood, Community Programs to Promote Youth Development offers authoritative guidance to policy makers, practitioners, researchers, and other key stakeholders on the role of youth development programs to promote the healthy development and well-being of the nation's youth.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!