Strengths and Limitations

Like the Teen Outreach Program evaluation, this evaluation was excellent in many ways. However, it also had limitations. First, a substantive theory of change was not explicitly stated and mediating causal processes were not investigated. Nonetheless, the program was designed with some attention to theory and explicit cause and effect relations: for example, it emphasized the importance of community through the service component; it taught responsibility through the program requirements; it provided opportunity through the rich set of offerings; it taught investment skills through earned stipends and college savings; it stressed the importance of meaningful adult relationships throughout the high school years; and finally, it recognized the importance of continuity across time by creating a four-year program. Each of these program design components is compatible with the committee’s list of features of positive developmental settings.

The multiple components of the program produced multiple positive effects using a strong pre- and postexperimental design. Youth were not only randomly assigned to the program intervention and control groups, but they were also randomly selected prior to program recruitment. The positive effects of this study are particularly impressive considering the frequent difficulties in recruitment and retention of youth ages 15 to 18. The program was well implemented according to evidence of participation and retention rates, although there was no specific information regarding the quality of program activities. The theory behind the program was not well articulated in the evaluation design. Consequently, most of the goals proposed in the program design phase were not evaluated.

In summary, this program seems very promising and is grounded in research on the importance of service learning and mentoring. In general, the program effects look positive; however, there is no explicit theory of change. Consequently, we know little about why and for whom it worked. Like the Teen Outreach Program, the program offers a service-learning component for young people having difficulties in school. The program also has a mentoring component that the staff thinks is critical for positive results. The evaluation study was relatively small (100 youth from 4 cities in the intervention). Finally, the program is quite expensive. The extent to which the payments to the youth are essential has not been assessed.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement