good qualitative evaluation component would have provided preliminary answers to these questions, and such an evaluation is now under way. A third weakness is the failure to gather follow-up data. Thus we know nothing about the length of positive impact. Finally, because Big Brothers Big Sisters does not have an explicit theory of change—the analysis of implementation quality does not go beyond descriptions of the number of contacts between mentor and young person and beyond their reports of satisfaction with the relationship. And with regard to this measure, many pairs did not meet as often as recommended by the program. A much more detailed study of implementation is now under way.

Three other aspects of the evaluation need to be noted. First, the treatment group included some youth who were assigned to receive the program intervention but were never matched; thus the analyses may underestimate the effect size. Second, outcome effects were reported using relative rather than absolute percentages, thereby possibly inflating effect sizes.

In summary, Big Brothers Big Sisters is a very promising community program for youth. The program is grounded in the research showing that positive relationships with nonfamilial adults support positive development. The researchers included all youth assigned to the treatment when they estimated the effects of the program. This procedure is likely to underestimate the impact of the program on those who actually received the services. Nonetheless, the program yielded significant positive benefits on multiple fronts. The evaluation was exemplary in some ways and limited in others. A strong methodological design with random assignment was used to evaluate the program. However, one could be more confident about the results if there were long-term follow-up data, the analysis included site-level as well as individual-level information, and more was known about the characteristics of each of the eight sites.

CONCLUSIONS ABOUT PROGRAM FEATURES

The committee was very interested in understanding why programs are effective and particularly how and why they are effective in promoting adolescent development. The quality of information in most studies about the particular features of effectiveness, however, was minimal. Furthermore, these studies were not necessarily designed to collect information about those aspects of adolescent development outlined in this report. However, we read the descriptions of these studies carefully for



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement