tion that will help them improve what is going on at their site. Case studies are probably a better source of information than having no information about change and its causes, and program personnel want any edge they can get to improve what they do. Leavened by knowledge from the existing relevant literature, the data collected on site, and the staff’s other sources of program knowledge, the results of a case study can help local staff improve what they do.

However, when the purpose of the evaluation is to estimate the effectiveness of a program in helping participants, we have doubts about the utility of case studies. There are three reasons for this. First, it is often difficult to assess how much each participant has changed from their entry into the program to later. Second, and perhaps most importantly, there is no counterfactual against which to assess how much the participants would have changed had they not been in the program. And third, conclusions about effectiveness always involve a high degree of generalization—e.g., across persons, service practitioners, program inputs—and many qualitative researchers are reluctant to make such generalizations. They prefer to detail many known factors that make some difference to an outcome, and this is not the same as noting what average effect the program has or how its effectiveness varies by just a few carefully chosen factors.

Identifying the Most Appropriate Method

Experimental evaluation methods are often considered the gold standard of program evaluation and are recommended by many social scientists as the best method for assessing whether a program influences developmental outcomes in young people. However, many question the feasibility, cost, and time intensiveness of such methods for evaluating community programs for youth. In order to generate new, important information about community programs for youth, the committee recommends that the kind of comprehensive experimental evaluation discussed in this chapter be used under certain circumstances (see also footnote 1):

  1. The object of study is a program component that repeatedly occurs across many of the organizations currently providing community services to youth;

  2. An established national organization provides the program being evaluated through a system of local affiliates; and



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement