goals, processes, and impacts of projects, policies, and programs. Such “systematic investigations of the worth or merits” of a program (Joint Committee on Standards for Educational Evaluation, 1994) often pose questions like these (Rossi, Lipsey, and Freeman, 2004): What is the nature and scope of the problem? Is the particular intervention reaching its target population? Is the intervention being implemented well? Is the intervention effective in attaining the desired goals or benefits? Does the program have important unanticipated consequences? Are the program costs reasonable in relation to its effectiveness and benefits? The evaluation plan is typically organized around the questions posed by those who commissioned the evaluation, but it also should be responsive to the needs of other stakeholders (Standard U3). Program evaluations are expected to address the issues that matter, collect information that is relevant and meaningful for the goals of the evaluation, analyze the information using rigorous and fair methods, and communicate the results in a form that is usable and meaningful to decision makers.
There are two major types of evaluations: (1) those designed to distinguish worthwhile programs from ineffective ones, and (2) those designed to help improve existing ones in order to achieve certain desirable results. The former are often called formative evaluations, and they are conducted to provide information on how a program should be delivered or to furnish information for guiding program improvement (Scriven, 1991). The latter are called summative evaluations, and they are conducted to determine whether a program’s expectations are being met and what its consequences are (Scriven, 1991). This is the kind of evaluation that our charge required.
Summative evaluations generally focus on whether a given program (e.g., a social program, an educational intervention) is effective. For example, summative evaluations might study such issues as the program’s accomplishment of its intended objectives, impacts beyond those that were intended, how effectively resources have been used, the benefits of the program and what it costs to produce these benefits, and alternative interventions that might produce similar benefits. Summative program evaluations usually focus on the effects of a program on outcomes for a client population and consider the extent to which the program changes the outcomes for participants.
For example, the United States has a long history of commissioning evaluations of government-sponsored employment training programs designed to help unemployed workers or workers with relatively few skills find employment. Such evaluations attempt to infer the causal impact of enrolling in the program on the outcomes of interest for the participant, such as the probability of obtaining a job or the level of wages earned.
A particular challenge with this kind of an evaluation lies in trying to determine whether a change in outcomes for participants is in fact attribut-