than the effects of reform on government itself. One view of the purpose of the welfare reform legislation is that it was intended to change the nature of how government delivers assistance to the poor, away from a purely eligibility-oriented and check-writing function to a function of encouraging work, promoting self-sufficiency, and providing the right signals and incentives for those to occur. As Nathan and Gais (1999) have described, the reform is resulting in a major change in welfare bureaucracies. Although this is a legitimate issue, evaluating the effects of PRWORA on governments themselves requires different evaluation methods than the methods discussed in this report (although process studies, which we do discuss, are one component of such evaluations).
We organize our discussion of the general principles of impact evaluation in terms of four general issues that any impact study must address; we pose each in the form of a question:
What are the research and policy questions of interest, and what are the precise objectives of the study?
What are the study populations of interest, and what are the outcomes of interest on those populations?
What evaluation methodologies are appropriate for achieving the goals of the study?
What data sources are available to the study and how can they be used?
Having a solid understanding of these issues is not only important for the design of new welfare reform studies, but also for interpreting the results of those studies that are currently under way and will be issuing findings over the next few years. As Chapter 3 details, the current studies differ, often on critical dimensions, in the way in which each of the four issues listed above is addressed. Some answer different questions, many study different populations, they often use different methodologies, and they frequently use very different data. Melding the results of such a diverse set of studies into a single coherent picture of the effects of the latest wave of welfare reform is a challenge that requires a clear understanding of the issues that we discuss in this chapter.
Broadly speaking, the question of interest in all welfare reform studies is the effect of reform on adults and children. The types of reforms that are of interest and the geographic level at which these effects are assessed are major issues in the research community. One key distinction, for example, is whether interest centers on the effect of an entire "bundle" of reforms—that is, a package containing provisions for work requirements, sanctions, time limits, a particular set of support services, and other features—or whether one is interested in the effects of each component separately, holding the others fixed. Most welfare reforms that