Chapter 2 and above—as well as controlling for other state differences that might confound the effects of welfare policy.
To a great extent, the importance of these issues will be clear only when the results of the studies are available. If the diverse populations, cities, and evaluation designs yield a reasonably coherent and consistent set of findings that add up to a credible overall picture of the consequences of welfare reform, the issues and potential difficulties we have noted will be of only modest importance. In this case, the great diversity of the studies will work to the advantage of the overall national evaluation effort by demonstrating the robustness and consistency of estimates across a wide range of approaches and across different populations for study. If, on the other hand, the studies yield results that vary greatly and in ways that are not easily explainable by obvious differences, a mixed message will result and some assessment of the relative reliability of the different approaches will have to be undertaken. This could be a formidable challenge.
The state grantee studies funded by ASPE have made a good start in what is expected to be a longer term effort to assess the effects of welfare reform policies on the low-income population. To a great extent, the grantee studies should be viewed as initial investments in building data capacity at the state level, leading to improved administrative and survey data. However, much remains to be done for these data sets to fulfill their potential. As monitoring studies, the ASPE grantee designs have significant interest and should yield interesting findings on the outcomes of welfare leavers. While the cross-state comparability in these 14 studies will be much greater than among the welfare leaver studies conducted previously, thus strengthening their findings as a whole, many improvements in such comparability need to be made. The grantee designs also have thus far not proposed to identify the most important subgroups in the welfare recipient population, namely, long-term and short-term recipients and those with strong and weak work histories. As evaluation studies, the ASPE grantee designs face considerable challenges in demonstrating the credibility of a recipient-based cohort comparison design. Finally, the grantee designs thus far concentrate almost exclusively on welfare leavers; they do not include an examination of nonapplicants and divertees or of families who are still on the rolls (stayers).
The other welfare leavers studies that have been completed around the country have the same strengths and weaknesses, and the ASPE grantee studies promise to be of higher general quality. There are many other welfare reform projects under way around the country as well, both within and outside of government, and these will, when completed, yield a large number of assessments of welfare reform using different approaches and different data. Most of these projects also face significant evaluation challenges, which we have identified. Whether the set of findings that will emerge from these diverse studies will yield a consistent and coherent picture of the effects of welfare reform remains to be seen.