tion with outcome evaluations. Efforts to address these challenges deserve further attention in the evaluation research community.
The study of welfare reform and the evaluation of its effects presents many challenges. Examining the effect of complex bundles of individual reform programs, determining the influence of the composition of the welfare caseload on measured outcomes, developing a credible comparison group for those affected by welfare reform, and constructing an adequate database for measuring outcomes, as well as data describing policies across states, require thoughtful study designs as well as considerable resources.
We conclude that while nonexperimental methodologies for evaluation have become the dominant method of evaluation at the current time, experimental methodologies still have a role to play and should be kept on the table as one means of evaluation. We conclude that monitoring and descriptive studies of welfare reform are important, but that evaluation studies—which estimate the effect of a program reform—should be the ultimate goal of welfare reform research. We emphasize that there is a role for both national-level welfare reform evaluation, which yields a comprehensive assessment of the effects of reform in all the states around the country, and for purely state-level studies, which yield estimates for individual states.
Regarding data, the panel has found considerable weaknesses in the three elements of data infrastructure needed to evaluate welfare reform. Household survey data sets, which are rare at the state level, are more plentiful at the national level but suffer from small sample sizes, a lack of key variables, and the relative unavailability of comparable policy measures across states. State-level administrative data sets, which have traditionally been used for management rather than research purposes, are still at an early stage of development and need much more work before they can fulfill their potential. Comprehensive data on state welfare policies across states and over time on a comparable basis have yet to be published, and there is no systematic plan for collecting such data on a long-run, permanent basis within the federal government.