Data and Methodological Issues for Tracking Former Welfare Recipients: A Workshop Summary (1999)

of research. The panel is also specifically charged with reviewing data needs and methods for tracking and assessing the effects of program changes on families who stop receiving cash assistance and on families who are diverted or fail to apply for assistance.

To fulfill that component of the charge, the Panel on Data and Methods for Measuring the Effects of Changes in Social Welfare Programs convened a workshop to learn about the plans of the 14 jurisdictions that received grants from ASPE to track those who leave welfare. The November 13, 1998, workshop included representatives from the 14 jurisdictions, ASPE staff, staff from other government agencies with programmatic relations to social welfare policies, and researchers from academia and private research organizations involved in welfare reform research. ³ This report summarizes the workshop discussions and supplements the panel's interim report (National Research Council, 1999), which focuses on broader data and methodological issues for evaluating welfare reform and presents the panel's findings, conclusions, and recommendations to date. The panel's final report will completed late 2000.

The goal of the November workshop was to review state plans and discuss some general issues regarding the studies. The discussion of the workshop followed the outline of the general principles of impact evaluations, in the form of questions:

• What are the research and policy questions of interest, and what are the precise objectives of the study?

• What are the study populations of interest, and what are the outcomes of interest on those populations?

• What evaluation methodologies are appropriate for achieving the goals of the study?

• What data sources are available to the study and how can they be used?

This summary of the workshop discussion follows the outline of these four principles. First, however, because a salient discussion on the purpose and use of the 14 grants developed during the workshop, we begin with a brief summary of that discussion.

Recipients of ASPE grants to follow welfare leavers were asked to discuss how these leaver grants fit into their overall research agendas. Many state researchers and other workshop participants suggested that a key function of the leavers studies is to build the capacity of states and counties to perform further research. With the devolution of program responsibility to the state and sometimes to the county level, many research and evaluation efforts of these programs have also shifted to the

state level. Because policies can vary greatly across states, state-specific knowledge of the policies, program implementations, and data collection are now more of a necessity for conducting evaluations.

State-level administrators and researchers are in a good position to conduct these evaluations because of their knowledge of state administrative data, policies, and policy implementations. State and county representatives are interested in developing the capacity to collect, store, and use administrative data sets. This includes not only those data sets that contain information on TANF recipients and other public assistance recipients (food stamps and Medicaid, for example), but also administrative data sets on employment, child support, child welfare, Supplemental Security Income (SSI), unemployment insurance (UI), tax revenues, and others. The goal of most state projects is to link all of these administrative data bases to provide a richer set of measures on individuals. The ability to link data sets is one capacity-building aspect of the grants.

Another capacity-building role of the grants is to help states develop the ability to conduct surveys. A few states are conducting, or have already conducted, their own surveys (South Carolina, Washington, and Wisconsin), and others are beginning to develop their abilities to conduct surveys. Some of the states and counties are contracting with private firms, universities, or other government agencies to conduct the surveys. For these jurisdictions, the capacity-building exercise will be developing contacts with survey researchers.

Another purpose of the grants discussed at the workshop was to provide descriptive data on the status and well-being of the former recipients who have left welfare. Many state and county researchers said that policy makers and the general public in their jurisdictions are interested in the status of those no longer receiving welfare. Are they finding employment? What kinds of nongovernment supports are they finding? Are they returning to welfare? These are some of the questions that state researchers hope to be able to answer with respect to welfare leavers.

The grantees also said they think that the results of the studies would be used to make cross-state comparisons. Many representatives from states agreed that policymakers will be less interested in comparing how recipients were doing before and after TANF, but more interested in comparing results across states. Some state grantees were hesitant to compare results across states because the policies enacted in the states are quite different, the macroeconomic conditions vary across states, and the caseload dynamics differ across states. Some participants also urged caution in making cross-state comparisons because definitions of outcomes may not be comparable across states. For example, a former recipient may be counted as employed in one state if the individual is working at least 20 hours a week, while in another state, “employed” may mean the individual is working any number of hours per week; outcomes derived from these measures are not comparable.

Many participants agreed that one very positive contribution these grants could make is to fully characterize the actual policies of the states and counties. State- and county-level researchers are often on the front line of the administration of policies: this puts them in a unique position to be able to inform other researchers and administrators at all levels of government about the treatments that welfare recipients are actually receiving, in other words, what policies were enacted and how the policies are actually being implemented.

Another purpose of the grants emphasized at the workshop is to monitor caseloads over time. Program development could be improved by knowing the kinds of services former recipients need in order to stay employed and to stay off of welfare. The monitoring function could also detect possible fluctuations in the caseload due to changes in the economy. Knowing how many people