Evaluating Welfare Reform: A Framework and Review of Current Work, Interim Report (1999)

The Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) of 1996 fundamentally changed the nation's social welfare system, replacing a federal entitlement program for low-income families, called Aid to Families with Dependent Children (AFDC), with state-administered block grants, the Temporary Assistance for Needy Families (TANF) program. PRWORA furthered a trend started earlier in the decade under so called "waiver" programs—state experiments with different types of AFDC rules—toward devolution of design and control of social welfare programs from the federal government to the states. The legislation imposed several new, major requirements on state use of federal welfare funds but otherwise freed states to reconfigure their programs as they want. The underlying goal of the legislation is to decrease dependence on welfare and increase the self-sufficiency of poor families in the United States.

In summer 1998, the Office of the Assistant Secretary for Planning and Evaluation (ASPE) of the Department of Health and Human Services (DHHS) asked the Committee on National Statistics of the National Research Council to convene a Panel on Data and Methods for Measuring the Effects of Changes in Social Welfare Programs. The panel's overall charge is to study and make recommendations on the best strategies for evaluating the effects of PRWORA and other welfare reforms and to make recommendations on data needs for conducting useful evaluations.

Under the broad charge, the panel is considering many evaluation issues, such as the proper mix of national-level and state-level evaluations; the appropriate roles of experimental and nonexperimental evaluation methodologies; and the importance of monitoring versus evaluation. The panel is also considering the

needs for data from surveys and administrative records at both the federal and state levels, as well as general issues of data quality for studies of the low-income and welfare populations. The panel began its work in fall 1998 and will issue a final report in late 2000.

This interim report presents the panel's initial conclusions and recommendations. Given the short length of time the panel has been in existence, this report necessarily treats many issues in much less depth than they will be treated in the final report. The report has an immediate short-run goal of providing DHHS-ASPE with recommendations regarding some of its current projects, particularly those recently funded to study ''welfare leavers''—former welfare recipients who have left the welfare rolls as part of the recent decline in welfare caseloads.

Many of the conclusions reached by the panel in its initial examination of welfare reform evaluations under way around the country concern the data bases used for evaluation. The devolution of program responsibility begun under the pre-PRWORA waivers and institutionalized by PRWORA has led to wide variation in programs across states and even within states. While this proliferation of programs has some advantages from an evaluation standpoint—it makes use of federalism as a laboratory for testing (as it has been so often used in American history)—it imposes the need for a significant data infrastructure. Such an infrastructure should include national-level data sets that can capture state variations in policies and outcomes; state-level data sets that are sufficiently detailed and standardized to permit comparisons of welfare policies as implemented and family and individual outcomes (employment, income, etc); and state-level data sets for evaluating the effects of welfare reform in each state, even if not intended for comparison with other states. The panel has concluded that such a data infrastructure is not in place at the current time.

Gaps in the data infrastructure for determining the effects of welfare reform are numerous. The new welfare reform environment imposes great strain on traditional national-level survey data sets, both those supported by the Census Bureau as well as those supported by other federal agencies. These data sets usually have relatively small sample sizes for the welfare-relevant population, and they often do not have all the major variables needed to comprehensively assess the effects of welfare reform. The Survey of Program Dynamics, a national-level panel data set intended to remedy some of these defects, has so far not been as successful as expected. A further difficulty straining the usefulness of national-level surveys is the relative lack of information on the policies that states have actually adopted since 1996. The 1996 legislation not only devolved operational and design responsibility to the states, it also removed many requirements for state reporting to the federal government on their policies. Federal data-gathering activities on the details of state policies have been slow and hap-

hazard, and it is only since early 1999 that an institutional structure has begun to be put in place for collecting such data on a permanent and long-run basis.

Program devolution has also constrained the ability of the federal government to mandate data collection or research to track the effects of the changes in social welfare programs required by the new legislation. Yet it is clear that the federal government and Congress will need a national-level assessment of how the new programs are working, especially to make informed decisions about the renewal of PRWORA. In addition, a national-level assessment containing results from all the states should be of interest to state policy makers, who are interested in how other state programs are working in comparison with their own and who are interested in what other states' efforts have been successful.

The new welfare reform environment also imposes burdens on state-level data sets. Most state-level data sets are obtained from administrative records that have historically been used for management, not evaluation, purposes. These data sets are being put under strain as states seek to develop and use them in more sophisticated ways in order to meet the legal requirements—for example, to keep counts of time limits as well as to satisfy federal reporting requirements on counts of recipients in different categories—and to conduct evaluations of their own state's policies. Considerable effort is required to convert these databases for evaluation and research uses, for which they were not originally intended. The use of these state-level databases for cross-state comparison research or, more ambitiously, as a substitute for national-level data sets, is further hindered by noncomparable variables and data constructs across states and by the same lack of comparable policy information that is hindering national-level surveys. State-level household surveys have been only rarely conducted in the past, but are now in great demand, and states are hurrying to build the capacity to conduct them.

The panel also reached some initial conclusions about evaluation methodology. Both national-level and state-level evaluations have roles to play in welfare reform evaluation, and neither should be pursued to the exclusion of the other. Moreover, cross-state comparability in state-level data sets and the development of comparable measures of state policies are important for both national-level and state-level evaluations. Although nonexperimental methods have become the dominant evaluation method for PRWORA, the panel believes that experimental methods should be kept on the table and still have a role to play in the future.

The panel also concludes that monitoring and evaluation both have great value, but that the distinction between the two has not always been sufficiently delineated and that evaluation should be the ultimate goal. Relatedly, the panel concludes that many existing studies do not have credible groups for which policy comparisons can be made and hence are of limited value as evaluation studies. Other studies use comparison groups that will require great care in reaching sound conclusions. Specifically, the panel concludes that welfare leaver studies have great value but represent only one group potentially affected by

welfare reform; they miss those who were diverted or discouraged and even those families remaining on the welfare rolls.

Although the panel will be issuing a broad set of recommendations to address the many issues of methods and data for welfare reform evaluation in its final report, there are several immediate, short-run recommendations it wishes to make to DHHS-ASPE concerning some of its current sponsored activities. In particular, for the purposes of cross-state comparisons and national-level assessments, there are benefits, including economies of scale, for DHHS to take a leading role in obtaining data and guiding research. Several key areas in which guidance in data collection and research is needed, for which the department already has some initiatives under way, include: (1) identifying key policy issues over the next 3-5 years that should guide priorities for data collection and research; (2) taking steps to ensure that research addresses key populations of interest for social welfare policy analysis; (3) fostering improved capabilities for social welfare program-related data collection and research at the federal and state levels; (4) encouraging efforts to make data and research results comparable across jurisdictions; and (5) documenting state policies.

Because resources for data collection and research on social welfare programs are limited across all levels of government, it is important to set priorities so that key policy questions can be answered. While not all states will have the same specific policy concerns, there is likely a core set of issues that will be the focus of federal and state policy making over the next 3–5 years and on which it makes sense to concentrate scarce resources for data collection and research.

The department can take steps toward identifying critically important issues by maintaining an information-gathering operation that obtains input from a variety of sources, such as the Congress and other relevant federal departments (Agriculture, Housing and Urban Development, and Labor), state welfare agency staff, and state legislators and their staffs. Maintaining close relationships with interstate coordinating mechanisms that the states have already established would also be useful. The department already has many of these kinds of ties.

Much attention has been given to the effects of changes in social welfare programs on those who leave welfare. While leavers are an important group to track, policy decisions—for example, the decision to renew or modify PRWORA—will not be adequately informed if other key populations are not analyzed as well. Many states have active diversion programs that discourage potential TANF recipients from enrolling, and many families who are eligible to receive benefits may not apply for benefits. "Stayers," those who remain on the rolls, should also not be neglected in a study of the impact of welfare reform. In order to understand the extent to which PRWORA and other social welfare program changes are achieving their goals, it is important to look at these other key groups among the low-income population.

Features of the new program environment have made it difficult to clearly define such key population groups as leavers, divertees, and eligible non-applicants. The devolution of program responsibility to state and substate jurisdictions has led to different eligibility provisions across jurisdictions. Moreover, "assistance" has been redefined to include not only cash, but also a variety of noncash benefits and services. This feature, in combination with the blurring of the lines of responsibility among program agencies (e.g., "welfare" caseworkers may now serve as brokers to a variety of services for clients and often are employment and training counselors for clients) has made it difficult to determine when someone has ''left" welfare or has been diverted from the program. The increase in child-only cases, some of which are the result of partial family sanctions, presents a similar definitional challenge.

DHHS can make a major contribution to improving the analytical rigor and cross-area comparability of data and research on the effects of changes in social welfare programs by addressing the definition of key population groups and proposing standard definitions for use in research and data collection and ensuring that grant and contract research programs adequately cover key groups. The department has already taken steps in this direction. The next round of ASPE grants for leaver studies will include studies of divertees as well as leavers. ASPE is also planning a research program to study entry as well as exit effects.

DHHS should give priority to improving the capabilities for cost-efficient, policy-relevant data collection and research at both the federal and state levels. In doing so, the department should also encourage and facilitate cost-effective steps to make data and evaluations comparable across states so that all levels of government can make better use of scarce resources for data collection and research to evaluate changing programs.

We encourage the department to undertake capacity-building and cross-state comparability activities, such as: (1) facilitating efforts by states to form networks for exchanging information and technical assistance; (2) facilitating lower-cost survey development and more comparable data and analysis (e.g., by providing tested questionnaires for states to consider using in surveys); and (3) for future rounds of grants, consider hiring one or more contractors with research and survey expertise to serve as consultants to all grantees. To build capacity, the department can also encourage recipients of departmental grants and other state and local jurisdictions to invest in increasing staff skills for conducting surveys and analysis. To foster cross-area comparability of research analyses and results in subsequent rounds of leaver study grants, ASPE can facilitate the following actions: (1) collecting information on the educational level and employment history and experience of leavers to use as stratifiers in analyzing results; (2) collecting information on the welfare recipiency history of leavers to allow stratification of results by recipients' statuses as short-termers, cyclers, and long-termers, which can at least partially control for differences in outcomes across states that may result simply from differences in the mix of these types of recipients; (3) comparing outcomes and characteristics of leavers to stayers; and (4) including full descriptions of the welfare programs and economic environments in place over the life of the cohorts studied in grantee reports, as a matter of routine practice.

Keeping up-to-date information about state and local program rules and how the rules are implemented is essential in order to make cross-state comparisons and to improve the capabilities for national data sets to estimate program eligibility and to evaluate policy changes. Before PRWORA, each state was required to complete a form for the federal government that detailed the rules, benefit levels, and a wide variety of information on their welfare programs, and the rules were published by DHHS annually. Under PRWORA, states must provide an annual report on the characteristics of their programs, but there is less standardization of how these characteristics are reported and the questions that states must answer about their programs are now more open-ended. Without standard methods for reporting, the reports provided to the department are likely to be more varied and more difficult to use in a research setting. In addition, it is unclear as yet whether the reports will contain a complete account of programs for the TANF-eligible population that are supported by state funds.

DHHS is in the best position to guide an effort for standard, comprehensive, and regular reporting of program rules. The department's efforts in this direction thus far have evolved quite slowly, in part because of the undeniable complexity of the welfare rules that are developing at the state and even the county level. Only recently were final reporting requirements established (The Federal Register, April 12, 1999). The department has sponsored a subcontractor to collect such information, but nothing has been produced as this report was being completed. An institutional structure that will ensure the long-run continuity of documenting state and county program rules needs to be put into place, and the department should take responsibility for assembling this information.

Because the current round of ASPE leaver study grants is well under way, some of the ideas suggested above for ways to further enhance the cross-area comparability of data items—beyond the significant efforts that ASPE has made in this regard with the current grantees—must wait until subsequent rounds of grants. However, there is time to foster greater cross-area comparability in the analysis and documentation of methods and results. For example, grantees should be encouraged to use standard tabulation and reporting categories for key

stratifiers, such as education, employment, and welfare recipiency history; include information on the welfare program provisions in effect for the cohorts studied and key features of the economic environment (e.g., state and local unemployment rates); and fully document their data collection and analysis methods. For surveys, ASPE should foster agreement on a standard definition of response rates and other indicators of survey quality.

For subsequent rounds of studies of welfare leavers, the panel recommends that ASPE ask grantees to specify a broad definition of leavers that includes the widest possible set of families, such as child-only cases. Most of the grantees are excluding these cases from their leaver studies. While there are reasons to exclude child-only cases, the panel believes the arguments are stronger to include them, so that it is possible to assess the circumstances of families that are receiving reduced cash assistance as well as those that are receiving no cash assistance. If administrative records systems are redesigned to track recipient families, it may be possible to analyze child-only cases at relatively low cost.