Evaluating Welfare Reform: A Framework and Review of Current Work, Interim Report (1999)

The United States is in the midst of a major social experiment with its social welfare and safety net programs for the poor. The Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) of 1996, the most fundamental and far-reaching reform of the social welfare system since 1935, replaced the federal entitlement program for low-income families and children (Aid to Families with Dependent Children, AFDC) with a program financed by state-administered block grants, the Temporary Assistance for Needy Families (TANF) program. PRWORA furthered a trend that began earlier in the decade under so-called "waiver" programs—state experiments with different program rules—towards devolution of design and control of social welfare programs from the federal government to state governments. The legislation imposed several major new requirements on state programs (e.g., lifetime time limits on receipt of benefits and minimum work requirements), but otherwise allowed states to reconfigure their programs as they each see fit.

There is tremendous interest among federal and state officials, policy analysts, and the public in knowing the consequences of this experiment. Determining those consequences poses several challenges to the federal government, state governments, and the research and evaluation community. One challenge arises because the devolution of program responsibility has led to wide variation in programs across states and even within states—for several states have passed responsibility for program design to counties,¹ and many states have given coun

ties and smaller entities administrative freedom to operate programs. The task of simply recording each of the new policies that have been legislated in each state or county, how those policies have been implemented, and how they are evolving over time has become much more difficult and will require major effort because of the variation in program rules. Reporting requirements in the 1996 Act and in the federal regulations that have followed it require the states to report some aspects of their policies on an annual basis, but far less than what is required for a comprehensive accounting of the welfare programs that are available to the poor families in each state.

A second challenge is to the nation's data infrastructure on poor and welfare populations, which must necessarily be the basis for assessing the effects of PRWORA. The extensive variation of programs across the country puts great strain on national-level data sets based on household surveys—such as those conducted by the Census Bureau and other federal statistical agencies—because the samples of the welfare-relevant and poor populations in those surveys may not be large enough to capture the effects of welfare reform. State-level data sets, both administrative and survey based, will be asked to provide new information far beyond anything they have previously needed to provide, for such data have historically been primarily used for management rather than evaluation purposes.

A third, more analytic, set of challenges is that of assessing the effect of legislation relative to other forces and trends in the economy and society. Foremost among these is the necessity to separate the effects of the improving U.S. economy and the PRWORA legislation, both of which occurred almost simultaneously. Choosing a comparison group with which to assess the effects of PRWORA is made especially difficult by this occurrence. Additionally, cross-state comparisons are complicated by the need to account for differences in economic and policy environments other than those in PRWORA-related policies. There are also challenges in assessing which of the many provisions of PRWORA have effects and which do not, for all the provisions were enacted at roughly the same time. Another part of this analytical challenge is to assess the effects of PRWORA on participation in other social welfare programs that were not substantially changed in the legislation (such as food stamps and Supplemental Security Income [SSI]).

It is in this context that the Office of the Assistant Secretary for Planning and Evaluation (ASPE) of the Department of Health and Human Services (DHHS) asked the Committee on National Statistics of the National Research Council to convene the Panel on Data and Methods for Measuring the Effects of Changes in Social Welfare Programs, using funds in a congressional appropriation to ASPE for this purpose. The same congressional appropriation provided funding to ASPE for data collection and evaluation of the effects of welfare reform on families who have left welfare, commonly called "welfare leavers." Language accompanying the appropriation requested that the new panel provide guidance

on the ASPE research plan for tracking former welfare recipients and suggest directions for future welfare-related research.

The charge to the panel is to review data needs and methods for evaluating the outcomes of changes in social welfare programs on families and individuals. The panel is specifically charged to assist the department in: (1) identifying how best to measure and track program eligibility, participation, child well-being, and other outcomes; (2) evaluating data, research designs, and methodologies for the study of welfare reform outcomes; and (3) identifying needed areas and topics of research. In doing so, the panel may consider alternative federal and state data sources, the limitations of currently available data, appropriate evaluation design and methods for analysis and inference, and, finally, findings from previous research and evaluation. The panel is also specifically charged with reviewing data needs and methods for tracking and assessing the effects of program changes on families who stop receiving cash assistance. To fulfill that component of the charge, a considerable portion of this interim report focuses on studies of welfare leavers.

Before the passage of PRWORA, the main cash program for the poor, AFDC, did not vary greatly across the country. Although states were allowed to set benefit levels (which varied a great deal—from $120 per month in Mississippi to almost $600 in most of California in 1996), the basic program rules (income eligibility, asset limits, treatment of two-parent families) were standardized across the states. In the late 1980s and early 1990s, federal administrators issued more and more waivers to states to enact and operate programs that deviated from federal requirements. PRWORA continued this trend and gave states even more discretion for designing their own social welfare programs. Broadly speaking, the law imposed only three types of major mandates:²

1. All persons receiving assistance for at least 2 years must participate in work or work activities in order to receive assistance, with different specific requirements for single-and two-parent families and for single parents with young children.

2. There is a lifetime limit to the number of years a family may receive assistance paid out of federal funds. This limit was set at a maximum of 5 cumulative years, although states can legislate shorter lifetime limits and can exempt up to 20 percent of the caseload from the limit.³

3. In order for unmarried minor teenage parents to receive assistance, they must live with an adult or in an approved adult setting and they must participate in educational activities leading to high school graduation or receiving a general equivalency degree (GED).

With only these three mandates, states are free not only to set benefit levels, as they did under AFDC, but also to set eligibility requirements and financial incentives, such as earnings disregards; to alter the relative importance of cash benefits and support services, such as job search assistance, child care, and the like (and to shift toward support services and away from cash benefits, if they desire); to set sanctions and diversion policies without restriction; to mandate participation in substance abuse or educational programs; and to vary their programs in numerous other ways. States appear to be taking up the challenge of devising their own programs which differ in many ways in all of these and other dimensions.⁴

Keeping track of all of the policies and determining the program rules that families and recipients are experiencing—that is, knowing what policy rules are being applied to each welfare case—is a formidable task for research and data collection. A further complication to the task of defining the program rules actually faced by recipients arises because policy implementations vary across states and across welfare agencies within states and across counties. For example, sanctions may be strictly or loosely enforced, in ways that the formal rules do not indicate. There is also variance across states and within states in the way in which the rules are publicized to the eligible population, with many agencies trying to alert potential applicants of significant changes in the rules and to change applicants' expectations of the program application and participation processes. Relatedly, some states and counties have formal diversion programs in which up-front payments, job search services, or other support services such as transportation support or child care are provided to applicants to keep them from enrolling or re-enrolling in the cash assistance program. Other states have informal, less well documented, diversion policies to discourage application and benefit receipt. The degree to which these formal and informal diversion programs are applied in different welfare offices makes determination of the actual policy environment facing a given low-income family more difficult.⁵

These difficulties in measuring the policies faced by families in different locations represent a barrier to any evaluation that uses cross-state comparisons as a means of evaluation.⁶ Thus, welfare reform evaluations conducted with national-level household surveys, such as the Survey of Income and Program Participation, the Current Population Survey, the Panel Study of Income Dynamics, or the National Longitudinal Survey of Youth, are disadvantaged by this problem of policy measurement. Comparisons of individual state-level studies across states is also hindered because the policy descriptions provided by states may not be provided on a comparable basis.

The 1996 legislation changed the state reporting requirements, which also have implications for data collection and program evaluation. States must now produce quarterly reports on monthly measures of many characteristics of a sample of families and children receiving cash assistance, such as their demographic characteristics, employment status, and program use. States are also required to report some information about a sample of closed cases. In order to meet these requirements, states must further develop and link their administrative data sets and, perhaps, supplement this information with survey data, especially for adults in child-only cases. The federal government will then face the difficult task of interpreting the state information, as the definitions of the characteristics and the data used may not be comparable across states because reporting systems and definitions are not fully standardized.

The changes initiated in the PRWORA legislation also have implications for the methodologies that researchers use to evaluate the programs. In the late 1980s and early 1990s, the dominant methodology in program evaluation was the small-scale, randomized experiment which administered a set of program reforms to an experimental group but not to a randomized control group, to whom they could be compared.⁷ Several of the randomized experiments that began in this period are still being conducted. However, the large-scale changes in the TANF program resulting from PRWORA, the evolving nature of welfare reform policy at the state level, and the difficulty of maintaining a control group in an environment in which systemwide changes are occurring have led to a decline in use of the experimental method. A heavier reliance upon nonexperimental methods of analysis has been the result.

A further complication in evaluating changes in policies arises from the fact that states have enacted ''bundles" of reforms simultaneously—various forms and combinations of time limits, work requirements, family caps and other rules.

Although the entire bundle may be the initial policy issue of interest, it will be difficult to gauge the effect of specific components of the bundle if and when further reform is considered.

The membership of the panel was constituted in the summer of 1998, and the first meeting of the panel in September 1998 brought together representatives from ASPE and the Administration for Children and Families, both in DHHS, to discuss their welfare reform activities with the panel, as well as representatives from several other private and public welfare reform studies under way around the country. After this overview of the landscape of welfare reform, the panel turned to an examination of the ASPE-funded welfare leavers studies. Because part of its charge is to review and evaluate the designs of the 14 awards ASPE made in September 1998 to study welfare leavers and because of the need to make recommendations to ASPE regarding these studies as soon as possible, the panel convened a workshop in November 1998 to learn more about the designs of these studies. The ASPE awards were granted to 14 jurisdictions (states, counties, and county groups) to assist them in tracking those who leave the welfare rolls and to develop administrative data, survey data, and linkages of these two to form the information base to study welfare leavers. The November 1998 workshop brought representatives of the 14 studies to Washington, D.C., to discuss their plans with panel members and other welfare experts. Topics discussed at the workshop included data issues, the outcomes of interest, the populations of interest, and the appropriate methodologies for data analysis (see National Research Council, 1999).

From these activities and its own deliberations, the panel has produced this interim report, which has three purposes. The first is to outline a broad framework for evaluating and studying welfare reform that clarifies the basic principles of good evaluation design and which discusses the special evaluation and data issues involved in welfare reform analysis. The panel found it essential that these basic principles be laid out explicitly before assessing the ASPE welfare leaver studies or any other welfare reform study. The second purpose is to conduct a detailed review of the 14 ASPE leaver studies and to comment on their designs. As part of this review, the panel also briefly surveyed other leaver studies that have been conducted, as well as other welfare reform studies under way around the country. The third purpose of the report is to make immediate, short-run recommendations to ASPE and the department on its welfare reform agenda, with an emphasis on the issue of continued funding for welfare leaver studies. With these three goals achieved, the panel is currently engaged in a broader evaluation of welfare reform data and methodologies that will lead to its final report in 2000.

The rest of this report has three sections. The first section (Chapter 2)

provides an overall framework for evaluating welfare reform, considering both evaluation methodologies and data needs. The material in this section will already be quite familiar to evaluation experts and is primarily targeted at state and federal policy-makers who have not been exposed to discussions of evaluation methodology. The discussion is intended to place evaluation methods and data sources into the context of current social policy programs and dynamics. Chapter 3 provides, at the request of ASPE, an in-depth discussion of 14 recently funded state-level welfare leaver studies. The designs of these studies, which are still in progress, are evaluated by applying the general principles of evaluation reviewed in Chapter 2, leading to an assessment of the strengths and weaknesses of the evaluation designs and databases planned for these projects. The 14 studies represent only a fraction of welfare reform evaluation activity around the country and only a fraction of even those studies that have been completed or are under way. Consequently, the discussion in the chapter is targeted primarily to those conducting the 14 studies and sponsoring officials at ASPE. However, the conclusions reached in this section have broader implications for welfare reform evaluation and therefore will be of some interest to the general welfare reform research and policy communities. This section also reports briefly on the other major welfare reform activities around the country.

The third and final section (Chapter 4) provides immediate, short-run recommendations directed primarily to ASPE regarding its research agenda for welfare reform and its further sponsoring of welfare leaver studies. The panel's final report will contain broader recommendations on evaluation and data needs for welfare reform.