With the passing of the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) of 1996, the United States embarked on a major social experiment with its social welfare and safety net programs for the poor. The most far-reaching reform of the cash welfare system for single mothers since 1935, PRWORA replaced the federal entitlement program for low-income families and children (Aid to Families with Dependent Children, AFDC) with a state-administered block grant program, the Temporary Assistance for Needy Families (TANF). Determining the consequences of this experiment is of great importance. Has welfare reform “worked?” What were the effects of the reforms on families and individuals? What reforms worked for whom and why? In looking toward the development of new policies to aid low-income families, which elements of the new welfare system need to be changed and which left as is?
For these fundamental questions to be answered adequately, two issues need to be addressed. First, how should one go about answering these questions— what methods should be used and what types of studies should be conducted in order to determine the effects of welfare reform? Second, what types of data are needed to measure the effects of welfare reform? Are federal and state data sources currently available sufficient to carry out needed evaluations, and, if not, what investments in that infrastructure are needed?
These two issues are the subject of this report.
To answer these questions, the Committee on National Statistics of the National Research Council formed the Panel on Data and Methods for Measuring
the Effects of Changes in Social Welfare Programs. This panel is sponsored by the Office of the Assistant Secretary for Planning and Evaluation (ASPE) of the U.S. Department of Health and Human Services (DHHS) through a congressional appropriation. The charge to the panel is to review methods and data needed to evaluate the outcomes of changes in social welfare programs on families and individuals. The panel is specifically charged with assisting the department in (1) identifying how best to measure and track program eligibility, participation, child well-being, and other outcomes; (2) evaluating data, research designs, and methods for the study of welfare reform outcomes; and (3) identifying needed areas and topics of research. In doing so, the panel was asked to consider alternative federal and state data sources, the limitations of currently available data, appropriate evaluation designs and methods for analysis, and findings from previous research and evaluation. The panel is also specifically charged with reviewing data needs and methods for tracking and assessing the effects of program changes on families who stop receiving cash assistance—i.e., welfare leavers.
The set of welfare reform projects that have been completed or are now under way is impressive in scope, volume, and diversity. The volume of research is unprecedented in comparison with any prior era of welfare reform. A large number of capable researchers in the private and public sectors are devoting major efforts to welfare reform research and have been producing a number of valuable and informative studies. Both ASPE and the Administration for Children and Families (ACF) in DHHS have substantial agendas for welfare reform research and have supported much high quality work, as have private foundations.
The panel finds that studies of welfare reform to date have done a reasonable job of monitoring the progress of the low-income and welfare populations—that is, tracking the well-being of these populations over time, although usually only after reform. More useful studies of this type are under way. However, monitoring studies are only the first step in assessing the effects of welfare reform. The second, more critical step is to evaluate the effects of welfare reform—that is, how it has changed the outcomes for families and individuals relative to what would have happened in the absence of reform. The panel finds that the evaluation studies that have been done are only able to address a small number of questions. There are many important questions that have not been addressed at all or not adequately addressed. Little is known about the effects of specific individual reform strategies, for example, a human capital approach versus a work-first approach or a set of relatively strict work requirements versus a set of less strict work requirements. Evaluations of other questions have been limited by weaknesses in data. These weaknesses are particularly limiting for studies that have assessed the overall effect of welfare reform and for national-level
studies of broad components of the reform, such as any time limit versus no time limit or work requirements versus no work requirements. Consequently, many important evaluation questions have not been adequately answered.
The panel also finds that the nation’s data infrastructure currently has serious limitations and weaknesses for the study of welfare reform, at both the national and state levels. These limitations have implications for both monitoring and evaluation studies. National-level survey data sets are of limited sample size, have significant problems of nonresponse, and are not readily able to adjust the content of questions on welfare program participation to the devolved structure of programs. Moreover, serious delays in producing key data sets have limited publicly available data for the post-PRWORA period, making it very difficult to examine TANF outcomes. Data on program characteristics and rules in the various states have only lately been developed. State-level administrative data have considerable potential but vary greatly in quality and quantity and lack comparability across the states. Matching different state-level administrative data sets across different programs would be of great value, but confidentiality and access rules limit the degree to which matches of data sets can be made. Finally, state-level surveys are in their infancy and have only provided limited data for monitoring and evaluation.
On an even more basic level, the panel finds that no overarching research agenda for evaluating the changes in welfare program policy has been established. There has been no concerted effort to outline which questions about the effects of the reforms need to be answered in order to assess whether the reforms were a success, or more generally, what set of outcomes the research and evaluation community should be examining. Existing welfare reform research consists of a large number of studies, funded by a wide and diverse set of public and private organizations, that, taken as a whole, are unfocused and uncoordinated and leave some questions unaddressed. Some types of studies have been overemphasized and have received disproportionate attention—those focusing on families who have left welfare, for example—while others have been underemphasized-for example, those that evaluate the effects of broad and specific components of reforms on families and individuals. Overall, the panel finds that the nation has largely failed in one of the most important goals of a mature and advanced society, namely, to be able to measure the effects of the policies it enacts so that these policies can be improved in the future.
KEY QUESTIONS OF INTEREST
Finding no systematic assessment of important questions that need to be addressed to evaluate welfare policy reforms, the panel took up the task of identifying the key questions of interest itself. The panel considered three separate issues in this respect: What are the populations of interest? What outcomes are of most interest? What formal evaluation questions should be answered?
Populations of Interest
For a reform as fundamental as that which has occurred under PRWORA, virtually all families in the low-income population may be affected, as the effects of reforms of particular welfare programs reverberate through all families in low income communities. Low-income families should, therefore, be the first and foremost population of interest for welfare reform evaluations. For the TANF program in particular, the population of all low income single mothers and their children is a broad population of interest, for virtually all have been affected by reform. A broad perspective has not been taken in most existing studies of the recent reforms, which have, rather, focused on fairly narrowly defined subgroups.
Within this broad population, many subgroups are indeed of interest. Those families who once were on welfare but subsequently left are one subgroup of interest, for example. Studies of welfare leavers have, in fact, constituted the major focus of research on the effects of welfare reform. However, leavers are only a small portion of the population of interest, for this group is not only just a subset of those families in the low-income population who are affected by welfare reform, but also only a subset of those who might be defined as the welfare population. Other important groups include families who are still receiving benefits—so-called “stayers” —and those who are not receiving benefits because they have been diverted, rejected, or discouraged from applying as a consequence of welfare reform. In addition, subgroups with special needs, such as those with mental or physical health problems, substance abuse problems, or other problems that may make their transition to employment and self-sufficiency more difficult, the “hard-to-serve” welfare population, should also be the subject of evaluation studies on specific groups within the low-income population. There have been some studies on these special need groups but much more is needed.
Outcomes of Interest
The welfare reform act of 1996 had many goals, ranging from increasing work and self-sufficiency for poor families, to reducing out-of-wedlock births and promoting marriage, to reducing welfare caseloads, and to giving states more flexibility and control over their own programs. Not surprisingly, different audiences—national legislators and administrative officials, state legislators and program administrators, and the general public—are interested in different outcomes.
The panel concludes that the set of outcomes of interest for measuring the effects of welfare reform should be defined broadly and should include outcomes of interest to all of these different audiences. Broadly defined, these outcomes include:
traditional measures of well-being for adults and families (including income, poverty rates, consumption of food, clothing, housing and other goods, employment, education and health);
traditional measures of child well-being (physical, cognitive, and behavioral);
measures of family structure and family formation (marriage, childbearing, out-of-wedlock birth, and living arrangements);
outcomes for governments themselves such as sizes of caseloads and expenditures on programs; and
changes in organizational structures for administering programs.
This list is not exhaustive of all the possible outcomes, but it covers the categories of outcomes that must be included in a complete assessment of the effects of welfare reform. Measures of well-being should be conceptually broad and cover all dimensions of adult, child, and family well-being—including health, economic, social, and safety—and should be operationally measured according to current scientific standards. How each of the specific outcomes are defined and operationally measured is a very important and difficult issue, but not one that the panel addresses in this report.
The body of welfare reform research conducted to date has been reasonably complete in addressing most of these outcomes in one place or another. There are certain areas that are understudied, such as the effects of welfare reform on family structure and on children, perhaps at least partially because these outcomes may not change as rapidly as some economic outcomes like employment. Nevertheless, the main limitations in studying different outcomes have been related to gaps in methods and data availability, not lack of interest in the outcomes themselves.
Research Questions of Interest
In outlining a broad research agenda for understanding the effects of welfare reform and for future assessments of reform, the panel has identified three types of questions of interest for understanding welfare reform: monitoring questions, which concern trends in the well-being of the low income population and its subgroups; questions about what rules govern recipients and how welfare reform has affected state and local welfare systems themselves; and formal evaluation questions, which assess the effect of welfare reform on individuals and families relative to what would have happened in its absence. The panel concludes that the key set of questions of interest for a comprehensive research agenda are as follows:
Conclusion 3.5 The monitoring questions of interest are the following: How has the well-being of the low-income population and key subgroups evolved subsequent to welfare reform? Which subgroups are doing well and which are doing less well? Which subgroups are in greatest need and deserve the attention of policy makers?
Conclusion 3.6 The descriptive questions of interest regarding program policy and implementation are the following: What policies, programs, and administrative practices have states and localities actually implemented as part of welfare reform? How wide is the variation across states and even within states in policy? How has implementation differed from officially described policy? How has the non-TANF programmatic environment changed?
Conclusion 3.7 The impact evaluation questions of interest are the following: What are the overall effects of the complete bundle of changes in policies, programs, and practices on the well-being of the low-income population, including the effects on both adults and children and on specific subpopulations of interest? What are the effects of the individual broad components of welfare reform on the well-being of the low-income population and subpopulations of interest? What are the effects of specific detailed strategies within each of the broad program components on the well-being of the low income population and the subpopulations of interest—what works and for whom?
The greatest weakness in the identification of the key questions of interest has been a lack of public articulation of the questions, and consequently, a failure to systematically ensure that all questions are addressed with appropriate emphasis. Ideally, a research framework that outlines the populations, outcomes, and research questions of interest, like the type the panel has composed, would have been established early in the post-PRWORA period. Because it has not, there are major gaps in what is known about the effects of reform. For future waves of welfare reform, this comprehensive listing of questions, populations, and outcomes is the responsibility of the federal government and should be conducted by an agency that is capable of taking a leadership role in guiding research on welfare reform. The most appropriate agency for that role, in the view of the panel, is ASPE.
Recommendation 3.1 The panel recommends that ASPE take primary responsibility for publicly defining the questions of interest for welfare reform research and evaluation, identifying emerging issues for social welfare programs, and defining alternative detailed strategies and policies that address the what-works-and-for-whom questions. In doing so, ASPE should expand its current activities in seeking input from states, private foundations, and other stakeholders on emerging policy and evaluation issues.
EVALUATION METHODS FOR THE QUESTIONS OF INTEREST
In its examination of evaluation methods for welfare reform, the panel asked what evaluation methods are best for addressing each of the three types of evaluation questions identified in Conclusion 3.7 above. Different methods are preferable for different questions and, therefore, each of the three evaluation questions must be approached with different methodological considerations. The most promising methods for addressing the first question, the overall effect of welfare reform, are nonexperimental methods such as time-series, caseload, and econometric modeling. However, these methods require good across-area data on programs, area characteristics, and individual characteristics and outcomes, for across-area variation is the primary means by which effects of welfare reform are inferred. Data limitations have constrained the ability of these studies to credibly estimate overall impacts, although there have been several good studies of this type.
The second question, concerning the effects of broad welfare reform components (e.g., time limits, work requirements, sanctions, or family caps) within a fixed overall reform environment, is best addressed with a combination of experimental methods and nonexperimental methods. However, the experiments that have been conducted to date have not been designed to estimate the impact of broad components, and nonexperimental methods have not been successful in doing so as well because of data limitations. Consequently, there have been virtually no credible studies of the effects of broad components.
The third question, concerning the impact of detailed welfare reform strategies (for example, a human capital versus a work first approach, or a 2-year time limit versus a 5-year time limit) are best addressed with experimental methods. There have been several experiments designed to assess the effects of detailed strategies in welfare reforms, but they have been significantly weakened by design problems that threaten their validity. However, there is considerable promise for the use of experiments for this purpose in the future.
Conclusion 4.2 Experimental methods could not have been used for evaluating the overall effects of PRWORA and are, in general, not appropriate for evaluating the overall effects of large-scale, systemwide changes in social programs.
Conclusion 4.3 Experimental methods are a powerful tool for evaluating the effects of broad components and detailed strategies within a fixed overall reform environment and for evaluating incremental changes in welfare programs. However, experimental methods have limitations and should be complemented with nonexperimental analyses to obtain a complete picture of the effects of reform.
Conclusion 4.4 Nonexperimental methods, primarily time-series, and comparative group methods, are best suited for gauging the
overall effect of welfare reform and least suited for gauging the effects of detailed reform strategies, and as important as experiments for the evaluation of broad individual components. However, nonexperimental methods require good cross-area data on programs, area characteristics, and individual characteristics and outcomes.
DATA FOR MONITORING AND EVALUATING SOCIAL WELFARE PROGRAMS
Addressing the research questions of interest for welfare reform require data from multiple sources (survey, administrative, qualitative, and program description data) and across multiple levels (national, state, and local). Although the current data infrastructure contains many excellent sources, limitations in the infrastructure are sufficiently severe that important questions concerning the effects of PRWORA and other welfare reforms have been, and will continue to be, very difficult, if not impossible, to answer. As a consequence, much work needs to be done to make them useful for research.
The report contains many recommendations for improvements in the current data infrastructure, both for national-level data sets and for state- and local-level data sets. These recommendations are geared toward addressing specific limitations of currently available data. However, limitations in the current data infrastructure for human service and social welfare program research are partly the result of inadequate governmental structures to support the collection and maintenance of data on these programs. Current responsibilities and functions for collection of such data are spread across several different agencies, none of whose primary purpose is the maintenance and development of data.
Within the DHHS, both the ACF and ASPE are responsible for components of the entire data collection system. ACF is primarily a programmatic department charged with administering social welfare programs aimed at families and children. It is also responsible for collecting administrative data on TANF and related programs from the states, but these data are collected to assess state performance and compliance with federal mandates; they are not collected with the primary purpose of research or program evaluation. ASPE is responsible for strategic planning, policy development, and evaluation of all health and human service programs. It has supported many data collection activities in the past and currently is supporting data collection for welfare leaver and diversion study grants. However, data collection is not part of its specific charge, and ASPE does not have the resources to fully address the extensive data needs. DHHS has a number of other agencies that collect data covering health topics and health programs, but none of these is charged with collecting data for social welfare programs. Other federal departments, such as the U.S. Department of Labor, U.S. Department of Education, and U.S. Department of Justice have agencies that
are charged with collecting data needed to administer and evaluate programs and to carry out the missions of their larger agencies; DHHS does not contain such an agency for carrying out data collection for social welfare programs.
Conclusion 6.1 No agency within DHHS has distinct administrative authority and responsibility for the collection and development of data relevant to social welfare and human service policies and programs. This administrative gap is a major reason for many of the inadequacies in the data infrastructure for monitoring and evaluating welfare policies.
The need for methodological leadership, increased capacity for data collection and analysis, technical assistance to states for developing their own surveys and administrative data, leadership in addressing data confidentiality issues, and guidance in the development of data archives dedicated to social program data leads the panel to recommend that alternative administrative mechanisms be considered. Consideration should be given to several alternatives. For example, the functions that the panel believes need to be performed could be placed within an existing statistical agency in DHHS, such as the National Center for Health Statistics. Alternatively, a new statistical agency within DHHS could be created to handle social welfare program data. Another option would be to expand one of the other agencies within DHHS with increased statistical staff and to assign that agency the responsibility for working with both federal agencies and states in developing and maintaining data. What option is chosen will require careful consideration and joint discussions between all the relevant agencies and departments. Reassignment of functions from one agency to another would be required, and departments and agencies outside DHHS would have to be involved because they have authority over other welfare programs (e.g., the Department of Labor, Department of Agriculture, and Department of Education, to name only three).
Recommendation 6.2 The panel recommends that an organizational entity be identified or created within DHHS, and that this entity be assigned direct administrative responsibility and authority for carrying out statistical functions and data collection in the area of social welfare programs and the populations they serve. The entity would also coordinate data collection and analysis activities between states and the federal government.
However the entity is achieved, it is critical that it is separate and independent from other programmatic and policy agencies within DHHS, which is important for ensuring that data collected will have credibility with both data suppliers and users.
Because devolution has made states responsible for TANF and other related programs, state data collection and coordination functions must necessarily be a
part of the responsibilities to be assigned. Coordination of data collection activities will require strong cooperation between the states and DHHS, an effort that the panel concludes will be most effectively conducted if the federal government takes the lead. Such a federal-state program would probably require the creation of state agencies to work with the federal government and to ensure that state-level data relevant to social welfare programs are available. Cooperatively developing data programs is necessary, as the DHHS entity should provide both technical assistance and some funding for states to develop their data collection systems.
The panel does not offer a specific blueprint for administrative arrangements, but we are specific about the types of functions that should be carried out. These functions fall under the topics of national surveys, administrative data, technical assistance, reports, and a data archive.
National Surveys The organizational entity that is assigned responsibility would be the primary sponsor of the national surveys used to monitor and evaluate human service and social welfare programs and, in general, content related to the low income population. It would contract with the Census Bureau or with private survey organizations to conduct these surveys. These include the Survey of Income and Program Participation and the Survey of Program Dynamics, and perhaps parts of other surveys, like the topical modules in the Current Population Survey that cover social welfare program topics. As the entity with lead responsibility for content and design of these surveys, it would also work with other agencies that have interests in these surveys. It would also explore the linkage of national-level administrative data to the national survey data that address social welfare program topics.
Administrative Data The development and management of a cooperative welfare and social statistics data and information effort with the states would also be a needed function. Existing or new state statistical agencies should be full partners in this effort. Funding or financial incentives for the states to provide data to the federal agency and determining the form and content of the data submission should also be part of the responsibilities of the federal authority. Periodic reporting would be part of this program. Benefits Reporting Areas should be considered.
The development of standards for the use of administrative data for research purposes is an additional needed function. These standards should include definitions of services and benefit units, recipients and case members, data formats, and processes for documenting administrative data files.
In order to promote sharing of data resources for welfare and social statistics research and evaluation, coordination with other federal and state data collection agencies would also be required.
Leadership in advancing the use of and accessibility to all data provided by the states to DHHS for monitoring and social welfare program evaluation purposes is another important function.
Technical Assistance Another need is the provision of technical assistance to states on the use of administrative data and on the development, conduct, and analysis of surveys. The technical assistance could be used as a tool to promote the goals of comparability, improved data quality, data linkages, and data security and access.
Reports The federal entity should have responsibility for producing periodic reports on topics related to social welfare program utilization and the well-being of those who utilize these programs. One set of reports would be based on the data submitted by the states through the cooperative data collection effort mentioned above. It should also collect and publish social welfare program rules and policies, particularly for TANF and related separate state programs, for every state and every sub-state area where appropriate.
Data Archive for Continuing Research Needs A leadership role is needed in developing data archives on particular topics for use in social welfare program evaluation and research. Archives may include state surveys and administrative data, for which the agency would be responsible for preparing the surveys or administrative data for use by researchers. Maintaining an archive of welfare policies and programs description data throughout the states, and where relevant, in local areas, should also be a responsibility.
Carrying out these functions of the proposed data collection system will require strong leadership and sustained support at both the federal and state levels. If welfare programs continue to be operated in a devolved system, the need for and benefits from such a federal-state system will continue to grow.