Evaluating Welfare Reform in an Era of Transition (2001)

Our investigation of data needs in Chapter 5 demonstrated that the national infrastructure for data collection on social welfare programs and populations has many severe limitations. These limitations are so important that they have seriously constrained the ability of the government and private analysts to monitor and evaluate the effects of welfare reform in the 1990s and of the PRWORA legislation. Despite the best intentions of analysts, there is unlikely to be any significant additional progress made in learning the effects of welfare reform until the nation’s data infrastructure undergoes major improvements.

The inadequacies appear at all levels, federal, state, and local, and in both survey and administrative data. National-level survey data sets are of limited sample size, and have suffered serious problems of nonresponse, and their questions on welfare program participation are not adequate to capture the new devolved structure of programs. In addition, serious delays in producing key data sets have limited publicly available data for the post-PRWORA period, constraining the analysis of outcomes for that important time frame. Furthermore, national-level administrative data sets, such as those based on TANF reporting requirements from the states, which should provide information on those still on TANF, are of dubious quality for research purposes and are unlikely to be used for this purpose. Data on what program characteristics and rules states and local areas have been adopted are a necessary ingredient in knowing what welfare reform actually has constituted. But these data have only lately been developed and will need continuing and significant support to be maintained.

State-level administrative data have considerable potential to yield information on families still on TANF, as well as families who have left TANF or been

discouraged from applying, but they have historically been used primarily for management purposes and need much additional work to be made useful for research. State data sets of this type also vary greatly in quality and quantity across states. States often use noncomparable definitions and categorizations, making it difficult to use them for cross-state assessments. In addition, state-level administrative data sets are not often available for many past years because states do not generally archive them; so, for example, it is often difficult to compare current TANF families to those who were on AFDC prior to PRWORA. State-level TANF administrative data sets also have significant limitations in terms of content. These limitations could be partly addressed by matching different state-level administrative data sets across different programs, but such matching has been limited by confidentiality and access rules. Finally, state-level surveys of current TANF recipients, former recipients, and other groups affected by welfare reform—which could provide important information on how families are doing and what their needs are that are not available elsewhere—are in their infancy and need support and development if they are to be a significant resource, for evaluation.

Although there are many reasons for this discouraging state of affairs, the panel concludes that it is partly the result of inadequacies in the structure of federal administrative authority and responsibility for data collection in this area. No existing federal agency has general authority and responsibility for data collection on social welfare programs and populations (nor the necessary staff to implement such authority). To remedy the situation and improve the data infrastructure, there is a set of distinct, specific administrative functions that need to be carried out. We list these functions later in this chapter. None of these functions are now seen as the responsibility of any agency.

In this chapter, we first describe the current structure of federal responsibility for data collection on social welfare programs and populations and identify gaps. We then discuss what the federal role should be in an era of devolution. Finally, we lay out the functions that need to be carried out, and we discuss some alternative organizational structures that might be developed to carry out those functions.

Responsibilities for collection of data relevant to social welfare programs are currently spread across several different federal agencies, none of whose primary purpose is the maintenance and development of data on these programs. Within the U.S. Department of Health and Human Services, both the Administration for Children and Families (ACF) and the Office of the Assistant Secretary for Planning and Evaluation (ASPE) are responsible for components of the data collection system, but neither has general responsibility. Neither agency is a statistical agency and so all data collection activities conducted by these agencies are sec-

ondary to its main mission. Those activities tend to be short-run in nature and designed only to carry out other current responsibilities.

ACF is responsible for administering the department’s social welfare programs for families and children, of which TANF is one of the major programs. As part of this charge, ACF is responsible for collecting TANF and related administrative data from states. However, ACF’s primary responsibility is to use these data to check the compliance of states with various provisions of PRWORA and to assess performances of states for monetary awards under PRWORA. For example, ACF is charged with monitoring the compliance of caseload employment requirements by state TANF programs. ACF also monitors aspects of child support enforcement provisions and other provisions that can entail the administration of sanctions and bonuses to states by the federal government. Therefore, the mission of ACF is considerably narrower than what is required to support a general-purpose data infrastructure for social welfare and human service programs at the federal and state levels.

ASPE is responsible for strategic planning, policy development, research, and evaluation for the Department’s programs, including TANF and related social welfare programs for families and children. As part of these duties, ASPE currently has some responsibilities to collect data, to support data collection undertaken by others, and to support data use for policy planning, development, and evaluation. In the area of state-level data sets, ASPE has taken a new leadership role in guiding the states in data development. As noted above, for example, ASPE is currently funding 17 state and local welfare leaver and diversion studies and has worked with the contractors for these studies to develop state-level databases for such welfare research, including the provision of technical assistance on data collection (both administrative and survey). ASPE also made an attempt to persuade these states and localities to use common definitions of variables in their analyses of welfare leavers. This activity is a relatively new role for ASPE, which has not traditionally been as heavily involved in state-level data collection. ASPE has been led in this direction by the devolution of responsibility for welfare programs to the states and by the need to sponsor high-quality, state-level welfare research. But the ASPE staff are not survey statisticians and methodologists; they are primarily from other disciplines and see their mission as consumers of data rather than producers of data. Furthermore, ASPE’s resources are far too limited in terms of funding and staff to be able to take a major oversight role in state-level data collection. ASPE also does not have the resources or administrative responsibility for ensuring that investments are made in the long-run state data infrastructure, for ASPE is usually fully engaged in carrying out short-run policy functions needed by the administration and by Congress. Consequently, while ASPE has done very well with the resources it has had and has been a major force in achieving what gains there have been in the improvement of state data sets, it would require a major change in resource levels, staffing, and mission for ASPE to take on the larger role of carrying out the significant

changes to such data that are needed, while yet simultaneously fulfilling the other parts of its mission.

ASPE has been and continues to be heavily involved in the development of federal-level administrative and survey data sets. It is the primary research and evaluation agency overseeing the Survey of Income and Program Participation (SIPP), for example, although other agencies are involved in SIPP oversight as well. (SIPP is sponsored and operated by the Census Bureau and not by ASPE.) In its role in welfare reform, ASPE has also been involved in the “new hires” data set, the matching of administrative data (e.g., social security records) to survey data for research purposes, and a number of other related important activities. ASPE has a long history of supporting data collection for national-level data sets. However, in order to fulfill its planning and evaluation role within DHHS, ASPE has traditionally been a data user rather than a data collector. Its mission, as already noted, is research and evaluation; it is not a statistical agency.

The lack of any agency within DHHS that has the distinct administrative authority and responsibility, and the requisite staff expertise, for federal and state-level data collection on social welfare programs and populations has, the panel believes, been partly responsible for the limitations in the data infrastructure.

Other agencies outside DHHS are, of course, also involved in data collection for human service and social welfare programs. Most notable is the Census Bureau, which currently has primary responsibility for SIPP, the chief on-going nationally representative survey on participation in various social programs, as well as the Survey of Program Dynamics (SPD), the offshoot of the SIPP that was mandated under PRWORA to monitor the progress of welfare reform. As part of these survey programs, the Census Bureau is responsible for developing the content of the surveys (the questions on program participation, income, and resources and background characteristics to be asked of survey respondents) as well as fielding the surveys and producing and maintaining public-use data sets developed from them.

It is an atypical organizational structure for the Census Bureau, which is a data collection agency, to have primary responsibility for such mission-specific data sets as SIPP and SPD. Usually, primary responsibility for such data sets rests with a sponsoring agency that uses, or represents the users of, the data being collected. More typically, the agency administering programs for which the data are collected or whose mission is most closely related to the substance of the

survey has primary responsibility for survey content but works together with the Census Bureau (or other outside contractors, in some cases) in the translation of that survey content into a survey instrument form.

One example of the more typical model is the employment and wage data collection program. For this program, the Census Bureau collects the data through the Current Population Survey (CPS) and the Area Wage Survey; the Bureau of Labor Statistics (BLS) contracts with the Census Bureau for the data collection, but has primary responsibility for the content of these surveys. Similar contracting arrangements with the Census Bureau are used by the U.S. Department of Transportation, the U.S. Department of Justice, the National Center for Health Statistics, and others to collect population-level information relevant to the mandates of these agencies. These contracting-based arrangements with the Census Bureau have the advantage of ensuring that responsibility for the substantive content of the data collection resides in the agency that is closest to the relevant policy concerns and substantive issues that drive the need for these data. Such an arrangement is missing in the case of the collection of nationally representative data on social welfare programs.

This mismatch of responsibility and expertise for data collection among federal agencies has hindered the collection of data for social welfare programs. A previous National Research Council panel charged with reviewing SIPP noted the weaknesses of the organizational structure and the need for the Census Bureau to seek outside input on the content and basic design of the survey. The study recommended a different management structure within the Census Bureau to produce income and program participation statistics partly because of the lack of a social welfare program data collection agency within DHHS that would have been a logical agency to sponsor the survey (National Research Council, 1993b). Because devolution has made the collection of information on welfare program participation even more difficult for national-level surveys, such as the SIPP and SPD, the need for such substantive expertise in the responsibility for these surveys is even greater. Because there is no agency in DHHS that has direct responsibility for such survey data development, this need is not being fully met.

The lack of direct administrative authority and responsibility for social welfare program data collection within DHHS has also hindered the development of administrative data for research purposes, which has become an increasingly important source of data in the state-based programmatic environment. In a number of social program areas, including TANF, states now must provide DHHS with microdata—individual case-level records with client and service characteristics. In addition to their use for program administration and program compliance, these data are also being developed for research purposes. Because these data are not explicitly collected for research purposes, a substantial effort is required to convert the data to usable research form, and these efforts have been ongoing at the state level and in various divisions of DHHS. But because they are primarily agencies for administering programs, they do not always have the

resources and expertise to develop the data for research purposes. However, they are not independent from the programmatic functions of the agency, which is a crucial condition that improves the credibility of a data collection agency as being independent from political interests of the agency (National Research Coun cil, 2001b).

The federal government’s need to evaluate one of its major social welfare programs, TANF, is constrained by the realities of a decentralized program environment. Devolution has transferred much of the policy-making functions for welfare programs to states. As a result, federally centralized efforts to collect data for welfare program research, such as the national-level surveys, face more challenges than ever before, especially in collecting data for cross-state analyses and for measuring participation in state-based welfare programs. State-based but federally coordinated monitoring and evaluation efforts, such as the ASPE-sponsored welfare leaver studies, face challenges of comparability in defining terms and standards for data collection. Even the efforts of DHHS to collect data describing each state’s policies and administrative data from states to check program compliance and assess performance of state programs are hindered by these same comparability issues.

Even though devolution gave states authority over program design, the panel concludes that it is necessary to reaffirm the leading role of the federal government to ensure that data needed to evaluate these programs and to monitor well-being of populations that may use the programs are collected. The federal government should clearly oversee national-level data sets because the benefits of information gained from these data accrue to all the states. The federal government has a role to play in state-level data sets, as well. Only the federal government is in a position to coordinate comparability across state-level data sets, which would make the state data sets much more than the sum of their parts because cross-state evaluation and research comparisons could be made. Those types of comparisons would benefit all states. Yet no single state has the resources or the incentives to undertake steps toward comparability by itself. In addition, states do not generally have the expertise in data collection that exists at the federal level. Nor would it be efficient for them to do so, for a concentration of expertise in each of the states would be redundant and wasteful. The federal government, with its concentration of such staff, is in a position to provide technical assistance and guidance to all the states in a more efficient manner.

While the federal government is in the best position to take the lead, cooperation with states is necessary as well. States are more in tune with the realities of program operations and thus can help improve federal data collection activities and provide up-to-date information on ground-level developments related to social welfare programs and the populations they serve. Federal data collection agencies, therefore, have much to gain from cooperation with the states. States interested in developing their own data infrastructures can also gain from cooperation with the federal government. The federal data collection agencies have more expertise in technical matters for data collection and data use than most states do. While the panel concludes that the federal government should retain responsibility for collecting high-quality data for social welfare program monitoring and evaluation, coordination and cooperation with states is a necessary and critical part of developing a data infrastructure.

The need for methodological leadership to address the complex data collection, linkage, storage, and access needs for social program data leads the panel to recommend that alternative administrative mechanisms for lodging responsibility and authority for social welfare program data collection in some entity within DHHS be considered. The panel does not offer a blueprint of such a structure, for it does not have the expertise, time, resources, and charge to do so. There are many alternatives. For example, the functions that the panel believes need to be performed could be placed within an existing statistical agency in DHHS, such as National Center for Health Statistics. Alternatively, a new statistical agency within DHHS could be created to handle social welfare program data. Another option would be to expand one of the other agencies within DHHS with increased statistical staff and to assign that agency the responsibility for working with both federal agencies and states in developing and maintaining data. Which option is chosen will require careful consideration and joint discussions between all the agencies and departments involved. Reassignment of functions from one agency to another would be required, and departments and agencies outside DHHS would have to be involved because they have authority over other welfare programs (e.g., the Departments of Labor, Agriculture, and Education, to name only three).

The programs whose responsibility and authority for data collection would be assigned should also be considered by the relevant parties. There are human service and social welfare programs other than TANF that could be considered— excluding those concerned specifically with health programs or programs for the aged. Such programs include child care, child support enforcement, child protective services (foster care and abuse and neglect reporting), child developmental

disabilities programs, Low-Income Home Energy Assistance Program, and community service programs. These programs would all be candidates for inclusion in a centralization and reorganization of responsibility and authority for data collection.¹

Whichever organizational entity is assigned these functions, the entity should be separate from other programmatic and policy agencies within DHHS. Independence is required in order to confer credibility of the data with both data suppliers and data users. As noted elsewhere (National Research Council, 2001b; Hotz et al., 1998), the integrity of data can be compromised in situations in which the same agency that collects the data is also required to use the data to administer sanctions and other administrative actions. In order to maintain the integrity of the data, the agency needs to maintain a separation from real and perceived political partisanship and ensure data providers that their data will be kept confidential.

Data needs for monitoring the well-being of populations relevant to these programs and for evaluating these programs are on-going and will continue to change as the programs change. The panel believes that the short- and long-term data needs will be best met if some organizational entity within DHHS is given responsibility and authority for data collection activities.

Although the panel cannot be more specific on the type of entity to be assigned responsibility or its programmatic coverage, the panel can do two things. First, in this section we describe how these other statistical agencies operate to illustrate how the functions that are required for human service and social welfare programs are addressed in other areas. In the next section we list the functions that need to be performed by the organizational entity, whatever form it takes.

In the area of health, the U.S. Department of Health and Human Services has the National Center for Health Statistics (NCHS); in the area of employment and earnings, the U.S. Department of Labor has the Bureau of Labor Statistics (BLS);

and in the area of education programs, the U.S. Department of Education has the National Center for Education Statistics (NCES). The data collection functions needed to be fulfilled for human service and social welfare programs are quite similar to those performed by these agencies. Indeed, whatever organizational unit within DHHS is assigned the new functions would necessarily work with these agencies and others. One reason for establishing separate statistical structures is the need to involve other departments and agencies in data collection and development activities (National Research Council, 2001b). For social welfare programs data, there is a need to coordinate across many departments and levels of government.

NCHS was established administratively in 1960 (by law in 1974) with a broad mandate to collect, analyze, and disseminate data on a wide range of health-related topics. To meet its mission of monitoring the nation’s health, NCHS fields a complementary set of data collection mechanisms to meet current and emerging health data needs. NCHS coordinates the national vital statistics system, working with states to produce the nation’s official birth and death statistics. NCHS also supports the development of state and local data through the use of telephone survey methodology linked to existing household surveys in conjunction with the Healthy People process, which sets and tracks progress in reaching health objectives at the national, state, and local levels; and through research and collaborative efforts to adapt and apply national data sets and methodology to state-based applications.

The BLS is a federal statistical agency that collects or sponsors the collection of, processes, analyzes, and disseminates statistical data on labor markets and related topics to the public, Congress, other federal agencies, state and local governments, business, and labor. The BLS has long-standing relationships with cooperating state agencies in the production of statistical data, specifically to produce economic statistics on such topics as employment and worker safety and health. The BLS contracts with the Census Bureau to conduct several surveys, for example the CPS, which is used to produce the monthly unemployment statistics. Timeliness of these data is often of utmost importance, since the unemployment indicators are required for monitoring the national economy. Working together, both agencies are able to meet the timeliness demands as the unemployment figures must be released within 22 days after the reference period for which they are collected.

NCES has overall responsibility for planning, design, statistical analysis, reporting, and dissemination of elementary, secondary, and postsecondary education, and library surveys at the national, state, and local levels. Its mandate is also to ensure that statistical quality and confidentiality are maintained. NCES sets the statistical standards, administers technology support programs, and provides state-of-the-art technology and statistical support to federal and nonfederal organizations and entities involved in statistical work in support of the Department of Education. In addition, the staff develops and operates a system of licensing for

individuals and organizations to acquire access to confidential data for statistical purposes.

A notable commonality in the functions of these three agencies is an explicit mandate to coordinate activities with state counterparts and, in many instances, provide funding support for state data collection.² NCHS operates the cooperative federal-state vital statistics program referred to above. BLS cooperatively works with state employment security agencies to collect and provide data for programs such as Current Employment Statistics and Local Area Unemployment Statistics. Under this cooperative agreement, the states collect the data and provide them to the BLS, while BLS provides most of the funding for the data collection and defines the procedures for collecting the data, the data are then useful and comparable for state-level estimates. NCES works cooperatively with states to produce comparable data on elementary and secondary school statistics. Although NCES does not provide funding to states for this data collection, it does have funding for a technical assistance program for helping states produce high quality data that are comparable and timely. NCES also sponsors an annual data conference on educational statistics in conjunction with its National Forum on Education Statistics, where representatives from all states and from federal agencies with interests in educational statistics meet to discuss data collection activities.

Most of these agencies and others contract out survey data collection efforts to the Census Bureau or to private survey organizations. Examples of surveys contracted out to private organizations include the National Survey of Family Growth, which is contracted out by the NCHS; the Child Welfare Panel which is contracted out by ACF; the Early Childhood Longitudinal Study cohorts and the National Education Longitudinal Survey, which are contracted out by NCES; and parts of the Medical Expenditures Panel Survey, which are contracted out by the Agency for Healthcare Research and Quality. Many of these surveys involve complex longitudinal designs with which private survey organizations have more experience than the Census Bureau.

State data collection and coordination functions must necessarily be a part of the administrative responsibility for human service and social welfare programs. As in all federal-state cooperative efforts, this will create a need for organizational entities at the state level to be responsible for coordination of data collection activities with the federal government and with other states. Cooperatively developing data programs is necessary, and the DHHS entity should provide both technical assistance and some funding for affiliated state statistical centers. Indeed, such a federal-state program may require the creation of new state agencies to work with the federal government and to ensure that state-level data relevant to human service and social welfare programs are made available. One possible

approach is for the federal entity to create a “benefits reporting area” or “human services reporting area” composed of a few states with well-developed social welfare and human services data systems who would agree to jointly work towards a common framework for data collection and reporting. Over time, more states could be added to the reporting area. Federal birth and death registration data developed this way; first with a few states and with federal financial assistance, and then gradually expanding to include all states. Other possible arrangements for state-level data collection systems should be considered as well.

We provide here an itemization of the functions that would need to be performed by whatever new or existing organizational entity within DHHS is given responsibility and authority to carry out the mission.

National Surveys The organizational entity that is assigned responsibility would be the primary sponsor of the national surveys used to monitor and evaluate human service and social welfare programs and, in general, content related to the low income population. It would contract with the Census Bureau or with private survey organizations to conduct these surveys. These include the Survey of Income and Program Participation and the Survey of Program Dynamics, and perhaps parts of other surveys, like the topical modules in the Current Population Survey that cover social welfare program topics. As the entity with lead responsibility for content and design of these surveys, it would also work with other agencies that have interests in these surveys. It would also explore the linkage of national-level administrative data to the national survey data that address social welfare program topics.

Administrative Data The development and management of a cooperative welfare and social statistics data and information effort with the states would also be a needed function. Existing or new state statistical agencies should be full partners in this effort. Funding or financial incentives for the states to provide data to the federal agency and determining the form and content of the data submission should also be part of the responsibilities of the federal authority. Periodic reporting would be part of this program. Benefits Reporting Areas should be considered.

The development of standards for the use of administrative data for research purposes is an additional needed function. These standards should include definitions of services and benefit units, recipients and case members, data formats, and processes for documenting administrative data files.

In order to promote sharing of data resources for welfare and social statistics research and evaluation, coordination with other federal and state data collection agencies would also be required.

Leadership in advancing the use of and accessibility to all data provided by the states to DHHS for monitoring and social welfare program evaluation purposes is another important function.

Technical Assistance Another need is the provision of technical assistance to states on the use of administrative data and on the development, conduct, and analysis of surveys. The technical assistance could be used as a tool to promote the goals of comparability, improved data quality, data linkages, and data security and access.

Reports The federal entity should have responsibility for producing periodic reports on topics related to social welfare program utilization and the well-being of those who utilize these programs. One set of reports would be based on the data submitted by the states through the cooperative data collection effort mentioned above. It should also collect and publish social welfare program rules and policies, particularly for TANF and related separate state programs, for every state and every sub-state area where appropriate.

Data Archive for Continuing Research Needs A leadership role is needed in developing data archives on particular topics for use in social welfare program evaluation and research. Archives may include state surveys and administrative data, for which the agency would be responsible for preparing the surveys or administrative data for use by researchers. Maintaining an archive of welfare policies and programs description data throughout the states, and where relevant, in local areas, should also be a responsibility.

The principles and practices of statistical agencies are described more fully in National Research Council (2001b). The panel strongly believes that following these principles and practices is a necessary condition for addressing our recommendations regarding data collection.

A federal-state data collection system, as the one proposed above, will not develop overnight. It will require strong leadership and sustained support at both the federal and state level. If the trend of devolution persists, the need for and benefits from such a system will continue to grow.