National Academies Press: OpenBook

Evaluating Welfare Reform in an Era of Transition (2001)

Chapter: 5. Data Needs and Issues

« Previous: 4. Evaluation Methods and Issues
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

5
Data Needs and Issues

Successful monitoring and evaluation of welfare reform are not possible without good data, regardless of how clearly the questions of interest are delineated and how strong the available evaluation methodologies are. Good data are therefore critical to studying welfare reform.

In Chapters 3 and 4 we showed that there are a wide variety of questions of interest and that answering those questions requires a number of different methods. It should not be surprising that addressing the questions of interest for monitoring and evaluating welfare reform will therefore require use of multiple types of data from multiple sources. We categorize data sources into four generic types for our discussion. First are household surveys, both national and state level, which have been the data sources that have informed much of what is known about the low-income population. Second are administrative records from social welfare and other programs, which are a somewhat newer and emerging data source for studying welfare reform. Much of the new administrative data is available at the state level, but there are also a few federal-level data sets. Third are data describing policies and programs at the state and local levels. Fourth are qualitative data, another source of data that are increasingly being used in policy evaluation. Together, these four types of data constitute the data infrastructure for monitoring and evaluating welfare reform.

Good data have many characteristics. They have reasonably good coverage of the population in question. They contain measures of the key variables of interest for welfare reform study, either characteristics of policies or of individuals and families. They are reasonably accurate and contain few response errors, understatements, or missing values. Good data are also available for a reasonably

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

long time frame and are comparable across time. The sample sizes in good data sets are large enough for reliable statistical estimation, in many cases at the state and local level. Finally, for the purposes of many welfare reform studies, though not all, good data are comparable across states and use the same concepts and definitions, so that valid cross-area comparisons can be made.

What constitutes good data for addressing one question of interest may not suffice for another question of interest. Correspondingly, different evaluation methods require different types of data with different strengths and weaknesses.

This chapter describes the various sources of data available for welfare program research. In doing so, we discuss their potential use for addressing the research questions of interest using the available evaluation methods, their use in current studies under way, and we compare them with the characteristics of good data. We also include a section on data confidentiality that has implications for almost all data collection.

We note that most of these recommendations for improvements are specific to the current data infrastructure for social welfare program monitoring and evaluation. The panel concludes that this infrastructure has limitations and that the devolved nature of social welfare programs has exacerbated the limitations. The main limitation is that no agency within DHHS has the specific responsibility to collect data to monitor the well-being of the low-income population nor for evaluating the effectiveness of social welfare programs. The panel believes that this responsibility needs to be allocated to some administrative entity within DHHS to coordinate data collection activities at the federal level and to work with states to coordinate data collection activities at the state and local level. The final chapter of this report discusses this need in more detail. However, many of the specific recommendations for data improvements made in this chapter could be addressed more easily if this administrative authority is assigned. Therefore, in discussing these specific recommendations, we highlight areas where the existence of such an authority will help spur data improvements.

Devolution has had important consequences on data needs for the study of welfare reform. It has resulted in a proliferation of different programs in different states and localities around the country, each mixing a different bundle of reform components and strategies and each targeting somewhat different populations. Adding these differences to existing cross-state variation in Medicaid and other welfare programs and to ever-present differences in labor markets, demographic profiles, and general socioeconomic environments, the demands for state-level and local-area data have grown tremendously.

Devolution lies behind many of the recent developments in data collection for welfare reform studies and behind much of our discussion here. It affects the value of national-level surveys which have less to contribute in the current wave of reform than they had in past reforms. The number of state-level and local-area surveys, which historically have been quite rare, is growing. The value of state-level administrative data, which have the potential to capture state and local-area

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

details in a way that has not been important in past reforms, is also growing. At the same time, the comparability of these state and local data sets is of greater concern if generalizations about the consequences of reform are to be understood on a broader basis. Finally, it affects the need to collect state and local area program and policy information itself, a need which is now much greater.

SURVEY DATA

Data from question-and-answer surveys have been heavily used for social welfare program monitoring and evaluation in the past and will continue to be useful for future studies. This section of the chapter discusses the strengths and limitations of existing surveys for monitoring and evaluating social welfare programs and gives the panel’s recommendations for improving survey databases. It covers national-level surveys, which are designed to be representative of the U.S. population, and state- and local-level surveys, which are designed to be representative of state or local populations or subpopulations (like welfare leavers). The merits of particular surveys for particular purposes are discussed in terms of population coverage, sample size, content, nonresponse, response error, and periodicity.

National-Level Survey Data

There are several national-level surveys that are relevant to welfare program monitoring and evaluation. They cover such content areas as income, earnings, employment, program participation and benefit receipt, adult and child well-being measures, family structure, and demographic and other background information. The surveys discussed here (and summarized in Appendix D) include the long form of the decennial census, the March Supplement of the Current Population Survey (CPS), the American Community Survey (ACS), currently in the development stage, the Survey of Income and Program Participation (SIPP), the Survey of Program Dynamics (SPD), the National Survey of America’s Families (NSAF), the National Longitudinal Survey of Youth (NLSY), and the Panel Study of Income Dynamics (PSID). The CPS, ACS, SIPP, SPD and the census long form are all surveys funded by the federal government and conducted by the U.S. Census Bureau. NLSY is privately conducted but funded by the U.S. Department of Labor and the National Institute for Child Health and Human Development of the U.S. Department of Health and Human Services. PSID is conducted by the University of Michigan and supported by grants from the federal government and private foundations. NSAF is conducted by the Urban Institute with private funding. The NLSY, PSID, SPD and SIPP are longitudinal data and so have the added feature of tracking the changes in the well-being and outcomes of sample members over time. Table 5–1 contains basic summary information about these surveys.

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

Although these surveys all include content relevant to welfare reform evaluations, most are conducted for purposes other than welfare population monitoring and evaluation. Only the Survey of Program Dynamics has as its primary purpose the evaluation of welfare reform, although the SIPP, which was established before PRWORA, also has a special role as it was created to measure government and welfare program participation.

Population Coverage

National-level surveys are designed to produce analyses that are representative of the national population and, hence, are useful for producing estimates of the well-being of the nation as a whole.1 Most of these surveys are not, however, representative of smaller geographic areas, which is a significant disadvantage for studying welfare reform in an era of devolution. Data from the census long form are representative of state and local areas, but they are only produced once every 10 years, and so are not appropriate for timely monitoring or evaluation purposes. The ACS will be representative of smaller areas on an annual basis, and, hence, will be a major improvement in providing state and local level data on a far more timely basis. The ACS will be representative of states, large counties and governmental units with populations over 65,000. Eventually, multiyear averaged data representative of smaller areas will also be produced. Other national-level data sets (SIPP, SPD, NLSY, PSID) are of limited use for state-level monitoring because the state sample sizes are too small for precise estimates of state-level measures. The March CPS is large enough to produce annual state-level estimates, but the precision of estimates in most states is low.2 Other national surveys are representative of some states. For example, the Urban Institute’s NSAF is designed to be representative in 13 states (Alabama, California, Colorado, Florida, Massachusetts, Michigan, Minnesota, Mississippi, New Jersey, New York, Texas, Washington, and Wisconsin), though sample sizes for the low-income welfare participant population per se are quite modest. The SIPP and SPD are not representative of all states, but sample sizes are large enough in some states that state level estimates of outcomes can be produced with reasonable precision.

1  

Except for the census long form, these surveys exclude institutionalized persons, much of the military, and the homeless populations, although the NLSY and PSID studies do follow sample members in and out of institutions. The NLSY79 is representative only of those aged 14–22 in 1979; the NLSY97 is nationally representative of youths aged 12–16 in 1997. The NSAF is representative of the nonelderly population.

2  

The Census Bureau has funding to improve the precision of state-level estimates of the number of children with health insurance coverage by family income, age, race and ethnicity. Initial plans call for a significant increase in the sample size of the March Supplement, which should enhance the use of the CPS for state-level monitoring and evaluation.

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

TABLE 5–1 Key Features of National-Level Surveys Relevant for Monitoring Low-Income Populations and Evaluating Welfare Reform

Feature

2000 Census Long Form

American Community Survey (ACS)

March Current Population Survey (CPS)

National Longitudinal Surveys of Youth (NLSY79)

Populations Represented

U.S. households and individuals

U.S. households and individuals

U.S. civilian noninstitutionalized population age 15 and over

Youth aged 14–21 in 1979

Levels of Geography

National, state, local

National, state, local

National

National

Social Welfare Program Content

Limited benefits information, specifically public assistance benefit amount and SSI benefit amount; income

Limited benefits information—includes SSI, food stamps, cash assistance, and housing; income

Moderate benefits information—includes AFDC, SSI, food stamps, cash assistance, school lunch, and public housing; income

Extensive benefits information—includes AFDC, food stamps, cash and other public assistance; child outcomes; income

Sample Size and Design

Cross-section of approximately 18 million housing units in 2000

“Rolling” cross-section of three million households per year

Rotating panel design of 50,000

Panel of approximately 13,000 initially; roughly 12,000 in 1985 when discontinued interviewing the military sample

Oversampling

Small governmental units

Small governmental units

Hispanics

Hispanics, blacks, economically disadvantaged non-blacks and non-Hispanics (until 1990), enlisted military youth (until 1984)

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

Periodicity

Once a decade

Monthly

Annual (income supplement)

Annual until 1994 and biennial since

Data Collection Mode

Mail survey, personal follow-up for nonresponse

Mail survey, phone follow-up, then personal follow-up for one-third of mail and phone non-respondents

First and fifth interviews in person; other six interviews by phone

Personal interviews, except in 1987 when phone interviews were conducted due to budgetary constraints

Response Rates

1990 mailback response rate=60%; 2000 mailback response rate=54%

Weighted response rate of more than 95%

Until recently, response rates have been quite high. In recent years, they have been in the 80–82 percent range.

Round by round response rates are high. Cumulative retention rate through 1998 is 84% of original sample (not adjusting for mortality)

Data Release Dates

Long-form data planned to be released in 2002 for 2000 census

Goal is to publish six months after data collection

Income and poverty data published for nation and population groups 6 months after data collection; limited data published for states on the basis of 3-year averages

Most recent data available in 2000 was from 1998 survey; publish update biennially

New Features

Long form may not be included in 2010 or later censuses

May replace census long form

Recently received funding to expand sample size for state estimates of low-income children not covered by health insurance

NLSY97 began in 1997 with nationally representative sample (oversample of blacks and Hispanics) of roughly 9,000 youths aged 12–16 years old

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

Feature

National Survey of America’s Families (NSAF)

Panel Study of Income Dynamics (PSID)

Survey of Income and Program Participation (SIPP)

Survey of Program Dynamics (SPD)

Populations Represented

U.S. civilian noninstitutionalized population under age 65

U.S. civilian noninstitutionalized population of households and of low income families

U.S. civilian noninstitutionalized population

U.S. civilian noninstitutionalized population in 1992–1993

Levels of Geography

National and 13 states

National

National

National

Social Welfare Program Content

Extensive benefits information, including AFDC, SSI, food stamps, WIC, and school lunch; child outcomes; income

Extensive benefits information, including AFDC, SSI, food stamps, low income health services, and housing subsidies; child outcomes; income

Extensive benefits information, including public assistance, food stamps, school lunch, health insurance, and WIC; income

Extensive benefits information, including AFDC, SSI, food stamps, WIC, and health insurance; child and adolescent outcomes; income

Sample Size and Design

Repeated cross-section of 48,000 households

Panel, originally 4,800 families and 6,434 as of 1999

Panel, size has varied from 11,000 to 37,000 households

Panel of 18,500 households for 1998

Oversampling

Children and low income

Blacks; Hispanics in 1990–1995

Low income starting in 1996

Children and low income

Periodicity

1997 and 1999; planned for 2002

Annual until 1997 and biennial since

Every four months

Annual

Data Collection Mode

Telephone for those with access to a phone; in person and by cell phone for people in the area sample without a telephone

In 1999, 97.5% by telephone; all interviews used computer-based instruments

First, second, and one interview in each subsequent year of a panel in person; other interviews by phone

In-person and two self-administered surveys, one for adolescents; will conduct reinterview

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

Response Rates

Overall response rate for adults was 62% in 1997 and 65% for children.

97–98.5% annually (mortality-adjusted) PSID does not attempt to interview attriters. Cumulative response rate was 39% in 1994

91–95% households to first wave (cumulative response rate was 69% by wave 8 of 1996 panel)

82% in 1997, 85% in 1998, 1999, 2000. Cumulative non-response was near 50% by 1998. Since then, attempts to interview noninterviews from previous years were made and the cumulative response rate has been not risen.

Data Release Dates

First data from 1997 released in January 1999

1999 data were added to the Wealth Files in February 2000; most recent data in all other files is 1997

Historically, one to two year (or more) lag from data collection to publication

1997 and 1998 data available. Longitudinal file for the years 1992–1998 will be released in the summer of 2001.

New Features

1999 survey incorporated ways of measuring changes in child well-being

Child Development Supplement (CDS) began in 1997 with in-home interviews of 3,500 children aged 0–12 years with oversample for blacks; these children will be followed into adulthood

Requested funding to expand sample size and number of panels and to implement state-representative design

Will be conducted through 2001 to collect data that enable evaluation of the 1996 federal welfare reform legislation and its impact on the American people

SOURCES: Information from Brick (2000) and National Research Council (2000b).

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

National-level surveys have experienced some problems with undercoverage. The CPS has some population coverage problems, especially for specific subgroups that might be of interest for welfare program research. The average monthly coverage ratio (the ratio of the CPS population estimate compared with the census-based population estimate) for the CPS in early 1996 was 0.93. But for specific subgroups of black women between 16 and 39, the coverage ratio ranged from 0.82 to 0.87 (U.S. Census Bureau and Bureau of Labor Statistics, 2000). Continuing SIPP panels do not represent new entrants into the population such as immigrants and people returning from institutions (e.g., jail or longer term substance abuse facilities) which are relevant to welfare policy studies.

Population Subgroups

The national-level surveys are aimed to be representative of the population as a whole, but this implies that sample sizes for the population subgroups relevant to welfare reform, namely, single mother families who have lower incomes or welfare recipients specifically, may be problematic. For broader subgroups, such as single mother households, many national-level surveys are, in general, large enough to produce reliable estimates of well-being for the nation as a whole but rarely can they provide state-level reliable estimates.3 Moreover, studying specific subgroups who may be of interest for welfare reform, such as immigrants, disabled adults, or families with disabled children is more difficult, for the sample sizes in national-level surveys are almost always quite small for these groups, even for the nation as a whole.

Subgroup analysis is necessary not only for description and monitoring but also for nonexperimental evaluations of the overall effect of welfare reform and of the broad components of reform. Of the currently available data sets, only the CPS has the sample size and statistical power for needed subgroup analyses, and even its usefulness is limited to estimating the overall effect of PRWORA and only for low variance outcomes, like the program participation rate (see the discussion of power in Chapter 4). The ACS, as it is currently being developed, has considerable potential for use in such cross-area subgroup analyses of broad components of reform, but it is untested.

Nonresponse

A major threat to the representativeness of all surveys is nonresponse. Nonresponse may bias estimates of outcomes if those who do not respond are systematically different from those who do respond. In a longitudinal setting, the inability to reinterview families for multiple waves of surveys may also cause bias problems. For surveys of low income populations, there may be particular

3  

Table 5–1 shows which subpopulations are oversampled in national level surveys.

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

reasons to suspect there is bias due to nonresponse and attrition: it is likely that factors making it difficult to interview and reinterview respondents (in a longitudinal setting) may be correlated with their outcomes. For example, a lack of stable residence may indicate financial trouble for a family and may make it more difficult to locate a survey sample member. Homeless persons are rarely included in survey sample frames, as we noted earlier, and longitudinal survey respondents who become homeless are generally lost for future reinterview attempts.

Weighting and imputation procedures can potentially reduce nonresponse biases although they can rarely eliminate them (see Kalton and Kasprzyk, 1986 and Little and Rubin, 1987). With specific attention to surveys of low-income populations, Groves and Couper (2001) discuss survey design considerations for reducing nonresponse and nonresponse adjustments and Mohadjer and Choudhry (2001) provide more detail on weighting adjustment procedures. Incentive payments to encourage sample members to respond to surveys have also been effective in increasing response rates in surveys. Initial evidence from a small number of experiments further suggests that incentive payments may be particularly effective with low-income populations (Singer and Kulka, 2001). There has been some movement towards using incentives payments for the SIPP and SPD.

Response rates in the key national-level surveys vary considerably. CPS response rates are around 94–95 percent each month, although the response rates for the March CPS Supplement are a little lower.4 The ACS is still undergoing field tests and response rates are not available. However, in a 1996 test in four sites, the weighted response rates for the ACS were about 95 percent. The NSAF, which oversamples low-income households, had an overall response rate of 70 percent for the 1997 round and about 64 percent for the 1999 round (Safir, Scheuren, and Wang, 2001).

For the longitudinal surveys, nonresponse and attrition over multiple waves is a significant threat to data quality. For the SIPP and SPD, response rates in the initial waves were high (between 91 and 95 percent for first panels of SIPP from 1984–1996 and 91 percent for the first wave of the SPD—which corresponded to the 1992 and 1993 panels of SIPP), but many first-wave respondents in both surveys have not been reinterviewed. By the eighth wave, the cumulative nonresponse rates for the 1984–1991 panels were between 21–22 percent, 25 percent for the 1992 and 1993 panels and 31 percent for the 1996 panel. This attrition seems to be the result of refusals, rather than the inability to track sample members (U.S. Census Bureau, 1998). The SPD sample is comprised of the 1992 and 1993 SIPP panels. The first SPD survey, the “bridge survey” in 1997, attempted

4  

For example, in the 2000 CPS March Supplement, the response rate for the basic monthly labor survey was just over 93 percent, but 8 percent of the basic sample did not respond to the supplement and so the total response rate was 86 percent (U.S. Census Bureau and Bureau of Labor Statistics, 2000).

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

to interview all 1992 and 1993 SIPP panel members who had responded to each intervening SIPP wave, which was only 73 percent of the original 1992 and 1993 SIPP sample members. The bridge survey interviewed 82 percent of those households. For the next SPD interview in 1998, budget constraints resulted in a decrease in the sample size. Of the eligible households for the 1998 collection, 89 percent were interviewed. Accumulated over these waves, the response rate through the 1998 survey was only 50 percent. The Census Bureau has explored the degree to which attrition has affected the representativeness of the sample; it concluded that in comparing estimates of population well-being measures and characteristics, the SPD produces estimates similar to those of the CPS (Weinberg and Shipp, 2000). (We discuss this further in the section on longitudinal data.) Cumulative nonresponse has also been a problem for the PSID. The NLSY 1979 cohort has had good cumulative response rates, which can probably be attributed in part to keeping sample members in the sample even if they are nonrespondents for one or more waves of the survey.

Timeliness

In order to be useful for continual monitoring of the well-being of welfare prone populations, the data used in monitoring studies should be produced on a regular basis. The March CPS and SPD collect data on an annual basis. The decennial long-form collects data every 10 years. The NSAF collected data in 1997 and 1999. A fully implemented ACS will collect data every month. The SIPP collects data every 4 months.

The timeliness of the release of the data is just as important, and is a severe limitation for some surveys. While the March CPS data are produced on a very timely basis (the data are usually available in the fall after collection), other data sets are not and are thus less useful for monitoring the well-being of the low-income population than they could be. This has been an especially significant problem for the SIPP, for which data release has often taken much longer than a year. For example, while the core data from all 12 waves of the 1996 panel have now been released, only the first few topical modules have been released as of yet, and no longitudinal file has been produced. Data release for the SPD is only slightly better; 1998 data were available in early 2001 and the first longitudinal file of the data set, covering 1992–1998, is scheduled to be released mid-2001. Thus, for two of the key surveys for monitoring welfare program participation, only very limited post-welfare reform data are available in early 2001, nearly 5 years after the reforms were enacted.

Survey Content

Another important aspect of the data for studying welfare reform is content. All of the national-level surveys discussed collect basic information relevant to

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

welfare and low-income populations (employment, income, and public assistance benefits). Some are more inclusive in their coverage of income (CPS, NSAF, NLSY, NSAF, PSID, and SIPP) and include questions about many resources that might be available to the sampled individual or household, while other surveys do not collect as much detail on resources (the long form and the ACS). Some also include more detailed measures of employment than others (again, the long form and ACS are short surveys that do not include many questions detailing employment). Obtaining more detailed information on these two types of measures is important in many monitoring and evaluation settings in order to get a fuller picture of well-being.

Receipt of public assistance benefits is also an important part of these surveys. The census long form covers very limited program benefit receipt information (cash assistance and supplemental security income [SSI]). The ACS collects a bit more program benefit information (receipt of and amount of cash assistance, SSI, food stamps, public housing, and energy assistance). The March CPS collects more detailed information about receipt of public assistance benefits, but it does not collect much information about noncash benefits, such as job search assistance, wage subsidies, or transportation benefits. The major surveys that provide the most detail on program participation and benefit receipt are the SIPP, SPD, and NSAF.

The devolved and ever-changing nature of welfare programs has made it more difficult for all of the national-level surveys to capture the welfare program benefits received by survey respondents. Under AFDC, there was a common program name for benefits across states, so that a common question naming the common benefit (AFDC) could be asked of all respondents, nationwide. However, there is now no common name for cash assistance benefits across all areas, which makes it more difficult to design a survey question that is relevant for respondents in different states. A further complication is that cash assistance is only one of the entire range of services that states now offer low-income families. This wide range creates a major barrier for surveys trying to measure benefit and service receipt and has so far limited the use of national-level surveys to address both monitoring and program evaluation questions. Efforts to incorporate survey questions to probe sample members in different states about the benefits and services received have been hindered by the slow development of good data on programs and policies enacted in each state. Furthermore, recognizing the need for new questions and then developing, testing, and incorporating the questions into the major national surveys takes time and adds to the problem these surveys have in keeping up to date with the changes to state programs. As a result, these surveys are not well suited to fully capture benefit receipt and program participation.

Some of the national surveys collect considerable data on other topics that are relevant for monitoring the well-being of low-income and welfare populations. Most of the longitudinal national surveys collect information on moving

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

and on the state of residence. Therefore, it is possible to measure migration across the country with the longitudinal data sets. The 1999 SPD included a topical module on child well-being that will be repeated in 2001. The 1998 SPD included a module that interviewed adolescents that will be repeated in 2001, although the 1998 module had sizable nonresponse (Bass and Downs, 1999; Downs and Bass, 1999). The NSAF also collects extensive measures of child and adolescent well-being and a great deal of information on health and health care. These surveys are unique in their national coverage of child outcomes, and so will be valuable for monitoring child well-being on a national level. The NLSY and PSID both collect extensive child well-being measures, but these surveys are quite small for some purposes.

Measurement Error

Measurement error for key concepts of interest for welfare program monitoring and evaluation, such as income, benefit receipt, employment and earnings, is also a concern for the quality of data from national-level surveys. For income and earnings measurement in SIPP and CPS, there is some evidence that these are measured well (see Hotz and Scholz, 2001 for a review). Reporting of program participation and benefit levels in national surveys is more problematic. Marquis and Moore (1990) found small differences in overall participation rates in transfer programs when comparing SIPP data with administrative records. However, reports of participation in the SIPP were underreported when comparing the individual survey responses of those who (according to administrative records) actually participated in the program. Underreporting of food stamp participation in SIPP has also been found (Bollinger and David, 2001). For the CPS, underreporting of welfare program participation has been documented for some time, although the extent of underreporting varies from year to year (Bavier, 1999). There is some evidence that the amount of benefits reported (for both AFDC and food stamps) is getting worse (Primus et al., 1999). Moore et al. (1997) review the literature in general on reporting of income from programs and include a discussion of early assessments of reporting for the CPS and other surveys. Mathiowetz et al. (2001) and Hotz and Scholz (2001) both review the literature on survey reports of program participation and benefit levels in more detail. Accurately measuring program participation and benefit levels is likely to continue to be problematic for national surveys as the services offered and program names become more diverse across the country.

Longitudinal Data

For some of the questions of interest for welfare reform monitoring and evaluations identified in Chapter 3, there is a need for longitudinal survey data to track the same individuals and families over time. Longitudinal survey data can

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

be used to help understand the dynamics of welfare program participation as families move on and off public assistance as their life situations change. More broadly, longitudinal survey data can also help understand the dynamics of the income and economic status and the marital, fertility, family composition, and migration across localities or states.

The SIPP, SPD, PSID, 1979 NLSY and 1997 NLSY are all longitudinal data sets. The SIPP interviews sample households every 4 months so that data on marriage, fertility, and family composition are collected over time frames that are short in relation to the frequency and duration of change. Income and economic data are collected every 4 months as well, but respondents report these on a monthly basis. Most of the other longitudinal studies collect information annually, obtaining most characteristics as of the interview date and a few variables for the past year (e.g., income). Thus, the SIPP is unique in providing short-term information on the dynamics of poverty, income, and family situations. Further, understanding the interplay between the family composition and economic situations of sample members (e.g., a decrease in household income after one household member moves or enrollment in a welfare program after the birth of a child) is more feasible with SIPP because these changes are measured over short time frames. Both of these features make the SIPP particularly useful for some monitoring and evaluation questions that require longitudinal data.

Of the other longitudinal data sets, the SPD was specifically designed to study outcomes of families before and after the 1996 reforms (1992–2001). It has a large sample size compared with NLSY and PSID, but nonresponse, attrition, and sample size reductions after budget cuts have hurt the overall size of the sample. Although the Census Bureau has concluded that attrition has not severely hurt the representativeness of the sample in comparison with other national-level surveys, attrition rates are higher for sample members with lower incomes (Weinberg and Shipp, 2000). Cumulative attrition through the 1997 SPD for those with incomes of less than half of the poverty level was 53 percent, compared with 43 percent for those with incomes at the poverty level, and 35 percent for those with incomes twice the poverty level (Weinberg and Shipp, 2000). This analysis also found that the 1997 and 1998 SPD overall interviewed samples have significantly fewer high school dropouts (which is another subpopulation particularly relevant to welfare policy studies) than the 1997 and 1998 March CPS surveys. Thus, there is reason to doubt that this attrition is random and that the population of interest for studies of welfare reform are adequately represented in the SPD.

The NLSY and PSID surveys are both long-term longitudinal studies and have much information on behavioral outcomes that require a longer time frame to study (e.g., some child outcomes, life time family and fertility decisions, or intergenerational welfare dependency). They both collect extensive information on family formation and dissolution and on child-bearing—information that is important for monitoring these outcomes over time. However, both have small

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

sample sizes and the NSLY surveys do not contain data on people outside of the age cohorts sampled.

Use for Monitoring and Evaluation

The national-level data sets currently available are of considerable value for monitoring welfare reform, but still fall short on some criteria. The data sets are of less value for addressing formal program evaluation questions.

The CPS is probably the best all-around national data set for monitoring the well-being of the adult population at the national level because it contains sufficient measures of most income and employment outcomes for individuals and families, is reasonably representative of the overall population and major subgroups of interest, is produced on a timely basis, and is available prior to PRWORA. However, it has many weaknesses as well. Its survey content relevant to welfare participation is quite limited, measuring only cash receipt and only over annual periods. Welfare receipt is also not measured at the same time as most other individual characteristics and, as a consequence, it is not possible, for example, to determine basic questions like whether families are working while on welfare. Further, welfare receipt is significantly underreported in the CPS. Finally, it is too small for state-level monitoring of welfare programs because there are too few observations of low-income, single-mother groups to produce estimates with acceptable levels of error.

The other national-level data sets are weaker for the monitoring function. Although the sample sizes are still adequate for estimating trends in well-being of the nation as a whole, they are inadequate for conducting extensive subgroup analysis and for state-level analysis. The SIPP has the advantage of more frequent periodicity of data collection and more extensive program participation coverage, but it has the significant disadvantage of being extremely slow in release, which greatly diminishes its usefulness for monitoring welfare reform. Another issue for monitoring is that the 1996 SIPP panel does not include some new entrants into the sample frame, (primarily immigrants and those who move from the institutional to the noninstitutional population). The 1996 SIPP also has differential attrition of higher and lower income sample members, which is a problem for monitoring income and poverty. The SPD has significant problems of nonresponse which may be correctable, but nevertheless reduce sample size, may introduce bias, and limit its ability to monitor outcomes. Panel data sets, such as the PSID and NLSY, are too small even for adequate monitoring. The NSAF is large and is representative of 13 states but have only been collected post-PRWORA (in 1997 and 1999 and next in 2002) and are not longitudinal in nature.

The usefulness of these surveys for formal evaluation is more limited. Their primary use is for nonexperimental evaluation of the overall effect of welfare reform at the national level (using pure time-series analysis or comparison-group

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

designs of the type discussed in Chapter 4). For these purposes, many of the data sets provide adequate sample sizes and at least the minimally necessary outcome and program participation information needed. However, for any analysis that is based on cross-state comparisons, the sample sizes in these data sets are either on the borderline of the minimum necessary or below it. The CPS is minimally adequate for estimating the overall effect of pre-PRWORA waiver evaluations, but neither the CPS nor the other data sets have sufficient sample sizes to reliably conduct evaluations of the incremental effect of broad components or evaluations of detailed strategies (see Chapter 4 and Appendix C).

In the absence of sufficient state-level household data sets, the ACS, if it is fully implemented and sustained, has the potential to be used in future program participation modeling at the state level that could assess the effects of future changes in broad components of reform. The survey will produce yearly state-level estimates of the number of individuals participating in broadly defined social welfare programs. It is not yet clear how well the questions included in the ACS will capture program participation, given the potential problems in measuring the wide array of services and benefits offered to poor families, which is a difficult problem for all the surveys. However, if measures of participation are sufficiently accurate, the ACS will be quite valuable for understanding changes in broad components of welfare policy and how larger macroeconomic and social conditions affect program participation outcomes. It is, however, not available for evaluation of the overall effect of welfare reform.

Using nonexperimental methods and national-level data sets to assess the overall effect of PRWORA and the effects of both broad and specific program strategies is also limited by confidentiality issues. Most national-level data sets do not allow researchers access to information that can identify where a sample member resides below the state level (although these data can be accessed with proper permission through the Census Bureau data research centers). Nonexperimental methods used to address evaluation questions about the overall effects of reform, along with the broad and specific effects of reform, must control for the larger program and economic environment faced by each sample member to separate out effects of these conditions from the effects of the policy in question. If the data cannot identify in sufficient detail where a sample member resides (localities, counties, or states), it is not possible to match data from other sources on local conditions to control for these conditions in the analysis. (We discuss these confidentiality restrictions on data below.)

Nationally representative longitudinal survey data are needed to address some monitoring and evaluation questions. For example, studies that evaluate the effect of policies on family formation use longitudinal data on individuals and families, tracking their behavior over time. Some specific nonexperimental evaluation methods require historical data on an individual level to control for individual characteristics that might be correlated with outcomes of interest (e.g., current labor force or program outcomes may be correlated with past employ-

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

ment or program participation histories). As Chapter 4 described, both time-series modeling and comparison group methods that use microdata require such longitudinal data to separate the effects of policies from differences due to individual characteristics.

Currently available longitudinal data sets meet some of these needs but fall short on others. The SIPP provides valuable information about the short-term dynamics of poverty, income, and family formation and composition that has a particular use for some policy questions. Some outcomes require a longer time frame with which behavior can be observed. All of the currently available national-level data sets have limitations in this respect. The SPD covers a longer time frame than SIPP, but it does not cover a long enough time frame to examine long-term outcomes, and has serious data quality issues as discussed above. The NLSY and PSID are long term, but are not large enough for precise estimates of outcomes for some populations of interest. Thus, another gap in the data infrastructure for welfare program monitoring and evaluation is the lack of a large, nationally representative longitudinal survey with welfare program-relevant content covering a long-run time frame.

Use in Current Welfare Projects

As we note in Chapter 2, several projects are using national surveys for monitoring purposes and are producing interesting results. The CPS has been used most often for these analyses. The SIPP and PSID have been used extensively for monitoring efforts, such as the ongoing series of reports on well-being and dependency issued by DHHS. However, the SIPP has been used mostly for pre-PRWORA monitoring because so little post-PRWORA data have yet been released. The NSAF has been used for a large number of descriptive studies of the welfare population as well. The SPD and ACS, as relatively new sources, have been used less. As for evaluation studies, the CPS has also been used for much of the caseload and econometric modeling we discussed in Chapter 2. Other surveys have been used less frequently for specific evaluation questions.

Conclusions and Recommendations

National-level data sets are particularly valuable for monitoring the well-being of the nation as a whole and for many relevant subpopulations of the nation. They are less valuable for the evaluation of welfare reform and welfare programs as they evolve. Overall, the current set of national-level surveys is inadequate for fully addressing the research needs for monitoring and evaluating welfare reform, and improvements will be needed to make them effective. Nonresponse is a significant threat to how well many of these surveys cover the populations of interest and, hence, limits their use for monitoring well-being. Furthermore, all the national-level surveys must deal with the new realities of

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

welfare programs, where there is no common set of definitions and terms for assistance and there is wide variation in states’ social welfare programs. The abilities of national-level surveys to fully capture all benefit receipt and program participation across the nation is hampered because of this. The lack of timely release of the key national surveys is also a threat to adequate monitoring of the population of interest. Evaluations of the overall effect of reform and the effects of different program components and detailed program strategies across states are also limited by sample size.

Conclusion 5.1 The panel finds that each of the major national household survey data sets most suitable for monitoring and evaluation has significant limitations in terms of sample size, nonresponse levels, periodicity, response error, population coverage, or survey content.

Although the national-level data sets have these limitations, there is a role for them to play in monitoring and evaluation of welfare programs. National-level surveys provide some of the best data for monitoring the well-being of the low-income population of the nation as the whole and for some broad subgroups of interest. They also often contain rich measures of child and adult well-being not found in other data sources. Some national-level data sources will be valuable for assessing the overall effects of welfare reform and the effects of broad components of reform. There are areas where improvement is needed, however, so that the key questions of interest for welfare reform research identified in Chapter 4 can be addressed.

Because a key purpose of monitoring studies is the early detection of changes in the population of interest and its well-being, it is essential that the data are produced on a timely basis. The SIPP is unique in that it is an on-going survey that provides detailed coverage of program participation and income for low-income populations and collects needed information on the short-term dynamics of program participation. Thus, it could be a very useful data set for monitoring the well-being and program participation status of the low-income population. However, the lack of a timely release of these data is problematic. The SPD also provides detailed information on program participation. It has a smaller sample, but follows respondents for a longer time frame than the SIPP and covers the years before and after PRWORA. Although it may be limited by nonresponse and attrition, it could be used for current and future monitoring purposes through the period over which the survey extends. However, the data release for this survey has been delayed. Because these data are not released in a timely manner, their value for monitoring purposes is significantly weakened.

Conclusion 5.2 Key national-level survey data sets used to monitor low-income and welfare populations are currently not being produced on a timely basis. The value of these data for monitoring low-

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

income and welfare populations would be enhanced if they are produced on a more timely basis.

Assessing the overall effect of welfare reform necessarily relies on data that were available at the time of the reform. National-level survey data, specifically the March CPS, is the best hope to address this evaluation question. As the analysis by Adams and Hotz shows (see Appendix C), the March CPS has reasonable sample size and power for detecting overall effects of reform on program participation. However, its power for detecting overall reform effects on outcomes with greater variance, such as employment and earnings, is limited. All other available national-level survey data sets are not large enough for assessing the overall effects of reform.

Nonexperimental methods of evaluating the effects of changes in the broad components of welfare programs, particularly those relying on cross-area variation, rely on national-level survey data. These data are important because they contain comparable measures of outcomes and other variables across states and because some collect longitudinal data on individuals before and after policy changes. However, sample sizes for conducting cross-state analyses is a serious limitation for these data. The CPS is the only currently available national-level data set that is a viable candidate, and its power for detecting the effects of broad components of reform is suspect. The ACS is a hopeful future alternative for cross-state analyses of broad policy components, but its power for detecting the effects of broad components and detailed components of reform is untested. Moreover, as it currently stands, the ACS does not collect much detailed information on program participation and benefit receipt that can be used to assess the effects of some specific program components or detailed strategies on outcomes.

Sufficient sample size in at least one of the data sets that measures program participation and benefit receipt is necessary so that reliable cross-state analysis of the effects of broad policy components can be conducted. This may mean that sample sizes in either the CPS or the SIPP will need to be increased substantially or that state-level supplements to these surveys are given more serious consideration. A promising development along these lines is recent funding for the Census Bureau to increase the March CPS, to the extent of almost doubling its size, to produce statistically reliable state-level estimates of the number of low-income children who do not have health insurance. The implications of this for evaluations of welfare reform need to be explored.

Recommendation 5.1 To improve the abilities of national-level survey data sets to measure the effects of changes in broad welfare program components across states, the panel recommends expansions or supplements to the CPS or other surveys.

The prototype American Community Survey has much to offer for welfare program evaluation and monitoring. Its key benefit is that it will be representa-

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

live of states and larger cities every year. It is likely to be large enough to use in conjunction with time-series and comparison group models for assessing the effects of future changes in broad policy components. It also includes broad measures of well-being and measures of welfare program benefit receipt and so it will be very useful for monitoring efforts, at the national, state, and city levels.

The ACS is currently in development stages and is being field tested throughout the country. Full funding has not yet been secured. The ACS has great potential for use in social welfare program monitoring and evaluation the survey and therefore deserves full funding and support.

Recommendation 5.2 A fully implemented and continuous American Community Survey has significant potential for use in future welfare policy research. The panel recommends that sufficient funds be devoted to fully implement the survey and that support for the survey at its currently proposed sample sizes is sustained over time.

The ACS does not and was not designed to collect detailed data on social welfare program participation. It was designed to provide general economic, social and demographic data to communities every year instead of the census long form. It includes several questions about public assistance benefits received by sample households, but it does not collect the more detailed program participation data that the SIPP and SPD, and to a lesser extent, the March CPS do. It does not, for example, ask about low-income child care benefits, transportation benefits, diversion payments, job search or job training benefits, all of which might be provided along with or instead of cash assistance. Furthermore, the reference period for program benefit receipt is the past 12 months, which is long enough to raise issues of the accuracy of respondent recall. Unlike the CPS, the ACS does not contain follow-up questions in the survey to serve as checks on the quality of reported benefits. A potential solution to the lack of detailed program participation data in the ACS is to use the population-based data from the ACS and link it to program-based data from state- or local-level administrative data sets that contain better program participation data. However, the ACS is a mandatory survey and is protected under Title 13, which means that individual-level data (which would be necessary for linking) cannot be released to anyone outside of the Census Bureau. One possible enhancement to the ACS is state-added supplemental questions.

The lack of sufficiently detailed questions on welfare program benefit receipt may mean that the ACS, as it is currently planned, will not reach its full potential for welfare program monitoring and evaluation. If sample size in the CPS is increased, the need for more detailed questions on program participation in a larger survey like the ACS is reduced. However, if the ACS is the only sufficiently large and reliable data source to use for nonexperimental, cross-state evaluations of welfare program components, this lack of detail in the ACS will be a serious limitation for national-level welfare policy analysis.

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

The ACS will have many competing demands for additional questions, and the number of questions that can be added to the survey is likely to be limited. However, because welfare program devolution has increased the need for state and local data, the panel believes that, with more detailed questions on program participation, the ACS could serve as a very useful data set for welfare program monitoring and evaluation.

Recommendation 5.3 The potential of the American Community Survey for evaluating welfare policies would grow considerably if the survey included more extensive questions about public assistance benefit and service receipt. The panel recommends adding more detailed questions on public assistance receipt to the survey questionnaire.

Even with more detailed questions on program participation, it is unlikely that the ACS, will be able to fully capture the wide array of benefits states and localities are offering welfare recipients. Indeed, capturing welfare benefit and service receipt is a problem for all national-level surveys. With state and local control over welfare programs, an increasingly wide array of benefits and services are now being offered. As noted above, there are no common names of, nor terms for, cash and noncash assistance programs and benefits across states. The Census Bureau, which conducts all of the federally funded national-level surveys relevant to welfare policy (March CPS, ACS, SIPP, and SPD), is aware of this problem in measuring welfare program participation and has taken steps to test new questions and incorporate them in surveys. The Census Bureau should be commended for beginning to address this issue, but it does not have the resources or expertise to do it alone. As a statistical agency, the Census Bureau has expertise in collecting survey data, but lacks it substantive knowledge of welfare programs and how states have implemented them. This expertise lies in DHHS and state welfare program agencies. This mismatch of expertise and responsibility has impeded data collection for program participation and is an area for which the proposed new organizational entity for collecting data relevant to social welfare programs outlined in Chapter 6 could be particularly beneficial in the long run. Short-term efforts will, however, need to rely on extensive coordination among the Census Bureau, DHHS, and state welfare agencies. Staff from the Census Bureau and from DHHS and other federal agencies offering assistance to low-income populations (e.g., U.S. Department of Agriculture, U.S. Department of Housing and Urban Development) should meet regularly with each other and with state welfare program agencies to stay on top of what states and localities are offering as part of their welfare programs so that survey questionnaires can better capture the program services and benefits received by survey respondents. Continued efforts by the Census Bureau to test and develop new questions for capturing program participation should be supported. In addition, the process by

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

which questionnaires are changed to reflect new program realities needs to be accelerated.

Recommendation 5.4 The wider array of services provided in social welfare programs and the variation in these programs across states both make measuring program participation and benefit receipt more difficult, especially on a national level. For national household surveys that measure participation in and benefits received from programs serving the low-income population, it is critically important to regularly and frequently review survey questions to keep in step with program and population changes. The panel recommends to the Census Bureau that more resources be devoted towards improving questions on program participation and benefit receipt to better capture program participation. The panel also recommends that DHHS work with the Census Bureau to develop mechanisms for regular communication with states to stay abreast of programmatic and implementation changes in the states.

State and Local Surveys

As states and localities have implemented their own TANF programs and have more control over designs of their programs, they now also have more of a vested interest in understanding the effects of programs and for monitoring population well-being. Thus, demand for state and local-level surveys has grown considerably.5

Most of the welfare-program-related state and local surveys that have been conducted thus far are not representative samples of the entire state’s population in the same way national surveys are of the national population. (An exception is NSAF, which is partly national and partly state specific and is representative of the states it samples.) Surveys of welfare leavers, for example, are one of the types of useful surveys being conducted, although, as we stressed in previous chapters, it is also important to survey stayers and other groups. Indepth surveys of the welfare participants or of the general low-income population in particular cities or neighborhoods can be useful because they provide depth in exchange for geographic breadth. Surveys of other special populations, such as immigrants, individuals with substance abuse problems, and others would also be valuable.6

State and local surveys confront issues of population coverage, coverage of subgroups, nonresponse, timeliness, survey content, and measurement error, just

5  

Appendix Table B-1 contains many examples of welfare program studies that use state and local level survey data.

6  

Matching these types of surveys to administrative data is an important issue as well (see below).

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

as national surveys do, although the relative importance and nature of the issues is not always the same. For population coverage, for example, generating a sample frame from the traditional counting and listing process of households at the block level is an expensive enterprise even for very local surveys. Also, to be effective for monitoring and evaluating programs serving low-income populations, surveys would need to oversample low-income households. Screening for low-income households can be an expensive task because initially contacted households may be reluctant to provide sensitive information about their income or because income is often measured with more error than other background variables otherwise used for screening (Cantor and Cunningham, 2001). Using random digit dialing telephone surveys instead of in-person surveys is an alternative that can cut expenses, but they raise issues of population coverage because a sizable proportion of the low-income population may be without telephones.7 Further problems are posed by caller identification screening and the use of multiple phones, although it is not clear how much of a problem these factors are for the low-income population.

State-level surveys of welfare populations (stayers and leavers) thus far have predominantly generated survey samples from lists of program participants—for example, persons receiving cash assistance or food stamps during a certain time period or persons in the control and treatment groups of a state-level experimental program. There are benefits to using such administrative lists for a sample frame (additional information from the administrative records on the universe of sample members is the key one). However, such a sample frame limits the population of interest to only those that participated in the program during that time. Therefore, results cannot be extended to the overall low-income population, which includes people who were not participating in the program at the time the sample was drawn.

Nonresponse is a serious issue for many state-level surveys of welfare recipients and other low-income families. Many of the early welfare leaver studies had low response rates (Acs and Loprest, 2001; U.S. General Accounting Office, 1999a). These populations are often hard to locate because they move more frequently than the full population (Groves and Couper, 2001; Cantor and Cunningham, 2001) and, as mentioned above, are more likely not to have telephone service. Conducting an interview may also be problematic for those with language barriers and those with literacy barriers. Contacting sample members can be problematic because contact information from administrative records (e.g., addresses and phone numbers) is frequently inaccurate. This lack of accurate initial contact information contributes to nonresponse in many of these studies (Cantor and Cunningham, 2001).

7  

Thornberry and Massey (1988) report that 30 percent of those in poverty do not have phones. Weiss and Bailar (2001) cite similar findings.

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

Like national surveys of low-income populations, error in measuring outcomes relevant for welfare program research, such as income, earnings, employment, and program participation is also an issue for state level surveys. State-level surveys must still address the problem of response error in collecting information on program participation, income, and other related concepts in general, although measuring program participation is likely to be less troublesome at the state level. Unlike the national-level surveys that must design questions about program participation for many states with many different programs, state-level surveys need only cover the programs in their own state. However, with the many different programs offered, survey respondents may have difficulty recalling each of the programs they have participated in and so may misreport such information on a survey. Data from state-level surveys can be linked to state-level administrative data to supplement survey data measures of program participation or to serve as a quality check of survey reports more easily than national-level data can be linked.

The lack of cross-state comparability of data limits the use of state-level surveys for cross-state monitoring and evaluation. In the current round of welfare reform, state-level surveys have been primarily developed for evaluations only in the state in which the survey is being conducted. Exceptions to this are the ASPE-sponsored studies of specific welfare populations (welfare leavers and divertees), for which a variety of states were given grants to follow these populations so that a national picture of the circumstances of these groups across the country could be developed. There was some coordination across these state- and county-level studies in terms of common definitions of populations and sharing of questionnaires. However, each state was inherently interested in different topics and so questionnaires were not coordinated. Furthermore, sample frames, designs, reference periods, and the timing and frequency of interviews varied across the surveys. Thus, while these surveys are more comparable than surveys of other welfare leavers, there are still issues of comparability that will make it difficult to compare data across states.

Use for Monitoring and Evaluation

An advantage of state and local surveys for monitoring and evaluation purposes, in comparison with national-level data sets, is that they are able to collect more detailed information relevant to the particular populations and programs of the state (e.g., information of particular interest for rural states or information on a particular state program). Most administrative data sets used in welfare program evaluation are state based. Therefore, matching state-level survey data to state-level administrative data is another advantage (see below). It is also more feasible to follow a local group of welfare recipients, leavers, or divertees or simply a sample of families in the low-income population in particular cities, counties, or other service areas at state or local levels.

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

A significant handicap at the state and local level thus far is the relative lack of experience of state governments in conducting and using survey information, as well as a lack of resources to conduct surveys. The expense and expertise required in traditional sampling frame methods makes most government agencies hesitate to use them, and they more often rely on telephone surveys or do not use survey data at all (relying on administrative data alone). More surveys have been conducted locally by private research organizations who are funded by foundations or federal-level research agencies. More needs to be done in this area, given the importance of state and local surveys for monitoring purposes.8 The quality of many of the telephone surveys that have been conducted is also rather low; these surveys need to be of higher quality if they are to be useful for monitoring and evaluation purposes.

State and local surveys used for evaluation purposes are thus far quite rare. For nonexperimental evaluations, cross-area comparisons within states require comparing different counties or different agency offices within states. Attempts to conduct surveys for such studies are rare. Within-area nonexperimental methods are also rare, although they are more feasible. This method of finding comparison groups for ineligibles or nonparticipating eligibles in a particular area has been used in the evaluation of other social programs and could be explored for welfare reform. Finally, there is an issue of whether surveys in multiple states could be used together to conduct cross-state nonexperimental comparisons, which would require sufficient comparability in the survey designs as to permit valid conclusions. This approach has not been attempted to date.

Surveys conducted in conjunction with experimental evaluations are more common, for those evaluations are almost always local in geographic coverage (many experiments use only administrative data, however). These surveys face the same nonresponse and measurement issues discussed above for surveys in general. However, with adequate resources and the use of experienced survey organizations, high-quality survey data in conjunction with experimental evaluations can be collected.

Use in Current Welfare Reform Studies

State and local surveys are only now beginning to be developed for welfare program and evaluation. Many of the welfare leaver studies described earlier in this report include state-level surveys of former welfare participants (see Acs and

8  

There are examples of federal support for state-based surveys for monitoring purposes in other fields. An example of particular relevance here is the Center for Disease Control’s Behavior Risk Factor Surveillance System, in which state health departments conduct or contract out monthly telephone surveys to track health trends and potential health problems. We discuss other examples in Chapter 6.

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

Loprest, 2001, for a review of leaver study surveys). There has been very little exploration into the quality of data from these studies: for example, whether the surveys are representative of their universe and how different measures of key variables, such as employment and wages, compare with measures from administrative data sources. For studies of formally and informally diverted populations (see Chapter 2), data issues for both survey data and administrative data are in some ways likely to be more severe. Sometimes very little information is gathered for cases that have been diverted, so there is very little information with which to track individuals in order to survey them. Some areas are using past participants in other social welfare programs who are not or have not participated in TANF to identify informally diverted cases. For these studies, there is the additional burden of finding these cases with what may be out-of-date information.

Surveys at the city, county, or local level are also being conducted as part of the Three-City Study, The Project on Devolution and Urban Change, and the Los Angeles Families and Neighborhood Survey. In a unique example of a national survey conducted at a state level, Iowa State University, in conjunction with the Census Bureau, conducted a modified SPD survey in Iowa (with modifications to questions that were of particular relevance to state policy makers) to explore the feasibility of conducting state-level surveys that could be integrated with the national-level surveys (Nusser et al., 2000). The state of California has also recently launched a large telephone survey to collect information about health and health care access. This survey will also collect information relevant to studying welfare policy.

Conclusions and Recommendations

Devolution has contributed to the growing demand for more localized data on low-income populations. States do not have a great deal of experience in sponsoring or conducting surveys, and thus far, data quality for some state-level surveys has been less than adequate. DHHS-ASPE has recognized the lack of experience for such surveys and has taken steps to help develop state-level capacity to conduct surveys or to manage surveys conducted through contractors. So far these efforts have been geared mostly towards those states and local areas that have grants to study those who leave or are diverted from cash assistance. For this group of states, ASPE has held conferences that provide technical assistance for conducting surveys and has hired a contractor with survey research expertise to provide technical assistance for these states. ASPE staff have also compiled information relevant to developing better surveys, such as survey instruments that include welfare-relevant questions and references to key survey methodology literature. Funding for further enhancements to surveys of welfare leavers and divertees for three states and two county groups with previous grants to track

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

leaver and divertee outcomes has also been allocated. The ACF annual welfare evaluation conferences have also included sessions on state-level surveys.

These capacity-building efforts for states and localities to conduct or contract for their own surveys should be greatly increased. Future activities should be expanded to states other than those conducting leaver or diversion studies. One such activity would be regular conferences that primarily focus on topics in surveying low-income populations with state-level research staff and with experts in survey methodology. DHHS could also consider funding short courses on survey methodology topics for state-level staff. Further interaction with federal-level survey practitioners (e.g., at the Census Bureau and other statistical agencies) could also be beneficial. Some states have more experience in conducting surveys than others and can share valuable hands-on experience and lessons learned. Mechanisms for fostering communication between these groups should be developed and is another area in which DHHS can take a leadership role.

Recommendation 5.5 State-level capacity to conduct household surveys of low-income and welfare populations is limited. DHHS has begun an important effort to build state capacity for conducting surveys. These efforts need to be continued and expanded.

ADMINISTRATIVE DATA

Administrative records that contain information collected as part of administration of programs or services are crucial sources of data for current and future welfare program monitoring and evaluation. Most data relevant for social welfare programs are collected at the local level but linked and maintained at the state level. Some programs further require states to provide administrative data to the federal government. States are required to provide microdata (data on individuals) to DHHS on persons receiving assistance from the TANF program and from separate state programs funded with TANF block grant funds under maintenance of effort requirements. The Child Support Enforcement Program also requires states to provide data to the federal government. This system, the Federal Parent Locator Service (FPLS), consolidates state data (earnings and employment status, employer information, employee social security numbers and names, and unemployment insurance benefit information) on mothers and fathers of children with child support awards and consolidates them in a national database that will be available for some research purposes. The Urban Institute program also maintains records of wage and employment information for individuals that have been used for social welfare program monitoring and evaluation. Administrative data on households, business firms and government entities are being linked to employer and household surveys from the Census Bureau in the Longitudinal Employer-Household Dynamics Pilot Project and the related

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

Dynamic Employer-Household Data and the Social Data Infrastructure Project. Child protective services, foster care, Women, Infants, and Children Program (WIC), Medicaid, and Food Stamp Program data are other sources of administrative data for social welfare program evaluation.

Administrative data in general have become increasingly important for monitoring and evaluation purposes in the devolved social welfare program environment. These data sets are available on the state and local level, which makes them particularly attractive for use in state- or local-level analysis (Hotz et al., 1998). Aggregate data on program caseloads have been used to estimate the effects of welfare waivers on outcomes. Such data could also be used on an aggregate level to assess the overall effect of PRWORA on cash assistance caseloads and benefits (as described in Chapter 4), with time-series modeling. Using administrative data for cross-state nonexperimental evaluations of changes in broad policy components and changes in specific detailed strategies is also possible since these data sets are typically larger state samples than national-level surveys. Cross-state comparability is a potential limitation for using administrative data for these purposes. Their use is further limited because they are not representative of all persons potentially eligible for programs. However, if information from population-based surveys can be linked with administrative records, these data could be used for evaluations of program participation. Administrative data also have an advantage over survey data in that administrative records contain more detailed and reliable data on program participation.

There are many examples of state- and local-level monitoring and evaluation efforts using administrative data. Administrative data are usually integral parts of experimental program evaluations. Often the outcome measures of program participants—both control and treatment group members-are tracked with administrative data. A number of states are linking administrative data from social welfare programs and from UI wage records to track the status of families that leave welfare. The administrative data used in welfare leaver studies are also used as sample frames for surveys conducted as part of these studies. Administrative data are also being used for other types of welfare program evaluations. For example, the Urban Change Project used administrative records from food stamps, Medicaid, and AFDC/TANF to identify their populations of interest for the four-county sites of the study and will use the data to track some outcomes. A Department of Labor study of six cities (Atlanta, Baltimore, Chicago, Ft. Lauderdale, Houston, and Kansas City—managed by the University of Baltimore) is using administrative records on program participation, employment, and earnings to understand the dynamics of welfare-to-work patterns of low-income individuals. Administrative data are also being used to assess the quality of and improve survey data collections related to welfare program surveys. For example, the California subsample of the SIPP will be merged with data from California UI wage records and AFDC/TANF administrative data to assess the accuracy of self-reported program participation data (Hotz et al., 2000). The Iowa SPD

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

Feasibility Study examined the use of samples of welfare program participants from administrative records to reach households targeted for surveys (Nusser et al., 2000).

General Issues

As is the case with survey data, population coverage, sample size, data quality, content, and periodicity are important aspects of the value of administrative data for the monitoring and evaluation of social welfare programs. When the question of interest requires data on both participants and nonparticipants, administrative data alone will not be of great use. For such needs, however, administrative data can be used in conjunction with survey data to provide needed information. If the population of interest is programmatically defined, such as those who once received welfare, then administrative data are a relatively inexpensive source of data for monitoring or evaluation questions concerning these populations. There may still be issues of coverage for administrative data sets.

A major weakness of administrative data is their limited content, for they typically include only the information on an individual or family necessary to establish eligibility and benefits. Thus more general household demographic characteristics or indications of health problems, transportation difficulties, or child care obstacles, to name only a few, are missing. Administrative data do not include information on household members who are not part of the benefit unit or whose characteristics are not considered for eligibility determination. This lack of data on other household members limits the use of administrative data for studying family-level outcomes. The problem can be reduced if other sources of administrative data containing information on other household members, such as UI wage records or tax records, can be linked. However, both UI and tax records have coverage problems (see Hotz and Scholz, 2001). It is also difficult to measure child-bearing, family composition changes, and family structure with administrative data.

Administrative data sets generally have much larger sample sizes than surveys, which is a major advantage for monitoring and evaluation. Administrative data sets are usually quite large since records for each person that participates in a program are kept. Similarly, administrative data sets do not have the problems that national surveys have in accurately collecting benefit information because good records on what benefits and services were received are crucial to operating the program. The diversity of programs and services does have implications for administrative data systems, however. In some states, TANF-funded services are being targeted to children and their noncustodial fathers in addition to their mothers, who have traditionally been the primary component of the TANF case unit through which services have been funneled. Thus, for some purposes, it may be important to track benefits received by families rather than by individuals. Doing so may require a greater degree of linking of data from different family members.

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

If historical information about program participants is needed for an evaluation, administrative data can be very valuable. Historical information about benefit receipt is particularly useful in some evaluation settings, such as the nonexperimental methods of time-series modeling and comparison group modeling, that require good information on the past program or employment status of individuals in the study. Longitudinal survey data can capture such information, but if questions are asked retrospectively, then measurement error can be a serious problem for surveys trying to collect this information. Administrative data would presumably provide very reliable data on benefit receipt, and if they can be linked with population-based survey data, they may be useful for measuring the effects of changes in program components on program entry or on measuring macroeconomic and other feedback effects. A challenge to doing so is that in the past, administrative data have not been maintained or retained for long periods of time, so that historical information is sometimes not available. How long these records are kept varies from data set to data set and from state to state (UC Data, 1999). Because states must now track time limits for cash assistance recipients, records must be kept for longer periods.

Data that are not crucial to administering benefits (e.g., the educational level of a woman applying for TANF) may not go through as rigorous quality checks as data crucial to administering the program (e.g., benefit levels) and so are more likely to be of suspect quality (Hotz et al., 1998). Other items may be collected only when a case is opened and so may not be accurate after the initial period (Goerge and Lee, 2001). For some items collected as part of eligibility determination, applicants may provide inaccurate or incomplete information, such as total earnings. Such issues for collecting administrative data can be key for understanding the quality of administrative data (Goerge and Lee, 2001; Hotz and Scholz, 2001).

There is generally a short lag in the availability of administrative data so they are potentially available on a very timely basis. However, they are collected for administrative purposes and thus are not always readily available for research purposes. To be used for research purposes, the data typically need to be cleaned—that is, the quality of the data needs to be assessed and where possible, improved.9 Preparation for research purposes also typically includes linkages to other administrative data sets or to survey data sets to improve coverage and content. Linking data often requires obtaining data from other agencies, which may have different definitions and data formats so that formats, definitions, and units of reporting between two data sets need to be reconciled before the data can be linked. Thus, comparability of definitions and data across programs is an

9  

This work includes making sure that items recorded the same way over time, that similar definitions are used throughout, that reported items are within a valid range, and, when possible, that comparisons of similar information are reported in other data sets.

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

issue. For national-level monitoring purposes, administrative data need to be comparable across areas.

Linked administrative data sets and linked survey and administrative data sets have great potential as sources for welfare program evaluations (for details, see Goerge and Lee, 2001; Hotz et al., 1998; National Research Council, 1992). Such linkages can improve coverage of populations. For example, linked data from many programs serving low-income populations (TANF, Medicaid, food stamps, and many others) can improve coverage of program participation. Linking administrative data sets can also improve coverage of needed content areas. For example, unemployment insurance data can provide some employment and earnings data. Child outcome measures, such as test scores from schools or child abuse and neglect information, can improve content coverage if they can be linked to other data sources (see Barth et al., 2001).

Administrative data may also be used to assess the quality of data from different sources. For example, the degree of nonresponse bias in surveys can be assessed by matching administrative records that are available for both respondents and nonrespondents to survey questionnaires to determine how different nonrespondents are from respondents on important variables of interest.10 Administrative data from food banks or homeless shelters may be a source of information on those who cannot be contacted or who do not respond to surveys.

There are, however, challenges to linking data sets. One challenge is in gaining access to data sources. Within a state, this can be a problem, as data sets often come from separate administering agencies, each with their own protocols for sharing data and for protecting confidentiality of the data. Data sharing across states is even more difficult, as agreements across multiple agencies in multiple states must be obtained. The federal government may be able to collect administrative data across states if it has legislative authority to do so, as in the case of the FPLS and the TANF reporting requirements. Without such authority, data sharing among states is voluntary.11

Another challenge to data linking is the availability of high-quality information on which data sets can be matched to one another. If data items that are used to actually link data sets (names, common identifiers such as SSNs, addresses) are of suspect quality or are incomplete, then matching across data sets becomes problematic. Reconciling differences in definitions, units, and protocols for collecting and storing data across different programs is another problem for data linking. This problem is even more severe when linking data across states be-

10  

Some welfare leaver studies are planning to compare earnings of survey respondents and nonrespondents. The Census Bureau is planning to match Social Security Administration records to the SPD to assess the effects of attrition in the SPD.

11  

Linkage of the FPLS data with TANF data would be a valuable research product in particular.

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

cause programs (especially TANF programs) are different across states and, therefore, data may not be easily converted to common standards.

The availability of administrative data at the state level and its relatively accurate measures of program participation measures make them a vital component of the data needed for evaluating welfare reforms. Its familiarity and accessibility to state-level program officials is also important as states now have more interest in monitoring and evaluating their own programs. The development of administrative data sources for research and evaluation purposes should, therefore, be a priority.

Because most administrative data for welfare program evaluations are collected at the state level, development efforts should be focused there. Many state- and local-level administrative data sets are only now beginning to be used for research purposes. Such efforts are uneven across states, although some states have been linking and using administrative data for such purposes for some time now (Cyphers and Kinsella, 2000). In general, states have varying degrees of expertise and resources for preparing and using administrative data sets for research. Technical assistance and funding to develop and link administrative data systems should be provided to help all states develop, clean, and improve the quality of administrative data. Some states also need guidance in negotiating data-sharing agreements across agencies in their states. For many of these capacity-building items, states that already have expertise in developing administrative data can help those with less experience. Federal statistical agencies also have expertise in matching administrative data that can be exploited.

Recommendation 5.6 Administrative data, primarily at the state level, are an important emerging source of information for both monitoring and evaluation. However, there are many significant challenges that prevent them from fulfilling their potential, including the conversion to research use from management use, preservation of data over time, improvements in the quality of individual data items, comparability of data across states, confidentiality and access, and barriers to matching across different administrative and survey data sets. Much more investment in this data resource is needed.

DHHS has sponsored some projects to develop administrative data for research purposes, such as assistance in the development of data for welfare leaver studies, a recent effort to develop public-use files for these studies, and the development of federal level administrative data sets (FPLS, the National New Hires Directory, and the TANF reporting requirement data) for research purposes. These efforts are important beginnings for the development of administrative data. The resource and expertise requirements for a more long-term and comprehensive effort to develop administrative data for social welfare program evaluation are great, however. Such an effort would require a great deal of

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

coordination with states, both in developing the data and in making them more comparable across states. Neither of the relevant agencies within DHHS (ACF and ASPE) can devote the kind of resources needed and still fulfill their own missions. The need for a more comprehensive, long-term effort to develop administrative data sources is another factor that leads the panel to propose the establishment of an authority within DHHS that is responsible for social welfare program data collection (discussed in Chapter 6).

Cross-State Comparability

Any multistate monitoring or evaluation study must confront the issue of comparability of administrative reporting. This is a special issue for TANF program administrative data since the programs and administrative structures for operating the programs now vary widely across states. Some state TANF administrative data sets contain certain types of recipients while other states contain other types of recipients. For example, Wisconsin has moved administration of child-only TANF cases to its child welfare system so these cases would not be part of the TANF data sets, unlike other states. Definitions of a case and family unit also vary across states. For example, some states classify cases in which the adult has been sanctioned as child-only cases, while other states consider children receiving TANF benefits that do not live with a parent as child-only cases. States are providing a wide variety of services to beneficiaries and, as the population being served changes, these services are evolving. As a result, there are few standard definitions of services or even types of services. The services provided under job search assistance in one state may not be classified the same way in another state. All these differences need to be well understood and reconciled if data from multiple states are to be used for research purposes.

Conclusion 5.3 The lack of cross-state comparability is a major barrier to the use of state-level administrative data sets for cross-state monitoring and evaluation.

The panel concludes that more can and should be done to improve the cross-state comparability of administrative data sets if these data are to reach their full potential. These improvements should move toward a common set of definitions of services and service units, which will not be an easy task. However, as the programs become more stable, it should be easier to identify broad types of services that can be defined and to implement common types of service units. Child-only cases are a good example of those for which a standard definition and service unit could be created.

Improvements in common data formats must also be made, including updated systems for storing and managing administrative data. Systems vary significantly across states and agencies. For example, some records are still kept on paper and so are far from being ready for research use, while others are readily

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

available on an electronic basis for research requests. Some data are organized around families or case units while others are organized on the basis of individuals. Not all administrative data systems can easily switch between organization units and so are difficult to match and use with other data. Documentation of formats and definitions is also not standardized in any way.

Improvements in comparability will require cooperation between state and federal agencies.12 The administrative databases must meet the requirements for use in each state individually. Therefore, complete comparability across all states is probably not feasible. However, the federal government should take the lead for stimulating and supporting improvements in comparability. DHHS can fund and support states to develop their administrative data sets for welfare program research and help develop standard definitions for services and service units and standard protocols for documenting data. DHHS has some experience in this area and can build upon it for future activities. For example, in ASPE-sponsored welfare leaver studies ASPE staff worked with state grantees to develop common definitions for the studies as much as possible across states. ASPE is also working with states conducting welfare leaver studies to produce restricted-access data files from the administrative and survey data collected that are as comparable across states as possible.

Recommendation 5.7 The panel recommends that DHHS, in conjunction with state social service agencies, take steps to further improve the comparability of administrative data across states. These steps should move toward comparable definitions of services and service units and data formats. Building comparability across states will have to be a cooperative effort between the federal government and states and will likely require federal funding of state activities.

Research Uses of TANF-Required Data

States are required to collect and report microdata to the federal government on TANF cases and on cases receiving benefits provided under other programs under maintenance-of-effort grants. States are also required to provide similar data on cases that stopped receiving benefits in the given time period. These data were required as part of state accountability measures in PRWORA and are used to assess whether these requirements are met and to award high performance bonuses. The final rule about what data need to be reported was established in 1999 and is effective for fiscal year 2000. Broadly, states must report informa-

12  

Again, this is an area where the sustained and coordinated efforts of an organizational entity responsible for social welfare program data collection that involves cooperative efforts with state data centers, as proposed in Chapter 6, could most effectively make needed improvements to the data infrastructure.

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

tion about how much and what assistance each case received, information collected for determining eligibility (earnings, work participation activities) and basic demographic data about adults and children in the case (age, race, ethnicity, gender). Monthly data are reported on a quarterly basis to the federal government.

Although these data are collected for federal administrative functions, they are a potential source of data for welfare program monitoring and evaluation. They are potentially useful for monitoring caseload characteristics, benefit levels, and participation in work activities, for example, on a national level. Social security numbers of cases are also reported so there is a potential to link these data with other administrative and survey data sets. For example, data on cases that stop receiving benefits could potentially be matched with other data sources (such as UI employment and earnings records) to provide a national picture of earnings of welfare leavers to compare with results now being reported from state-level studies. These data might also be used in conjunction with national-level surveys to supplement information available from them and to assess data quality (like the SIPP and California administrative data match noted above).

Time-series and cross-state, cross-time caseload modeling of the overall effect of PRWORA could be conducted with aggregated data on welfare caseloads from administrative records (see Chapter 4). To do so, the aggregate data on TANF participation that is being reported to ACF will be crucial. A critical factor for the use of these data for time-series modeling is whether the data are comparable over time. The AFDC program had its own data reporting definitions and protocols. The reporting requirements established in 1999 use new definitions and protocols. In between the passage of PRWORA and when the final reporting requirements went into effect, states reported data under emergency reporting requirements. It is not clear how comparable data reported under these three systems are. If these data are to be used for such purposes, the comparability between these needs to be explored.

Another limitation of these data is that their coverage of program benefits is not comprehensive. DHHS specified a definition of assistance for the reporting requirements—that is, states must report data on cases receiving benefits, where “benefits” are defined only as cash or vouchers for basic ongoing needs such as food, clothing, shelter and utilities (see Federal Register, April 12, 1999, for an exact definition). This definition does not include many types of benefits that are currently offered by states, such as nonrecurring short-term benefits, work subsidies, support services such as child care and transportation, employment-related services such as job retention and advancement services, counseling, or child care information and referral. The definition is thus quite limited: data for families receiving noncash types of benefits will not be reported. For example, Florida is using some of its TANF money for abstinence education programs. Other states are using TANF money for programs that specifically attempt to engage poor fathers in the lives of their newborn children. Participants in these programs will

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

presumably not be part of data collected through these federal reporting requirements. Thus, the data do not cover all types of benefits received.

Information on those receiving noncash benefits is also important as states are increasingly using TANF funds for noncash benefits. To do so, a broader definition for whom microdata needs to be reported is needed. It may still be desirable to retain a delineation of the different types of benefits received (e.g., cash assistance separate from noncash assistance, such as child care or work support services) in the data reporting, but a broader definition will give a more comprehensive picture of the services being received.

The definition of assistance for these reporting requirements was a point of contention between the federal government and the state governments. The original federal proposal for a definition was broader than the current version. States, however, wanted a narrower definition, partly to reduce the burdens of reporting, as the broader definition would require states to report more data. In many cases, states do not have a great deal of funding to develop these data and to get their information systems geared up for the reporting. The narrower definition, however, seriously limits the use of these data for program monitoring and evaluation.

Recommendation 5.8 The current definition of assistance used to guide state data reporting requirements is very narrow and will not capture many recipients of different forms of assistance provided by states. The panel recommends that the Administration for Children and Families consider broadening this definition to include as many types of assistance and services provided as possible.

Another limitation of the administrative data reported under TANF is that some states are reporting samples of data, while others are reporting the entire universe of data (i.e., data for all persons receiving benefits in the month). DHHS has given states the option to report either a sample or the universe. For research purposes, however, there are key advantages to having the universe of data available. If the universe of data is reported, data can be linked longitudinally so that past program participation and benefit receipt for each individual can be tracked. The data could also be used to collect information on past welfare receipt histories of cases that move across states. Linkages with other data sets, including survey data, will be more feasible if the universe of data is collected. These linkages could be used to provide additional information about individuals in surveys—particularly program participation and benefit receipt. The full universe of administrative records could also be used to assess the accuracy of survey data reports of program participation.

In some respects, it will be less burdensome for states to provide data on the universe of their cases than on a sample, since a sample of their cases will not have to be drawn. The federal government would have an extra burden of storing the full universe of data, although electronic storage is becoming much less

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

expensive. However, the assessment of data quality and preparation of these data for research use will be significant endeavors. The panel believes, however, that efforts to obtain the universe of data and to make them available for research purposes should be pursued to the greatest extent possible.

Recommendation 5.9 Administrative data reported by states as part of the TANF reporting requirements will be of limited use for research purposes unless steps are taken to improve them. The usefulness of these data will be improved if the data can be linked to other data sets and if the full universe of cases is reported. The panel recommends that ACF take steps to improve the linkability of these data and encourage states to report the full universe of cases.

DATA ACCESS AND CONFIDENTIALITY

Answering important questions about the effectiveness of welfare programs requires a greater reliance on multiple sources of data and linkages between sources—including linkages among administrative data sets, between administrative and survey data sets, and across levels of government. Although the needs for multiple sources of linked data are great, restricted access to data, instituted to prevent individuals who provide data from being harmed by the improper use of the data, limit data currently available for use in research contexts. There are restrictions to access for both survey and administrative data. Such restrictions range from completely limiting the use of the data to only those within the collecting agency to no restrictions at all on the use of individual data. For many data sets, restrictions on data access are somewhere in between. Different models for data access include: granting access to group-level data instead of individual-level data; releasing scrambled data so that any data items containing individually identifying information are either not released or are modified to prevent individual identification; granting permission for researchers to access data at a centralized data holding center (e.g., as done in the Census Bureau data centers); housing data in secure holding centers but allowing researchers to specify the analyses they want conducted with the data as long as they are within the bounds of confidentiality requirements; and releasing individual data to those who agree to abide by terms for the use of the data and are subject to penalties if they are found to be using the data in ways other than the agreed-upon terms.

The panel believes that confidentiality and access restrictions are often drawn in ways that unnecessarily limit the use of important sources of data for welfare program monitoring and evaluation. Confidentiality protocols limit the use of survey data for evaluation purposes. For example, nonexperimental methods of evaluation that compare outcomes of individuals across different areas must be able to control for the economic conditions and other characteristics of the local areas in which study participants live. This requires information about where the

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

individuals live (for example, a county, census tract, or zip code). Often, this information is not available from surveys, and therefore, poses a barrier to use of these data for evaluation.

The state-based nature of the TANF program and limited budgets for evaluation magnify the importance of administrative data sets and linkages of administrative data sets for program evaluation. Although these data are needed now more than ever to inform policy decisions, they are often not accessible. A barrier to data access that is particularly acute with the devolved program structure for TANF is that rules and protocols governing data access vary considerably across states and even across agencies within states (Brady et al., 2001). A study examining state social service administration practices for releasing data found that some state social service agencies had well-developed policies for release, while other states did not (UC Data, 1999). Cross-agency data sharing at the federal level is also hampered by the variation in data sharing protocols among federal agencies (National Research Council, 1993 a, 2000a).

Variation in policies and practices regarding release of data and confidentiality protections across both state and federal agencies is a particularly complex problem for linking administrative data from different agencies for research purposes. Gaining access to data from multiple agencies is more difficult because data-sharing agreements with each agency providing data must often be negotiated separately, a process that may take considerable time. Linking data from different sources requires some individually identifying information for each data point to be matched with similar information from another data set. Social security numbers are a typical example of such information. However, some agencies have moved away from using SSNs as identifiers and assign their own identifying numbers to cases. A problem then for linking is that identifiers in each of the data sets being linked may differ across each agency.

Different agencies also have varying degrees of experience, resources, and technical know-how for releasing data for research purposes, and, thus, policies and practices for releasing data have developed unevenly. This is especially true across state agencies, partly because data collection and evaluation efforts were not previously focused at the state level and resources for data collection and linkage were more limited. With an increased focus on state-level data and evaluations, state agencies could use advice on different models for data sharing and new technical advances that can be used to protect confidentiality of individual data. This is another area for which DHHS can convene conferences or meetings with state and federal agency representatives and with experts in data access and confidentiality. In the longer term, leadership in coordinating data linkages from the proposed data collection authority in DHHS that is responsible for data collection for social welfare programs (see Chapter 6) could speed the coordination of confidentiality protocols and take steps to lower data access barriers.

It is crucial for data collection agencies to ensure that the individuals who

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

provide data are not harmed from misuse of the data. The reputation of the agencies charged with collecting data or administering programs may rest on whether individuals believe that these data will be used appropriately or that they will not be harmed by providing the data. The issues of data access and confidentiality are, however, much deeper and more complex than any single agency or state. The larger scientific, legal, and governmental communities will need to be involved in resolving current tensions between opening data access while protecting individual privacy.

One factor in the tension between open data access and protecting privacy is that current laws governing the use of data often do not specifically address the use of the data for research purposes.13 Most privacy laws allow administrative data to be used for activities in accordance with the program’s purposes or include clauses for “routine use” of the data. It is often through these provisions of the laws that agencies grant access to the data to policy analysts outside the agency (Brady et al., 2001). But because the laws do not directly address the use of data for research purposes, these laws can be interpreted quite differently. Policy makers and agency officials with control of the data are not always aware that researchers have no interest in the personal identities of specific individuals providing data, but rather, only have a need for information on random individuals. Because they do not understand how researchers use the data, agencies that control data too often limit access to the data for research purposes, even though the data could be valuable for monitoring and evaluation.

Rules governing the use of individual data collected from state and federal government agencies need to clarify how the data can be used for research and evaluation purposes. Provided such clarifications can be made and agreed on, an effort to implement them as consistently as possible across and within state and federal agencies is needed. The benefits of access to data for program monitoring and evaluation purposes need to be better communicated to agency officials that grant data access. Alternatives to simply eliminating access to data includes stricter enforcement of rules governing the use of restricted data to discourage improper disclosure (National Research Council, 2000a), as well as advancing the development of techniques for data disclosures that protect confidentiality, such as data masking or data perturbations (Brady et al., 2001). New technologies may help to provide external access to linked data sets while meeting confidentiality requirements. For example, the Longitudinal Employer-Household Dynamics Pilot Project at the Census Bureau and the related Dynamic Employer-Household Data and the Social Data Infrastructure projects are exploring the use of web-based and video teleconferencing tools to provide research data.

The panel has found that data access and confidentiality restrictions are

13  

See Brady et al. (2001) for a review of legislation specific to administrative data uses and National Research Council (1993a) for a review of legislation regarding data collected through federal agencies and for a review of agency-specific confidentiality practices.

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

significant barriers to the availability of data for social welfare policy evaluation. As these issues begin to be addressed on many fronts, the panel emphasizes the need for data linkages in welfare program evaluations and encourages steps to remove barriers to data access.

Recommendation 5.10 Confidentiality, privacy, and access concerns with administrative and survey data and the linking of multiple data sets are important issues, but are currently serving as a barrier to socially important evaluation of welfare reform programs. The importance of access to these data for monitoring and evaluation of programs should be emphasized and efforts to reduce these data access barriers while protecting privacy and maintaining confidentiality should be expanded.

PROGRAM DESCRIPTION DATA

Another type of data that are crucial for social welfare program evaluation are program description data, or descriptions of the TANF and related programs that states have adopted and are operating. The importance of such data for evaluation seems obvious—programs cannot be compared if the rules of the programs are not known. Yet as the panel stated in its interim report, good program description data on post-PRWORA state and local programs were slow to develop. This lack of data has been a major limitation for welfare reform evaluation, especially for cross-state monitoring and evaluation. In relation to monitoring the low-income population on a national level, it is important to know what the different state programs are so that national surveys can ask respondents the right questions about the program benefits they receive. Nonexperimental methods of evaluating the overall effect of welfare reform and of broad components of reform rely on fairly detailed program description data for both TANF policies and other related non-TANF policies in states and localities so that these variables can be included in models to control for the different policies that apply to each case. Program description data need to cover programs in effect during the entire study period. For national-level evaluations, program description data are needed from every state and for every locality within states for which different rules apply. For state-level evaluations, program description data are needed for a given state only, unless different localities within the state have different policies.

The federal government’s efforts to collect program description data are, thus far, limited. States are required to provide summaries of their state TANF programs every 2 years in order to receive block grants. The first collection was in 1997 and the second in 1999. States are given general guidance on what to report, but there are no specific requirements. As a result, the level of detail that the states have provided about the programs varies greatly. These state summa-

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

ries are just now being organized so that they can be made accessible to the public.

Aside from the state plans required to be reported to DHHS every 2 years, there are several efforts under way to document state policies and practices (see Chapter 2). The most comprehensive effort has been conducted by the Urban Institute as part of their Assessing the New Federalism Initiative. This database is the most comprehensive in time documenting rules from 1996–2000. It includes data on eligibility rules, asset rules, benefit levels, work activity requirements, time limits, sanction policies, behavioral requirements, and child support requirements. The data have been coded to be more readily usable in quantitative analyses and are publicly available (see http://www.wrd.urban.org). The 2000 round was sponsored by ACF and current plans call for updates in 2001 and 2002.

In another effort, the Center for Law and Social Policy and the Center for Budget and Policy Priorities are jointly collecting information on state TANF and Medicaid policies. Currently, this database includes information on state policies regarding: TANF applications, cash diversion programs, emergency assistance, categorical and financial eligibility rules, family cap policies, minor living arrangement provisions, school and training requirements for minors, abstinence education programs, and Medicaid (see http://www.spdp.org). This information was collected for legislation enacted before and updated through 1998. In the future, information about time limits, work activities and requirements, sanction provisions, child care assistance, child support, and drug-related provisions will be included in the database. While this database contains some information not included in the Urban Institute database, it does not cover the full time frame since PRWORA that the Urban Institute database does.

The Congressional Research Service (CRS) has also used state TANF plans reported to DHHS-ACF to produce reports summarizing state program rules on different topics. For example, the last summarized key features of state TANF programs, such as: treatment of earnings and savings, welfare diversion, work requirements, personal responsibility plans, division of the welfare caseload, and benefit levels (Burke et al., 1999). Every 6 months states are surveyed to collect information about financial eligibility and benefit determination rules in effect. The last update includes rules in effect through January 2000 (Abbey et al., 2000), and future updates of this report are also planned. Finally, the CRS has a continuing series of reports called Cash and Noncash Benefits for Persons with Limited Income that covers over 80 federal programs that serve low-income people that is produced every other year. The latest version was produced in 1999 (Burke et al., 1999), and another version is due in late 2001.

It may seem that all these ongoing efforts to collect TANF program description data would translate into sufficient, even redundant, program description data, but this is not the case. Most of these data collection efforts are not conducted on a regular basis and do not cover the entire time frame of interest. Only the Urban Institute database comprehensively covers programs and changes in

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

programs in operation between 1996 and the present. The state plans provided to ACF and summarized by the joint effort between ACF, Welfare Information Network, American Public Human Services Association, and National Governors Association (discussed in Chapter 2) will eventually be available every other year. The State Policy Document Project only covers programs in effect at one point in time, thus far.

Another weakness of many of these efforts is their coverage of all programs and rules. It is not at all clear that data submitted to the federal government will be detailed enough in every state and consistently reported across states to comprehensively cover all TANF and other state welfare programs, since there are few detailed requirements of what states have to report. Again, the Urban Institute’s database is probably the best source of detailed data, but it does not include information about other related non-TANF program rules. Other data collection efforts provide some needed information, but these efforts do not comprehensively cover all programs. Thus, evaluations that depend on full descriptions of TANF and non-TANF program rules (such as time-series modeling, comparison group modeling, and cross-state policy and timing variation models) will have a difficult time controlling for all the policies that may affect individuals in the study, and thereby, isolating the effects of specific policies. A further limitation of these data is caused by the changes in the ways agencies have actually implemented their policies. None of these sources of program description data collect detailed information on program implementation (see below).

A key issue for the collection of program description data is whether efforts like the Urban Institute’s can be sustained. There are positive signs in this direction as ACF funded the 2000 round of the rules update, and plans for future rounds are now set. However these data need to be continually updated and expanded to cover other related programs. Data collection of program rules, in a sufficiently comprehensive, detailed, and consistent manner across states, should be an institutionalized component of DHHS’s duties for administering social welfare programs. This work requires the institutional commitment of a government agency to ensure that the data are collected and provided to users in a readily usable form.

Recommendation 5.11 The monitoring and documentation of the actual policies, programs, and implementations of welfare reform at the state and local levels by the federal government has been minimally adequate to date. The panel recommends that the Department of Health and Human Services take active and direct responsibility for documenting and publishing welfare program rules and policies in every state and in every substate area where needed. Continuing updates documenting changes in state and local area rules should also be produced.

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

In Chapter 4 the panel recommends that more process studies should be funded in order to provide more detailed descriptions of how state and local policies have actually been implemented. These detailed descriptions are needed in evaluation efforts, especially in evaluating the effects of specific detailed welfare reform strategies, to fully characterize the policies that cases are subject to and in turn, to use these characterizations to assess the effect of the policies on outcomes. We emphasize the need for more process studies that can provide this detailed data on program policies.

QUALITATIVE DATA

Qualitative data on individuals are another important source of data for welfare program monitoring and evaluation. While Chapter 4 highlights potential uses of qualitative data for evaluation purposes, the discussion in this chapter focuses on data collection. The collection of qualitative data for addressing key monitoring and evaluation questions posed in Chapter 3 is likely to be most useful when conducted as part of other data collection efforts, such as surveys or administrative data collections (see Newman, 2001). Sometimes these data collections are incorporated into larger surveys, more often they are not. Openended questions, or questions that do not require a respondent to choose among a fixed set of answers (yes or no, numbered choices, etc.) allow respondents to provide detailed responses or explanations in their own words. These are typically conducted within the framework of a larger survey with other closed-ended questions. In-depth interviews are question-and-answer interviews that include more open-ended questions than a survey instrument and typically do not include fixed choice questions. Such interviews may be conducted on a subsample of a larger survey sample, or on an independent sample of individuals. Qualitative longitudinal studies with in-depth interviews conducted on the same individuals or families over multiple interview sessions are also common. Focus groups are conducted with small groups of individuals to discuss topics of interest, with a facilitator prompting participants and guiding discussion. Finally, participant observation fieldwork directly observes behavior of study participants. Data are collected through notes the researcher takes while observing the behavior of study participants, usually on a day-to-day basis. Participant observation is typically combined with interviews of study participants.

Process analyses may also collect qualitative data to provide detailed information on programs or on a system that administers benefits and are another use of qualitative data. Data for process studies is gathered by visiting program offices (often across multiple service delivery areas); conducting surveys, indepth interviews or focus groups with key stakeholders in the delivery system such as program participants, caseworkers, and administrators; directly observing client and caseworker interactions; reviewing documentation of individual cases;

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

and special tracking of a cohort of participants through the processes of the program.

The goals of qualitative data collection are generally to provide more detailed information and sometimes more subjective information on individuals or groups that is not easily quantified or measured with survey or administrative data. Besides providing more detailed insights into the outcomes of individuals, qualitative data can also be used to improve quantitative data collection. Openended questions embedded in surveys can be used to develop future survey questions (Newman, 2001). For example, an open-ended question may solicit answers that are fairly consistent across respondents. In future surveys, those responses might be incorporated as fixed-choice questions. Qualitative studies, perhaps through in-depth interviews or longitudinal in-depth interviews, might also be useful for understanding family organization, which can be used to improve the design of surveys that are designed to gather information about family members (Newman, 2001). Participant observers and in-depth interviews may be used to understand how surveys can be successfully targeted to hard-to-interview populations for surveys.

Despite their potential use for welfare program monitoring and evaluation, qualitative and ethnographic data collections have several limitations. Qualitative studies are not always designed to statistically represent a population of interest. Because qualitative and ethnographic studies are expensive to conduct, the sample sizes of these studies are generally small, so that even those studies with representative samples of the population of interest are often too small for precise statistical analyses. Thus, qualitative studies are often most effective when nested within a larger survey or administrative records study to complement the information collected from quantitative data sources. A data collection issue here that qualitative and ethnographic researchers disagree on is whether it is more desirable to draw a separate sample of individuals within the same geographic area and with similar demographic characteristics as the survey or administrative records sample (as is the case with the Urban Change Project and the Three City Study) or to draw a subset of the survey or administrative records sample (as the ethnographic component of the Fragile Families Study has done). While the subsample strategy has the advantage of providing quantitative and qualitative data for a subsample of the study participants, the in-depth nature of many qualitative studies increases respondent burden for the subsample, which may make differential attrition an issue (Newman, 2001). Yet recruiting a separate sample of study participants for the quantitative component can be costly.

Replication of results is rare for qualitative and ethnographic studies, which is a serious shortcoming. To remedy this problem, qualitative and ethnographic researchers should work closely with one another to coordinate their efforts so that there is some consistency in sampling and data collection across multiple locales; this is rarely done. The exception is for those studies that use qualitative

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

or ethnographic studies across multiple areas. Moreover, qualitative and ethnographic studies often do not sufficiently detail the methods by which the sample was selected and the data were collected, stored, sorted, and analyzed. This deficiency contributes to the problem of trying to replicate studies. Thus, it is incumbent on qualitative researchers to better specify their data and methods when reporting their results.

While the potential uses of qualitative data in evaluation settings are becoming more apparent, these data are not routinely collected as part of evaluations of programs. The barrier is usually budget constraints, when quantitative outcome evaluations are of foremost importance for the sponsoring agencies. This priority is especially true for state program agencies sponsoring evaluations of their programs. Although many administrators see a useful role for ethnographic research (Newman, 2001), government-sponsored ethnographic studies are rare, especially at the state level.14 In the absence of a great deal of funding for extensive large-scale qualitative studies, smaller-scale studies embedded in larger quantitative studies may be the most feasible way to obtain needed qualitative data (Newman, 2001).

SUMMARY AND ASSESSMENT OF THE CURRENT DATA INFRASTRUCTURE

This chapter reviews the state of currently available data for welfare program evaluation and monitoring needs. Our discussion has focused on four types of data: survey data, administrative data, data describing programs and policies, and qualitative data. Addressing the key questions of interest for evaluating welfare reform requires the use of all of these kinds of data.

While in many respects the existing data are very rich and extensive and furnish a wide range of valuable information on the low-income population and welfare recipients, in many critical respects they have significant weaknesses. These weaknesses are sufficiently severe that the panel has concluded that inadequacies in the nation’s data infrastructure for social welfare program study constitutes the major barrier to good monitoring and evaluation of the effects of reform.

National-level surveys have an increasingly difficult task of measuring program participation in a setting in which programs vary widely from state to state. In addition, the sample sizes of many national-level surveys, though completely adequate for nationwide totals, are inadequate for the study of most of the state-level welfare program components (such as time limits, work requirements, etc.). The surveys often have major problems of nonresponse, which is often concen-

14  

One exception is a study under way in South Carolina conducting in-depth interviews with a sample of welfare leavers (Medley et al., 1999).

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

trated in the lower income groups in the sample. Finally, timeliness of data release and availability is a major barrier to the analysis of welfare reform, a problem particularly acute for SIPP, which has released very little post-PRWORA data at the time of this writing.

State-level surveys are in much worse conditions, primarily because states are only beginning to conduct such surveys. Most such surveys are conducted by telephone and response rates are often very low. Currently, they cannot be a major resource for the study of welfare reform.

State-level administrative data hold more promise in an era of devolution because they provide fairly large samples at the state level. But their use as research tools, rather than as management information systems, is still in its infancy. Administrative data sets are usually quite complex and usable only by those who are expert in understanding the coding of the data, which makes the data difficult to use by any wide set of evaluators or researchers. The data are sometimes not available over time, because states have had no reason to save them in the past. The lack of historical information on individuals in data sets puts limits on their use for evaluation and monitoring. Evaluation efforts are also hindered by the lack of cross-state comparability in data items and variable definitions, which makes cross-area methods of evaluation problematic with these data. An inherent limitation of administrative data is, of course, that they are only available for those who receive welfare.

Data describing state policies and programs have improved in quality and quantity in the last year. However, the development of databases with this critical information will need to continue and will need significant support from DHHS. Furthermore, the databases need to be significantly widened to cover more than TANF program rules, while at the same time maintaining historical dimensions with regularly recorded rules for all the states going back several years. Collecting information about the actual implementation of the official rules continues to be a very difficult problem.

Qualitative data in the form of ethnographic information on families is an underused source of information in program evaluation on social welfare reform. Neither administrative nor survey data can fully characterize the complexity of individual families’ lives and the way different types of families respond to welfare reform. However, researchers working with qualitative data need to continue to develop standardized protocols for collecting data and documenting how the data were collected, and they need to extend the data to cover a more representative set of areas and population groups. To date, these data have not played a major role in welfare reform evaluation despite their potential.

There is a critical need to address the data barriers hindering good evaluation and monitoring as these data barriers limit what is known about the effects of PRWORA and welfare reform. As we note in our previous two chapters, ASPE has an important role to play in addressing these barriers. ASPE has already

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×

committed significant money and staff resources to the improvement and maintenance of national-level survey data sets and has assisted states in developing their own databases, both administrative and survey. However, much more needs to be done.

The panel concludes that the overall weaknesses in the data infrastructure have not been fully realized by the wider community of evaluators and policy makers. In part, this is because there is no established mechanism for assessing data quality and reporting on it to the public. Consequently, the panel believes that ASPE should include, in the Annual Report to Congress, a review of the state of the data infrastructure for welfare reform research. The review should cover survey and administrative data at federal and state levels. It should identify strengths and weaknesses of existing data and should note gaps that need to be filled and give an assessment of what the highest priorities for filling the gaps are so that resources can be effectively allocated to those projects.

Recommendation 5.12 In its Annual Report to Congress, ASPE should review current availability and quality of data for welfare reform research, identify high-priority data needs, and discuss its own research agenda for data development and technical assistance.

Building a data infrastructure for welfare program monitoring and evaluation will take a concerted effort from federal and state governments, all of whom have interests in social welfare program evaluation and monitoring.

Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 102
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 103
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 104
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 105
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 106
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 107
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 108
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 109
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 110
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 111
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 112
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 113
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 114
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 115
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 116
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 117
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 118
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 119
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 120
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 121
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 122
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 123
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 124
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 125
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 126
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 127
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 128
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 129
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 130
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 131
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 132
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 133
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 134
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 135
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 136
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 137
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 138
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 139
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 140
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 141
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 142
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 143
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 144
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 145
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 146
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 147
Suggested Citation:"5. Data Needs and Issues." National Research Council. 2001. Evaluating Welfare Reform in an Era of Transition. Washington, DC: The National Academies Press. doi: 10.17226/10020.
×
Page 148
Next: 6. Administrative Issues for Maintaining the Data Infrastructure »
Evaluating Welfare Reform in an Era of Transition Get This Book
×
Buy Hardback | $61.00 Buy Ebook | $48.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Reform of welfare is one of the nation's most contentious issues, with debate often driven more by politics than by facts and careful analysis. Evaluating Welfare Reform in an Era of Transition identifies the key policy questions for measuring whether our changing social welfare programs are working, reviews the available studies and research, and recommends the most effective ways to answer those questions.

This book discusses the development of welfare policy, including the landmark 1996 federal law that devolved most of the responsibility for welfare policies and their implementation to the states. A thorough analysis of the available research leads to the identification of gaps in what is currently known about the effects of welfare reform.

Evaluating Welfare Reform in an Era of Transition specifies what-and why-we need to know about the response of individual states to the federal overhaul of welfare and the effects of the many changes in the nation's welfare laws, policies, and practices.

With a clear approach to a variety of issues, Evaluating Welfare Reform in an Era of Transition will be important to policy makers, welfare administrators, researchers, journalists, and advocates on all sides of the issue.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!