Evaluating Welfare Reform in an Era of Transition (2001) / Chapter Skim
Currently Skimming:
5. Data Needs and Issues
5. Data Needs and Issues
Pages 102-148
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 102... ...
Second are administrative records from social welfare and other programs, which are a somewhat newer and emerging data source for studying welfare reform. Much of the new administrative data is available at the state level, but there are also a few federal-level data sets.
|
From page 103... ...
We note that most of these recommendations for improvements are specific to the current data infrastructure for social welfare program monitoring and evaluation. The panel concludes that this infrastructure has limitations and that the devolved nature of social welfare programs has exacerbated the limitations.
|
From page 104... ...
National-Level Survey Data There are several national-level surveys that are relevant to welfare program monitoring and evaluation. They cover such content areas as income, earnings, employment, program participation and benefit receipt, adult and child wellbeing measures, family structure, and demographic and other background information.
|
From page 105... ...
Population Coverage National-level surveys are designed to produce analyses that are representative of the national population and, hence, are useful for producing estimates of the well-being of the nation as a whole.1 Most of these surveys are not, however, representative of smaller geographic areas, which is a significant disadvantage for studying welfare reform in an era of Revolution. Data from the census long form are representative of state and local areas, but they are only produced once every 10 years, and so are not appropriate for timely monitoring or evaluation purposes.
|
From page 106... ...
106 be .= ~ ca ~ ;^ ~ so ~ V, o .~ ~ ~ to z == to to ~ two o C)
|
From page 108... ...
108 A o ¢ ED V, ca .~ o o V, o ~ C)
|
From page 109... ...
109 ;^ ca ~ C - ' ~ ~ I ~ =~ ~ ~ o ~ ~ ~ o , ~ , e ~ , .
|
From page 110... ...
Subgroup analysis is necessary not only for description and monitoring but also for nonexperimental evaluations of the overall effect of welfare reform and of the broad components of reform. Of the currently available data sets, only the CPS has the sample size and statistical power for needed subgroup analyses, and even its usefulness is limited to estimating the overall effect of PRWORA and only for low variance outcomes, like the program participation rate (see the discussion of power in Chapter 4~.
|
From page 111... ...
The first SPD survey, the "bridge survey" in 1997, attempted 4For example, in the 2000 cPs March Supplement, the response rate for the basic monthly labor survey was just over 93 percent, but 8 percent of the basic sample did not respond to the supplement and so the total response rate was 86 percent (u.s. census Bureau and Bureau of Labor statistics, 2000)
|
From page 112... ...
Data release for the SPD is only slightly better; 1998 data were available in early 2001 and the first longitudinal file of the data set, covering 1992-1998, is scheduled to be released mid-2001. Thus, for two of the key surveys for monitoring welfare program participation, only very limited post-welfare reform data are available in early 2001, nearly 5 years after the reforms were enacted.
|
From page 113... ...
The major surveys that provide the most detail on program participation and benefit receipt are the SIPP, SPD, and NSAF. The devolved and ever-changing nature of welfare programs has made it more difficult for all of the national-level surveys to capture the welfare program benefits received by survey respondents.
|
From page 114... ...
Underreporting of food stamp participation in SIPP has also been found (Bollinger and David, 2001~. For the CPS, underreporting of welfare program participation has been documented for some time, although the extent of underreporting varies from year to year (Bavier, 1999~.
|
From page 115... ...
Further, understanding the interplay between the family composition and economic situations of sample members (e.g., a decrease in household income after one household member moves or enrollment in a welfare program after the birth of a child) is more feasible with SIPP because these changes are measured over short time frames.
|
From page 116... ...
Although the sample sizes are still adequate for estimating trends in well-being of the nation as a whole, they are inadequate for conducting extensive subgroup analysis and for state-level analysis. The SIPP has the advantage of more frequent periodicity of data collection and more extensive program participation coverage, but it has the significant disadvantage of being extremely slow in release, which greatly diminishes its usefulness for monitoring welfare reform.
|
From page 117... ...
The survey will produce yearly statelevel estimates of the number of individuals participating in broadly defined social welfare programs. It is not yet clear how well the questions included in the ACS will capture program participation, given the potential problems in measuring the wide array of services and benefits offered to poor families, which is a difficult problem for all the surveys.
|
From page 118... ...
They are less valuable for the evaluation of welfare reform and welfare programs as they evolve. Overall, the current set of national-level surveys is inadequate for fully addressing the research needs for monitoring and evaluating welfare reform, and improvements will be needed to make them effective.
|
From page 119... ...
Conclusion 5.1 The panel finds that each of the major national household survey data sets most suitable for monitoring and evaluation has significant limitations in terms of sample size, nonresponse levels, periodicity, response error, population coverage, or survey content. Although the national-level data sets have these limitations, there is a role for them to play in monitoring and evaluation of welfare programs.
|
From page 120... ...
The implications of this for evaluations of welfare reform need to be explored. Recommendation 5.1 To improve the abilities of national-level survey data sets to measure the effects of changes in broad welfare program components across states, the panel recommends expansions or supplements to the CPS or other surveys.
|
From page 121... ...
The lack of sufficiently detailed questions on welfare program benefit receipt may mean that the ACS, as it is currently planned, will not reach its full potential for welfare program monitoring and evaluation. If sample size in the CPS is increased, the need for more detailed questions on program participation in a larger survey like the ACS is reduced.
|
From page 122... ...
This expertise lies in DHHS and state welfare program agencies. This mismatch of expertise and responsibility has impeded data collection for program participation and is an area for which the proposed new organizational entity for collecting data relevant to social welfare programs outlined in Chapter 6 could be particularly beneficial in the long run.
|
From page 123... ...
For national household surveys that measure participation in and benefits received from programs serving the low-income population, it is critically important to regularly and frequently review survey questions to keep in step with program and population changes. The panel recommends to the Census Bureau that more resources be devoted towards improving questions on program participation and benefit receipt to better capture program participation.
|
From page 124... ...
thus far have predominantly generated survey samples from lists of program participants for example, persons receiving cash assistance or food stamps during a certain time period or persons in the control and treatment groups of a state-level experimental program. There are benefits to using such administrative lists for a sample frame (additional information from the administrative records on the universe of sample members is the key one)
|
From page 125... ...
. Most administrative data sets used in welfare program evaluation are state based.
|
From page 126... ...
However, with adequate resources and the use of experienced survey organizations, high-quality survey data in conjunction with experimental evaluations can be collected. Use in Current Welfare Reform Studies State and local surveys are only now beginning to be developed for welfare program and evaluation.
|
From page 127... ...
Sometimes very little information is gathered for cases that have been diverted, so there is very little information with which to track individuals in order to survey them. Some areas are using past participants in other social welfare programs who are not or have not participated in TANF to identify informally diverted cases.
|
From page 128... ...
The Urban Institute program also maintains records of wage and employment information for individuals that have been used for social welfare program monitoring and evaluation. Administrative data on households, business firms and government entities are being linked to employer and household surveys from the Census Bureau in the Longitudinal Employer-Household Dynamics Pilot Project and the related
|
From page 129... ...
is using administrative records on program participation, employment, and earnings to understand the dynamics of welfare-to-work patterns of low-income individuals. Administrative data are also being used to assess the quality of and improve survey data collections related to welfare program surveys.
|
From page 130... ...
Administrative data sets are usually quite large since records for each person that participates in a program are kept. Similarly, administrative data sets do not have the problems that national surveys have in accurately collecting benefit information because good records on what benefits and services were received are crucial to operating the program.
|
From page 131... ...
Longitudinal survey data can capture such information, but if questions are asked retrospectively, then measurement error can be a serious problem for surveys trying to collect this information. Administrative data would presumably provide very reliable data on benefit receipt, and if they can be linked with population-based survey data, they may be useful for measuring the effects of changes in program components on program entry or on measuring macroeconomic and other feedback effects.
|
From page 132... ...
Linked administrative data sets and linked survey and administrative data sets have great potential as sources for welfare program evaluations (for details, see Goerge and Lee, 2001; Hotz et al., 1998; National Research Council, 1992~. Such linkages can improve coverage of populations.
|
From page 133... ...
The availability of administrative data at the state level and its relatively accurate measures of program participation measures make them a vital component of the data needed for evaluating welfare reforms. Its familiarity and accessibility to state-level program officials is also important as states now have more interest in monitoring and evaluating their own programs.
|
From page 134... ...
can devote the kind of resources needed and still fulfill their own missions. The need for a more comprehensive, long-term effort to develop administrative data sources is another factor that leads the panel to propose the establishment of an authority within DHHS that is responsible for social welfare program data collection (discussed in Chapter 6~.
|
From page 135... ...
The final rule about what data need to be reported was established in 1999 and is effective for fiscal year 2000. Broadly, states must report informai2Again, this is an area where the sustained and coordinated efforts of an organizational entity responsible for social welfare program data collection that involves cooperative efforts with state data centers, as proposed in Chapter 6, could most effectively make needed improvements to the data infrastructure.
|
From page 136... ...
Although these data are collected for federal administrative functions, they are a potential source of data for welfare program monitoring and evaluation. They are potentially useful for monitoring caseload characteristics, benefit levels, and participation in work activities, for example, on a national level.
|
From page 137... ...
These linkages could be used to provide additional information about individuals in surveys particularly program participation and benefit receipt. The full universe of administrative records could also be used to assess the accuracy of survey data reports of program participation.
|
From page 138... ...
DATA ACCESS AND CONFIDENTIALITY Answering important questions about the effectiveness of welfare programs requires a greater reliance on multiple sources of data and linkages between sources including linkages among administrative data sets, between administrative and survey data sets, and across levels of government. Although the needs for multiple sources of linked data are great, restricted access to data, instituted to prevent individuals who provide data from being harmed by the improper use of the data, limit data currently available for use in research contexts.
|
From page 139... ...
This is another area for which DHHS can convene conferences or meetings with state and federal agency representatives and with experts in data access and confidentiality. In the longer term, leadership in coordinating data linkages from the proposed data collection authority in DHHS that is responsible for data collection for social welfare programs (see Chapter 6)
|
From page 140... ...
New technologies may help to provide external access to linked data sets while meeting confidentiality requirements. For example, the Longitudinal Employer-Household Dynamics Pilot Project at the Census Bureau and the related Dynamic EmployerHousehold Data and the Social Data Infrastructure projects are exploring the use of web-based and video teleconferencing tools to provide research data.
|
From page 141... ...
In relation to monitoring the low-income population on a national level, it is important to know what the different state programs are so that national surveys can ask respondents the right questions about the program benefits they receive. Nonexperimental methods of evaluating the overall effect of welfare reform and of broad components of reform rely on fairly detailed program description data for both TANF policies and other related non-TANF policies in states and localities so that these variables can be included in models to control for the different policies that apply to each case.
|
From page 142... ...
, and another version is due in late 2001. It may seem that all these ongoing efforts to collect TANF program description data would translate into sufficient, even redundant, program description data, but this is not the case.
|
From page 143... ...
Data collection of program rules, in a sufficiently comprehensive, detailed, and consistent manner across states, should be an institutionalized component of DHHS's duties for administering social welfare programs. This work requires the institutional commitment of a government agency to ensure that the data are collected and provided to users in a readily usable form.
|
From page 144... ...
We emphasize the need for more process studies that can provide this detailed data on program policies. QUALITATIVE DATA Qualitative data on individuals are another important source of data for welfare program monitoring and evaluation.
|
From page 145... ...
Despite their potential use for welfare program monitoring and evaluation, qualitative and ethnographic data collections have several limitations. Qualitative studies are not always designed to statistically represent a population of interest.
|
From page 146... ...
National-level surveys have an increasingly difficult task of measuring program participation in a setting in which programs vary widely from state to state. In addition, the sample sizes of many national-level surveys, though completely adequate for nationwide totals, are inadequate for the study of most of the statelevel welfare program components (such as time limits, work requirements, etch.
|
From page 147... ...
Qualitative data in the form of ethnographic information on families is an underused source of information in program evaluation on social welfare reform. Neither administrative nor survey data can fully characterize the complexity of individual families' lives and the way different types of families respond to welfare reform.
|
From page 148... ...
Building a data infrastructure for welfare program monitoring and evaluation will take a concerted effort from federal and state governments, all of whom have interests in social welfare program evaluation and monitoring.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.