5
Data Needs and Issues

Successful monitoring and evaluation of welfare reform are not possible without good data, regardless of how clearly the questions of interest are delineated and how strong the available evaluation methodologies are. Good data are therefore critical to studying welfare reform.

In Chapters 3 and 4 we showed that there are a wide variety of questions of interest and that answering those questions requires a number of different methods. It should not be surprising that addressing the questions of interest for monitoring and evaluating welfare reform will therefore require use of multiple types of data from multiple sources. We categorize data sources into four generic types for our discussion. First are household surveys, both national and state level, which have been the data sources that have informed much of what is known about the low-income population. Second are administrative records from social welfare and other programs, which are a somewhat newer and emerging data source for studying welfare reform. Much of the new administrative data is available at the state level, but there are also a few federal-level data sets. Third are data describing policies and programs at the state and local levels. Fourth are qualitative data, another source of data that are increasingly being used in policy evaluation. Together, these four types of data constitute the data infrastructure for monitoring and evaluating welfare reform.

Good data have many characteristics. They have reasonably good coverage of the population in question. They contain measures of the key variables of interest for welfare reform study, either characteristics of policies or of individuals and families. They are reasonably accurate and contain few response errors, understatements, or missing values. Good data are also available for a reasonably



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 102
Evaluating Welfare Reform in an Era of Transition 5 Data Needs and Issues Successful monitoring and evaluation of welfare reform are not possible without good data, regardless of how clearly the questions of interest are delineated and how strong the available evaluation methodologies are. Good data are therefore critical to studying welfare reform. In Chapters 3 and 4 we showed that there are a wide variety of questions of interest and that answering those questions requires a number of different methods. It should not be surprising that addressing the questions of interest for monitoring and evaluating welfare reform will therefore require use of multiple types of data from multiple sources. We categorize data sources into four generic types for our discussion. First are household surveys, both national and state level, which have been the data sources that have informed much of what is known about the low-income population. Second are administrative records from social welfare and other programs, which are a somewhat newer and emerging data source for studying welfare reform. Much of the new administrative data is available at the state level, but there are also a few federal-level data sets. Third are data describing policies and programs at the state and local levels. Fourth are qualitative data, another source of data that are increasingly being used in policy evaluation. Together, these four types of data constitute the data infrastructure for monitoring and evaluating welfare reform. Good data have many characteristics. They have reasonably good coverage of the population in question. They contain measures of the key variables of interest for welfare reform study, either characteristics of policies or of individuals and families. They are reasonably accurate and contain few response errors, understatements, or missing values. Good data are also available for a reasonably

OCR for page 102
Evaluating Welfare Reform in an Era of Transition long time frame and are comparable across time. The sample sizes in good data sets are large enough for reliable statistical estimation, in many cases at the state and local level. Finally, for the purposes of many welfare reform studies, though not all, good data are comparable across states and use the same concepts and definitions, so that valid cross-area comparisons can be made. What constitutes good data for addressing one question of interest may not suffice for another question of interest. Correspondingly, different evaluation methods require different types of data with different strengths and weaknesses. This chapter describes the various sources of data available for welfare program research. In doing so, we discuss their potential use for addressing the research questions of interest using the available evaluation methods, their use in current studies under way, and we compare them with the characteristics of good data. We also include a section on data confidentiality that has implications for almost all data collection. We note that most of these recommendations for improvements are specific to the current data infrastructure for social welfare program monitoring and evaluation. The panel concludes that this infrastructure has limitations and that the devolved nature of social welfare programs has exacerbated the limitations. The main limitation is that no agency within DHHS has the specific responsibility to collect data to monitor the well-being of the low-income population nor for evaluating the effectiveness of social welfare programs. The panel believes that this responsibility needs to be allocated to some administrative entity within DHHS to coordinate data collection activities at the federal level and to work with states to coordinate data collection activities at the state and local level. The final chapter of this report discusses this need in more detail. However, many of the specific recommendations for data improvements made in this chapter could be addressed more easily if this administrative authority is assigned. Therefore, in discussing these specific recommendations, we highlight areas where the existence of such an authority will help spur data improvements. Devolution has had important consequences on data needs for the study of welfare reform. It has resulted in a proliferation of different programs in different states and localities around the country, each mixing a different bundle of reform components and strategies and each targeting somewhat different populations. Adding these differences to existing cross-state variation in Medicaid and other welfare programs and to ever-present differences in labor markets, demographic profiles, and general socioeconomic environments, the demands for state-level and local-area data have grown tremendously. Devolution lies behind many of the recent developments in data collection for welfare reform studies and behind much of our discussion here. It affects the value of national-level surveys which have less to contribute in the current wave of reform than they had in past reforms. The number of state-level and local-area surveys, which historically have been quite rare, is growing. The value of state-level administrative data, which have the potential to capture state and local-area

OCR for page 102
Evaluating Welfare Reform in an Era of Transition details in a way that has not been important in past reforms, is also growing. At the same time, the comparability of these state and local data sets is of greater concern if generalizations about the consequences of reform are to be understood on a broader basis. Finally, it affects the need to collect state and local area program and policy information itself, a need which is now much greater. SURVEY DATA Data from question-and-answer surveys have been heavily used for social welfare program monitoring and evaluation in the past and will continue to be useful for future studies. This section of the chapter discusses the strengths and limitations of existing surveys for monitoring and evaluating social welfare programs and gives the panel’s recommendations for improving survey databases. It covers national-level surveys, which are designed to be representative of the U.S. population, and state- and local-level surveys, which are designed to be representative of state or local populations or subpopulations (like welfare leavers). The merits of particular surveys for particular purposes are discussed in terms of population coverage, sample size, content, nonresponse, response error, and periodicity. National-Level Survey Data There are several national-level surveys that are relevant to welfare program monitoring and evaluation. They cover such content areas as income, earnings, employment, program participation and benefit receipt, adult and child well-being measures, family structure, and demographic and other background information. The surveys discussed here (and summarized in Appendix D) include the long form of the decennial census, the March Supplement of the Current Population Survey (CPS), the American Community Survey (ACS), currently in the development stage, the Survey of Income and Program Participation (SIPP), the Survey of Program Dynamics (SPD), the National Survey of America’s Families (NSAF), the National Longitudinal Survey of Youth (NLSY), and the Panel Study of Income Dynamics (PSID). The CPS, ACS, SIPP, SPD and the census long form are all surveys funded by the federal government and conducted by the U.S. Census Bureau. NLSY is privately conducted but funded by the U.S. Department of Labor and the National Institute for Child Health and Human Development of the U.S. Department of Health and Human Services. PSID is conducted by the University of Michigan and supported by grants from the federal government and private foundations. NSAF is conducted by the Urban Institute with private funding. The NLSY, PSID, SPD and SIPP are longitudinal data and so have the added feature of tracking the changes in the well-being and outcomes of sample members over time. Table 5–1 contains basic summary information about these surveys.

OCR for page 102
Evaluating Welfare Reform in an Era of Transition Although these surveys all include content relevant to welfare reform evaluations, most are conducted for purposes other than welfare population monitoring and evaluation. Only the Survey of Program Dynamics has as its primary purpose the evaluation of welfare reform, although the SIPP, which was established before PRWORA, also has a special role as it was created to measure government and welfare program participation. Population Coverage National-level surveys are designed to produce analyses that are representative of the national population and, hence, are useful for producing estimates of the well-being of the nation as a whole.1 Most of these surveys are not, however, representative of smaller geographic areas, which is a significant disadvantage for studying welfare reform in an era of devolution. Data from the census long form are representative of state and local areas, but they are only produced once every 10 years, and so are not appropriate for timely monitoring or evaluation purposes. The ACS will be representative of smaller areas on an annual basis, and, hence, will be a major improvement in providing state and local level data on a far more timely basis. The ACS will be representative of states, large counties and governmental units with populations over 65,000. Eventually, multiyear averaged data representative of smaller areas will also be produced. Other national-level data sets (SIPP, SPD, NLSY, PSID) are of limited use for state-level monitoring because the state sample sizes are too small for precise estimates of state-level measures. The March CPS is large enough to produce annual state-level estimates, but the precision of estimates in most states is low.2 Other national surveys are representative of some states. For example, the Urban Institute’s NSAF is designed to be representative in 13 states (Alabama, California, Colorado, Florida, Massachusetts, Michigan, Minnesota, Mississippi, New Jersey, New York, Texas, Washington, and Wisconsin), though sample sizes for the low-income welfare participant population per se are quite modest. The SIPP and SPD are not representative of all states, but sample sizes are large enough in some states that state level estimates of outcomes can be produced with reasonable precision. 1   Except for the census long form, these surveys exclude institutionalized persons, much of the military, and the homeless populations, although the NLSY and PSID studies do follow sample members in and out of institutions. The NLSY79 is representative only of those aged 14–22 in 1979; the NLSY97 is nationally representative of youths aged 12–16 in 1997. The NSAF is representative of the nonelderly population. 2   The Census Bureau has funding to improve the precision of state-level estimates of the number of children with health insurance coverage by family income, age, race and ethnicity. Initial plans call for a significant increase in the sample size of the March Supplement, which should enhance the use of the CPS for state-level monitoring and evaluation.

OCR for page 102
Evaluating Welfare Reform in an Era of Transition TABLE 5–1 Key Features of National-Level Surveys Relevant for Monitoring Low-Income Populations and Evaluating Welfare Reform Feature 2000 Census Long Form American Community Survey (ACS) March Current Population Survey (CPS) National Longitudinal Surveys of Youth (NLSY79) Populations Represented U.S. households and individuals U.S. households and individuals U.S. civilian noninstitutionalized population age 15 and over Youth aged 14–21 in 1979 Levels of Geography National, state, local National, state, local National National Social Welfare Program Content Limited benefits information, specifically public assistance benefit amount and SSI benefit amount; income Limited benefits information—includes SSI, food stamps, cash assistance, and housing; income Moderate benefits information—includes AFDC, SSI, food stamps, cash assistance, school lunch, and public housing; income Extensive benefits information—includes AFDC, food stamps, cash and other public assistance; child outcomes; income Sample Size and Design Cross-section of approximately 18 million housing units in 2000 “Rolling” cross-section of three million households per year Rotating panel design of 50,000 Panel of approximately 13,000 initially; roughly 12,000 in 1985 when discontinued interviewing the military sample Oversampling Small governmental units Small governmental units Hispanics Hispanics, blacks, economically disadvantaged non-blacks and non-Hispanics (until 1990), enlisted military youth (until 1984)

OCR for page 102
Evaluating Welfare Reform in an Era of Transition Periodicity Once a decade Monthly Annual (income supplement) Annual until 1994 and biennial since Data Collection Mode Mail survey, personal follow-up for nonresponse Mail survey, phone follow-up, then personal follow-up for one-third of mail and phone non-respondents First and fifth interviews in person; other six interviews by phone Personal interviews, except in 1987 when phone interviews were conducted due to budgetary constraints Response Rates 1990 mailback response rate=60%; 2000 mailback response rate=54% Weighted response rate of more than 95% Until recently, response rates have been quite high. In recent years, they have been in the 80–82 percent range. Round by round response rates are high. Cumulative retention rate through 1998 is 84% of original sample (not adjusting for mortality) Data Release Dates Long-form data planned to be released in 2002 for 2000 census Goal is to publish six months after data collection Income and poverty data published for nation and population groups 6 months after data collection; limited data published for states on the basis of 3-year averages Most recent data available in 2000 was from 1998 survey; publish update biennially New Features Long form may not be included in 2010 or later censuses May replace census long form Recently received funding to expand sample size for state estimates of low-income children not covered by health insurance NLSY97 began in 1997 with nationally representative sample (oversample of blacks and Hispanics) of roughly 9,000 youths aged 12–16 years old

OCR for page 102
Evaluating Welfare Reform in an Era of Transition Feature National Survey of America’s Families (NSAF) Panel Study of Income Dynamics (PSID) Survey of Income and Program Participation (SIPP) Survey of Program Dynamics (SPD) Populations Represented U.S. civilian noninstitutionalized population under age 65 U.S. civilian noninstitutionalized population of households and of low income families U.S. civilian noninstitutionalized population U.S. civilian noninstitutionalized population in 1992–1993 Levels of Geography National and 13 states National National National Social Welfare Program Content Extensive benefits information, including AFDC, SSI, food stamps, WIC, and school lunch; child outcomes; income Extensive benefits information, including AFDC, SSI, food stamps, low income health services, and housing subsidies; child outcomes; income Extensive benefits information, including public assistance, food stamps, school lunch, health insurance, and WIC; income Extensive benefits information, including AFDC, SSI, food stamps, WIC, and health insurance; child and adolescent outcomes; income Sample Size and Design Repeated cross-section of 48,000 households Panel, originally 4,800 families and 6,434 as of 1999 Panel, size has varied from 11,000 to 37,000 households Panel of 18,500 households for 1998 Oversampling Children and low income Blacks; Hispanics in 1990–1995 Low income starting in 1996 Children and low income Periodicity 1997 and 1999; planned for 2002 Annual until 1997 and biennial since Every four months Annual Data Collection Mode Telephone for those with access to a phone; in person and by cell phone for people in the area sample without a telephone In 1999, 97.5% by telephone; all interviews used computer-based instruments First, second, and one interview in each subsequent year of a panel in person; other interviews by phone In-person and two self-administered surveys, one for adolescents; will conduct reinterview

OCR for page 102
Evaluating Welfare Reform in an Era of Transition Response Rates Overall response rate for adults was 62% in 1997 and 65% for children. 97–98.5% annually (mortality-adjusted) PSID does not attempt to interview attriters. Cumulative response rate was 39% in 1994 91–95% households to first wave (cumulative response rate was 69% by wave 8 of 1996 panel) 82% in 1997, 85% in 1998, 1999, 2000. Cumulative non-response was near 50% by 1998. Since then, attempts to interview noninterviews from previous years were made and the cumulative response rate has been not risen. Data Release Dates First data from 1997 released in January 1999 1999 data were added to the Wealth Files in February 2000; most recent data in all other files is 1997 Historically, one to two year (or more) lag from data collection to publication 1997 and 1998 data available. Longitudinal file for the years 1992–1998 will be released in the summer of 2001. New Features 1999 survey incorporated ways of measuring changes in child well-being Child Development Supplement (CDS) began in 1997 with in-home interviews of 3,500 children aged 0–12 years with oversample for blacks; these children will be followed into adulthood Requested funding to expand sample size and number of panels and to implement state-representative design Will be conducted through 2001 to collect data that enable evaluation of the 1996 federal welfare reform legislation and its impact on the American people SOURCES: Information from Brick (2000) and National Research Council (2000b).

OCR for page 102
Evaluating Welfare Reform in an Era of Transition National-level surveys have experienced some problems with undercoverage. The CPS has some population coverage problems, especially for specific subgroups that might be of interest for welfare program research. The average monthly coverage ratio (the ratio of the CPS population estimate compared with the census-based population estimate) for the CPS in early 1996 was 0.93. But for specific subgroups of black women between 16 and 39, the coverage ratio ranged from 0.82 to 0.87 (U.S. Census Bureau and Bureau of Labor Statistics, 2000). Continuing SIPP panels do not represent new entrants into the population such as immigrants and people returning from institutions (e.g., jail or longer term substance abuse facilities) which are relevant to welfare policy studies. Population Subgroups The national-level surveys are aimed to be representative of the population as a whole, but this implies that sample sizes for the population subgroups relevant to welfare reform, namely, single mother families who have lower incomes or welfare recipients specifically, may be problematic. For broader subgroups, such as single mother households, many national-level surveys are, in general, large enough to produce reliable estimates of well-being for the nation as a whole but rarely can they provide state-level reliable estimates.3 Moreover, studying specific subgroups who may be of interest for welfare reform, such as immigrants, disabled adults, or families with disabled children is more difficult, for the sample sizes in national-level surveys are almost always quite small for these groups, even for the nation as a whole. Subgroup analysis is necessary not only for description and monitoring but also for nonexperimental evaluations of the overall effect of welfare reform and of the broad components of reform. Of the currently available data sets, only the CPS has the sample size and statistical power for needed subgroup analyses, and even its usefulness is limited to estimating the overall effect of PRWORA and only for low variance outcomes, like the program participation rate (see the discussion of power in Chapter 4). The ACS, as it is currently being developed, has considerable potential for use in such cross-area subgroup analyses of broad components of reform, but it is untested. Nonresponse A major threat to the representativeness of all surveys is nonresponse. Nonresponse may bias estimates of outcomes if those who do not respond are systematically different from those who do respond. In a longitudinal setting, the inability to reinterview families for multiple waves of surveys may also cause bias problems. For surveys of low income populations, there may be particular 3   Table 5–1 shows which subpopulations are oversampled in national level surveys.

OCR for page 102
Evaluating Welfare Reform in an Era of Transition reasons to suspect there is bias due to nonresponse and attrition: it is likely that factors making it difficult to interview and reinterview respondents (in a longitudinal setting) may be correlated with their outcomes. For example, a lack of stable residence may indicate financial trouble for a family and may make it more difficult to locate a survey sample member. Homeless persons are rarely included in survey sample frames, as we noted earlier, and longitudinal survey respondents who become homeless are generally lost for future reinterview attempts. Weighting and imputation procedures can potentially reduce nonresponse biases although they can rarely eliminate them (see Kalton and Kasprzyk, 1986 and Little and Rubin, 1987). With specific attention to surveys of low-income populations, Groves and Couper (2001) discuss survey design considerations for reducing nonresponse and nonresponse adjustments and Mohadjer and Choudhry (2001) provide more detail on weighting adjustment procedures. Incentive payments to encourage sample members to respond to surveys have also been effective in increasing response rates in surveys. Initial evidence from a small number of experiments further suggests that incentive payments may be particularly effective with low-income populations (Singer and Kulka, 2001). There has been some movement towards using incentives payments for the SIPP and SPD. Response rates in the key national-level surveys vary considerably. CPS response rates are around 94–95 percent each month, although the response rates for the March CPS Supplement are a little lower.4 The ACS is still undergoing field tests and response rates are not available. However, in a 1996 test in four sites, the weighted response rates for the ACS were about 95 percent. The NSAF, which oversamples low-income households, had an overall response rate of 70 percent for the 1997 round and about 64 percent for the 1999 round (Safir, Scheuren, and Wang, 2001). For the longitudinal surveys, nonresponse and attrition over multiple waves is a significant threat to data quality. For the SIPP and SPD, response rates in the initial waves were high (between 91 and 95 percent for first panels of SIPP from 1984–1996 and 91 percent for the first wave of the SPD—which corresponded to the 1992 and 1993 panels of SIPP), but many first-wave respondents in both surveys have not been reinterviewed. By the eighth wave, the cumulative nonresponse rates for the 1984–1991 panels were between 21–22 percent, 25 percent for the 1992 and 1993 panels and 31 percent for the 1996 panel. This attrition seems to be the result of refusals, rather than the inability to track sample members (U.S. Census Bureau, 1998). The SPD sample is comprised of the 1992 and 1993 SIPP panels. The first SPD survey, the “bridge survey” in 1997, attempted 4   For example, in the 2000 CPS March Supplement, the response rate for the basic monthly labor survey was just over 93 percent, but 8 percent of the basic sample did not respond to the supplement and so the total response rate was 86 percent (U.S. Census Bureau and Bureau of Labor Statistics, 2000).

OCR for page 102
Evaluating Welfare Reform in an Era of Transition to interview all 1992 and 1993 SIPP panel members who had responded to each intervening SIPP wave, which was only 73 percent of the original 1992 and 1993 SIPP sample members. The bridge survey interviewed 82 percent of those households. For the next SPD interview in 1998, budget constraints resulted in a decrease in the sample size. Of the eligible households for the 1998 collection, 89 percent were interviewed. Accumulated over these waves, the response rate through the 1998 survey was only 50 percent. The Census Bureau has explored the degree to which attrition has affected the representativeness of the sample; it concluded that in comparing estimates of population well-being measures and characteristics, the SPD produces estimates similar to those of the CPS (Weinberg and Shipp, 2000). (We discuss this further in the section on longitudinal data.) Cumulative nonresponse has also been a problem for the PSID. The NLSY 1979 cohort has had good cumulative response rates, which can probably be attributed in part to keeping sample members in the sample even if they are nonrespondents for one or more waves of the survey. Timeliness In order to be useful for continual monitoring of the well-being of welfare prone populations, the data used in monitoring studies should be produced on a regular basis. The March CPS and SPD collect data on an annual basis. The decennial long-form collects data every 10 years. The NSAF collected data in 1997 and 1999. A fully implemented ACS will collect data every month. The SIPP collects data every 4 months. The timeliness of the release of the data is just as important, and is a severe limitation for some surveys. While the March CPS data are produced on a very timely basis (the data are usually available in the fall after collection), other data sets are not and are thus less useful for monitoring the well-being of the low-income population than they could be. This has been an especially significant problem for the SIPP, for which data release has often taken much longer than a year. For example, while the core data from all 12 waves of the 1996 panel have now been released, only the first few topical modules have been released as of yet, and no longitudinal file has been produced. Data release for the SPD is only slightly better; 1998 data were available in early 2001 and the first longitudinal file of the data set, covering 1992–1998, is scheduled to be released mid-2001. Thus, for two of the key surveys for monitoring welfare program participation, only very limited post-welfare reform data are available in early 2001, nearly 5 years after the reforms were enacted. Survey Content Another important aspect of the data for studying welfare reform is content. All of the national-level surveys discussed collect basic information relevant to

OCR for page 102
Evaluating Welfare Reform in an Era of Transition expensive. However, the assessment of data quality and preparation of these data for research use will be significant endeavors. The panel believes, however, that efforts to obtain the universe of data and to make them available for research purposes should be pursued to the greatest extent possible. Recommendation 5.9 Administrative data reported by states as part of the TANF reporting requirements will be of limited use for research purposes unless steps are taken to improve them. The usefulness of these data will be improved if the data can be linked to other data sets and if the full universe of cases is reported. The panel recommends that ACF take steps to improve the linkability of these data and encourage states to report the full universe of cases. DATA ACCESS AND CONFIDENTIALITY Answering important questions about the effectiveness of welfare programs requires a greater reliance on multiple sources of data and linkages between sources—including linkages among administrative data sets, between administrative and survey data sets, and across levels of government. Although the needs for multiple sources of linked data are great, restricted access to data, instituted to prevent individuals who provide data from being harmed by the improper use of the data, limit data currently available for use in research contexts. There are restrictions to access for both survey and administrative data. Such restrictions range from completely limiting the use of the data to only those within the collecting agency to no restrictions at all on the use of individual data. For many data sets, restrictions on data access are somewhere in between. Different models for data access include: granting access to group-level data instead of individual-level data; releasing scrambled data so that any data items containing individually identifying information are either not released or are modified to prevent individual identification; granting permission for researchers to access data at a centralized data holding center (e.g., as done in the Census Bureau data centers); housing data in secure holding centers but allowing researchers to specify the analyses they want conducted with the data as long as they are within the bounds of confidentiality requirements; and releasing individual data to those who agree to abide by terms for the use of the data and are subject to penalties if they are found to be using the data in ways other than the agreed-upon terms. The panel believes that confidentiality and access restrictions are often drawn in ways that unnecessarily limit the use of important sources of data for welfare program monitoring and evaluation. Confidentiality protocols limit the use of survey data for evaluation purposes. For example, nonexperimental methods of evaluation that compare outcomes of individuals across different areas must be able to control for the economic conditions and other characteristics of the local areas in which study participants live. This requires information about where the

OCR for page 102
Evaluating Welfare Reform in an Era of Transition individuals live (for example, a county, census tract, or zip code). Often, this information is not available from surveys, and therefore, poses a barrier to use of these data for evaluation. The state-based nature of the TANF program and limited budgets for evaluation magnify the importance of administrative data sets and linkages of administrative data sets for program evaluation. Although these data are needed now more than ever to inform policy decisions, they are often not accessible. A barrier to data access that is particularly acute with the devolved program structure for TANF is that rules and protocols governing data access vary considerably across states and even across agencies within states (Brady et al., 2001). A study examining state social service administration practices for releasing data found that some state social service agencies had well-developed policies for release, while other states did not (UC Data, 1999). Cross-agency data sharing at the federal level is also hampered by the variation in data sharing protocols among federal agencies (National Research Council, 1993 a, 2000a). Variation in policies and practices regarding release of data and confidentiality protections across both state and federal agencies is a particularly complex problem for linking administrative data from different agencies for research purposes. Gaining access to data from multiple agencies is more difficult because data-sharing agreements with each agency providing data must often be negotiated separately, a process that may take considerable time. Linking data from different sources requires some individually identifying information for each data point to be matched with similar information from another data set. Social security numbers are a typical example of such information. However, some agencies have moved away from using SSNs as identifiers and assign their own identifying numbers to cases. A problem then for linking is that identifiers in each of the data sets being linked may differ across each agency. Different agencies also have varying degrees of experience, resources, and technical know-how for releasing data for research purposes, and, thus, policies and practices for releasing data have developed unevenly. This is especially true across state agencies, partly because data collection and evaluation efforts were not previously focused at the state level and resources for data collection and linkage were more limited. With an increased focus on state-level data and evaluations, state agencies could use advice on different models for data sharing and new technical advances that can be used to protect confidentiality of individual data. This is another area for which DHHS can convene conferences or meetings with state and federal agency representatives and with experts in data access and confidentiality. In the longer term, leadership in coordinating data linkages from the proposed data collection authority in DHHS that is responsible for data collection for social welfare programs (see Chapter 6) could speed the coordination of confidentiality protocols and take steps to lower data access barriers. It is crucial for data collection agencies to ensure that the individuals who

OCR for page 102
Evaluating Welfare Reform in an Era of Transition provide data are not harmed from misuse of the data. The reputation of the agencies charged with collecting data or administering programs may rest on whether individuals believe that these data will be used appropriately or that they will not be harmed by providing the data. The issues of data access and confidentiality are, however, much deeper and more complex than any single agency or state. The larger scientific, legal, and governmental communities will need to be involved in resolving current tensions between opening data access while protecting individual privacy. One factor in the tension between open data access and protecting privacy is that current laws governing the use of data often do not specifically address the use of the data for research purposes.13 Most privacy laws allow administrative data to be used for activities in accordance with the program’s purposes or include clauses for “routine use” of the data. It is often through these provisions of the laws that agencies grant access to the data to policy analysts outside the agency (Brady et al., 2001). But because the laws do not directly address the use of data for research purposes, these laws can be interpreted quite differently. Policy makers and agency officials with control of the data are not always aware that researchers have no interest in the personal identities of specific individuals providing data, but rather, only have a need for information on random individuals. Because they do not understand how researchers use the data, agencies that control data too often limit access to the data for research purposes, even though the data could be valuable for monitoring and evaluation. Rules governing the use of individual data collected from state and federal government agencies need to clarify how the data can be used for research and evaluation purposes. Provided such clarifications can be made and agreed on, an effort to implement them as consistently as possible across and within state and federal agencies is needed. The benefits of access to data for program monitoring and evaluation purposes need to be better communicated to agency officials that grant data access. Alternatives to simply eliminating access to data includes stricter enforcement of rules governing the use of restricted data to discourage improper disclosure (National Research Council, 2000a), as well as advancing the development of techniques for data disclosures that protect confidentiality, such as data masking or data perturbations (Brady et al., 2001). New technologies may help to provide external access to linked data sets while meeting confidentiality requirements. For example, the Longitudinal Employer-Household Dynamics Pilot Project at the Census Bureau and the related Dynamic Employer-Household Data and the Social Data Infrastructure projects are exploring the use of web-based and video teleconferencing tools to provide research data. The panel has found that data access and confidentiality restrictions are 13   See Brady et al. (2001) for a review of legislation specific to administrative data uses and National Research Council (1993a) for a review of legislation regarding data collected through federal agencies and for a review of agency-specific confidentiality practices.

OCR for page 102
Evaluating Welfare Reform in an Era of Transition significant barriers to the availability of data for social welfare policy evaluation. As these issues begin to be addressed on many fronts, the panel emphasizes the need for data linkages in welfare program evaluations and encourages steps to remove barriers to data access. Recommendation 5.10 Confidentiality, privacy, and access concerns with administrative and survey data and the linking of multiple data sets are important issues, but are currently serving as a barrier to socially important evaluation of welfare reform programs. The importance of access to these data for monitoring and evaluation of programs should be emphasized and efforts to reduce these data access barriers while protecting privacy and maintaining confidentiality should be expanded. PROGRAM DESCRIPTION DATA Another type of data that are crucial for social welfare program evaluation are program description data, or descriptions of the TANF and related programs that states have adopted and are operating. The importance of such data for evaluation seems obvious—programs cannot be compared if the rules of the programs are not known. Yet as the panel stated in its interim report, good program description data on post-PRWORA state and local programs were slow to develop. This lack of data has been a major limitation for welfare reform evaluation, especially for cross-state monitoring and evaluation. In relation to monitoring the low-income population on a national level, it is important to know what the different state programs are so that national surveys can ask respondents the right questions about the program benefits they receive. Nonexperimental methods of evaluating the overall effect of welfare reform and of broad components of reform rely on fairly detailed program description data for both TANF policies and other related non-TANF policies in states and localities so that these variables can be included in models to control for the different policies that apply to each case. Program description data need to cover programs in effect during the entire study period. For national-level evaluations, program description data are needed from every state and for every locality within states for which different rules apply. For state-level evaluations, program description data are needed for a given state only, unless different localities within the state have different policies. The federal government’s efforts to collect program description data are, thus far, limited. States are required to provide summaries of their state TANF programs every 2 years in order to receive block grants. The first collection was in 1997 and the second in 1999. States are given general guidance on what to report, but there are no specific requirements. As a result, the level of detail that the states have provided about the programs varies greatly. These state summa-

OCR for page 102
Evaluating Welfare Reform in an Era of Transition ries are just now being organized so that they can be made accessible to the public. Aside from the state plans required to be reported to DHHS every 2 years, there are several efforts under way to document state policies and practices (see Chapter 2). The most comprehensive effort has been conducted by the Urban Institute as part of their Assessing the New Federalism Initiative. This database is the most comprehensive in time documenting rules from 1996–2000. It includes data on eligibility rules, asset rules, benefit levels, work activity requirements, time limits, sanction policies, behavioral requirements, and child support requirements. The data have been coded to be more readily usable in quantitative analyses and are publicly available (see http://www.wrd.urban.org). The 2000 round was sponsored by ACF and current plans call for updates in 2001 and 2002. In another effort, the Center for Law and Social Policy and the Center for Budget and Policy Priorities are jointly collecting information on state TANF and Medicaid policies. Currently, this database includes information on state policies regarding: TANF applications, cash diversion programs, emergency assistance, categorical and financial eligibility rules, family cap policies, minor living arrangement provisions, school and training requirements for minors, abstinence education programs, and Medicaid (see http://www.spdp.org). This information was collected for legislation enacted before and updated through 1998. In the future, information about time limits, work activities and requirements, sanction provisions, child care assistance, child support, and drug-related provisions will be included in the database. While this database contains some information not included in the Urban Institute database, it does not cover the full time frame since PRWORA that the Urban Institute database does. The Congressional Research Service (CRS) has also used state TANF plans reported to DHHS-ACF to produce reports summarizing state program rules on different topics. For example, the last summarized key features of state TANF programs, such as: treatment of earnings and savings, welfare diversion, work requirements, personal responsibility plans, division of the welfare caseload, and benefit levels (Burke et al., 1999). Every 6 months states are surveyed to collect information about financial eligibility and benefit determination rules in effect. The last update includes rules in effect through January 2000 (Abbey et al., 2000), and future updates of this report are also planned. Finally, the CRS has a continuing series of reports called Cash and Noncash Benefits for Persons with Limited Income that covers over 80 federal programs that serve low-income people that is produced every other year. The latest version was produced in 1999 (Burke et al., 1999), and another version is due in late 2001. It may seem that all these ongoing efforts to collect TANF program description data would translate into sufficient, even redundant, program description data, but this is not the case. Most of these data collection efforts are not conducted on a regular basis and do not cover the entire time frame of interest. Only the Urban Institute database comprehensively covers programs and changes in

OCR for page 102
Evaluating Welfare Reform in an Era of Transition programs in operation between 1996 and the present. The state plans provided to ACF and summarized by the joint effort between ACF, Welfare Information Network, American Public Human Services Association, and National Governors Association (discussed in Chapter 2) will eventually be available every other year. The State Policy Document Project only covers programs in effect at one point in time, thus far. Another weakness of many of these efforts is their coverage of all programs and rules. It is not at all clear that data submitted to the federal government will be detailed enough in every state and consistently reported across states to comprehensively cover all TANF and other state welfare programs, since there are few detailed requirements of what states have to report. Again, the Urban Institute’s database is probably the best source of detailed data, but it does not include information about other related non-TANF program rules. Other data collection efforts provide some needed information, but these efforts do not comprehensively cover all programs. Thus, evaluations that depend on full descriptions of TANF and non-TANF program rules (such as time-series modeling, comparison group modeling, and cross-state policy and timing variation models) will have a difficult time controlling for all the policies that may affect individuals in the study, and thereby, isolating the effects of specific policies. A further limitation of these data is caused by the changes in the ways agencies have actually implemented their policies. None of these sources of program description data collect detailed information on program implementation (see below). A key issue for the collection of program description data is whether efforts like the Urban Institute’s can be sustained. There are positive signs in this direction as ACF funded the 2000 round of the rules update, and plans for future rounds are now set. However these data need to be continually updated and expanded to cover other related programs. Data collection of program rules, in a sufficiently comprehensive, detailed, and consistent manner across states, should be an institutionalized component of DHHS’s duties for administering social welfare programs. This work requires the institutional commitment of a government agency to ensure that the data are collected and provided to users in a readily usable form. Recommendation 5.11 The monitoring and documentation of the actual policies, programs, and implementations of welfare reform at the state and local levels by the federal government has been minimally adequate to date. The panel recommends that the Department of Health and Human Services take active and direct responsibility for documenting and publishing welfare program rules and policies in every state and in every substate area where needed. Continuing updates documenting changes in state and local area rules should also be produced.

OCR for page 102
Evaluating Welfare Reform in an Era of Transition In Chapter 4 the panel recommends that more process studies should be funded in order to provide more detailed descriptions of how state and local policies have actually been implemented. These detailed descriptions are needed in evaluation efforts, especially in evaluating the effects of specific detailed welfare reform strategies, to fully characterize the policies that cases are subject to and in turn, to use these characterizations to assess the effect of the policies on outcomes. We emphasize the need for more process studies that can provide this detailed data on program policies. QUALITATIVE DATA Qualitative data on individuals are another important source of data for welfare program monitoring and evaluation. While Chapter 4 highlights potential uses of qualitative data for evaluation purposes, the discussion in this chapter focuses on data collection. The collection of qualitative data for addressing key monitoring and evaluation questions posed in Chapter 3 is likely to be most useful when conducted as part of other data collection efforts, such as surveys or administrative data collections (see Newman, 2001). Sometimes these data collections are incorporated into larger surveys, more often they are not. Openended questions, or questions that do not require a respondent to choose among a fixed set of answers (yes or no, numbered choices, etc.) allow respondents to provide detailed responses or explanations in their own words. These are typically conducted within the framework of a larger survey with other closed-ended questions. In-depth interviews are question-and-answer interviews that include more open-ended questions than a survey instrument and typically do not include fixed choice questions. Such interviews may be conducted on a subsample of a larger survey sample, or on an independent sample of individuals. Qualitative longitudinal studies with in-depth interviews conducted on the same individuals or families over multiple interview sessions are also common. Focus groups are conducted with small groups of individuals to discuss topics of interest, with a facilitator prompting participants and guiding discussion. Finally, participant observation fieldwork directly observes behavior of study participants. Data are collected through notes the researcher takes while observing the behavior of study participants, usually on a day-to-day basis. Participant observation is typically combined with interviews of study participants. Process analyses may also collect qualitative data to provide detailed information on programs or on a system that administers benefits and are another use of qualitative data. Data for process studies is gathered by visiting program offices (often across multiple service delivery areas); conducting surveys, indepth interviews or focus groups with key stakeholders in the delivery system such as program participants, caseworkers, and administrators; directly observing client and caseworker interactions; reviewing documentation of individual cases;

OCR for page 102
Evaluating Welfare Reform in an Era of Transition and special tracking of a cohort of participants through the processes of the program. The goals of qualitative data collection are generally to provide more detailed information and sometimes more subjective information on individuals or groups that is not easily quantified or measured with survey or administrative data. Besides providing more detailed insights into the outcomes of individuals, qualitative data can also be used to improve quantitative data collection. Openended questions embedded in surveys can be used to develop future survey questions (Newman, 2001). For example, an open-ended question may solicit answers that are fairly consistent across respondents. In future surveys, those responses might be incorporated as fixed-choice questions. Qualitative studies, perhaps through in-depth interviews or longitudinal in-depth interviews, might also be useful for understanding family organization, which can be used to improve the design of surveys that are designed to gather information about family members (Newman, 2001). Participant observers and in-depth interviews may be used to understand how surveys can be successfully targeted to hard-to-interview populations for surveys. Despite their potential use for welfare program monitoring and evaluation, qualitative and ethnographic data collections have several limitations. Qualitative studies are not always designed to statistically represent a population of interest. Because qualitative and ethnographic studies are expensive to conduct, the sample sizes of these studies are generally small, so that even those studies with representative samples of the population of interest are often too small for precise statistical analyses. Thus, qualitative studies are often most effective when nested within a larger survey or administrative records study to complement the information collected from quantitative data sources. A data collection issue here that qualitative and ethnographic researchers disagree on is whether it is more desirable to draw a separate sample of individuals within the same geographic area and with similar demographic characteristics as the survey or administrative records sample (as is the case with the Urban Change Project and the Three City Study) or to draw a subset of the survey or administrative records sample (as the ethnographic component of the Fragile Families Study has done). While the subsample strategy has the advantage of providing quantitative and qualitative data for a subsample of the study participants, the in-depth nature of many qualitative studies increases respondent burden for the subsample, which may make differential attrition an issue (Newman, 2001). Yet recruiting a separate sample of study participants for the quantitative component can be costly. Replication of results is rare for qualitative and ethnographic studies, which is a serious shortcoming. To remedy this problem, qualitative and ethnographic researchers should work closely with one another to coordinate their efforts so that there is some consistency in sampling and data collection across multiple locales; this is rarely done. The exception is for those studies that use qualitative

OCR for page 102
Evaluating Welfare Reform in an Era of Transition or ethnographic studies across multiple areas. Moreover, qualitative and ethnographic studies often do not sufficiently detail the methods by which the sample was selected and the data were collected, stored, sorted, and analyzed. This deficiency contributes to the problem of trying to replicate studies. Thus, it is incumbent on qualitative researchers to better specify their data and methods when reporting their results. While the potential uses of qualitative data in evaluation settings are becoming more apparent, these data are not routinely collected as part of evaluations of programs. The barrier is usually budget constraints, when quantitative outcome evaluations are of foremost importance for the sponsoring agencies. This priority is especially true for state program agencies sponsoring evaluations of their programs. Although many administrators see a useful role for ethnographic research (Newman, 2001), government-sponsored ethnographic studies are rare, especially at the state level.14 In the absence of a great deal of funding for extensive large-scale qualitative studies, smaller-scale studies embedded in larger quantitative studies may be the most feasible way to obtain needed qualitative data (Newman, 2001). SUMMARY AND ASSESSMENT OF THE CURRENT DATA INFRASTRUCTURE This chapter reviews the state of currently available data for welfare program evaluation and monitoring needs. Our discussion has focused on four types of data: survey data, administrative data, data describing programs and policies, and qualitative data. Addressing the key questions of interest for evaluating welfare reform requires the use of all of these kinds of data. While in many respects the existing data are very rich and extensive and furnish a wide range of valuable information on the low-income population and welfare recipients, in many critical respects they have significant weaknesses. These weaknesses are sufficiently severe that the panel has concluded that inadequacies in the nation’s data infrastructure for social welfare program study constitutes the major barrier to good monitoring and evaluation of the effects of reform. National-level surveys have an increasingly difficult task of measuring program participation in a setting in which programs vary widely from state to state. In addition, the sample sizes of many national-level surveys, though completely adequate for nationwide totals, are inadequate for the study of most of the state-level welfare program components (such as time limits, work requirements, etc.). The surveys often have major problems of nonresponse, which is often concen- 14   One exception is a study under way in South Carolina conducting in-depth interviews with a sample of welfare leavers (Medley et al., 1999).

OCR for page 102
Evaluating Welfare Reform in an Era of Transition trated in the lower income groups in the sample. Finally, timeliness of data release and availability is a major barrier to the analysis of welfare reform, a problem particularly acute for SIPP, which has released very little post-PRWORA data at the time of this writing. State-level surveys are in much worse conditions, primarily because states are only beginning to conduct such surveys. Most such surveys are conducted by telephone and response rates are often very low. Currently, they cannot be a major resource for the study of welfare reform. State-level administrative data hold more promise in an era of devolution because they provide fairly large samples at the state level. But their use as research tools, rather than as management information systems, is still in its infancy. Administrative data sets are usually quite complex and usable only by those who are expert in understanding the coding of the data, which makes the data difficult to use by any wide set of evaluators or researchers. The data are sometimes not available over time, because states have had no reason to save them in the past. The lack of historical information on individuals in data sets puts limits on their use for evaluation and monitoring. Evaluation efforts are also hindered by the lack of cross-state comparability in data items and variable definitions, which makes cross-area methods of evaluation problematic with these data. An inherent limitation of administrative data is, of course, that they are only available for those who receive welfare. Data describing state policies and programs have improved in quality and quantity in the last year. However, the development of databases with this critical information will need to continue and will need significant support from DHHS. Furthermore, the databases need to be significantly widened to cover more than TANF program rules, while at the same time maintaining historical dimensions with regularly recorded rules for all the states going back several years. Collecting information about the actual implementation of the official rules continues to be a very difficult problem. Qualitative data in the form of ethnographic information on families is an underused source of information in program evaluation on social welfare reform. Neither administrative nor survey data can fully characterize the complexity of individual families’ lives and the way different types of families respond to welfare reform. However, researchers working with qualitative data need to continue to develop standardized protocols for collecting data and documenting how the data were collected, and they need to extend the data to cover a more representative set of areas and population groups. To date, these data have not played a major role in welfare reform evaluation despite their potential. There is a critical need to address the data barriers hindering good evaluation and monitoring as these data barriers limit what is known about the effects of PRWORA and welfare reform. As we note in our previous two chapters, ASPE has an important role to play in addressing these barriers. ASPE has already

OCR for page 102
Evaluating Welfare Reform in an Era of Transition committed significant money and staff resources to the improvement and maintenance of national-level survey data sets and has assisted states in developing their own databases, both administrative and survey. However, much more needs to be done. The panel concludes that the overall weaknesses in the data infrastructure have not been fully realized by the wider community of evaluators and policy makers. In part, this is because there is no established mechanism for assessing data quality and reporting on it to the public. Consequently, the panel believes that ASPE should include, in the Annual Report to Congress, a review of the state of the data infrastructure for welfare reform research. The review should cover survey and administrative data at federal and state levels. It should identify strengths and weaknesses of existing data and should note gaps that need to be filled and give an assessment of what the highest priorities for filling the gaps are so that resources can be effectively allocated to those projects. Recommendation 5.12 In its Annual Report to Congress, ASPE should review current availability and quality of data for welfare reform research, identify high-priority data needs, and discuss its own research agenda for data development and technical assistance. Building a data infrastructure for welfare program monitoring and evaluation will take a concerted effort from federal and state governments, all of whom have interests in social welfare program evaluation and monitoring.