Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 275
Studies of Welfare Populations: Data Collection and Research Issues 9 Measuring Employment and Income for Low-Income Populations with Administrative and Survey Data V.Joseph Hotz and John Karl Scholz With passage of the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) of 1996 and the expansions of the Earned Income Tax Credit (EITC) over the past decade, increasing attention has been paid to the employment experiences, labor market earnings, and transfer income received by disadvantaged individuals and households. This attention, prompted by explicit performance goals in PRWORA and implicit goals of the EITC expansions, focuses on whether low-income households can achieve self-sufficiency without resorting to Temporary Assistance for Needy Families (TANF) for other public assistance programs. Although income and employment levels are only partial indicators of the well-being of households, they continue to be ones most often used to assess the consequences, intended and unintended, of welfare reform. More broadly, good measures of income and employment for low-income families are necessary to (1) assess the well-being and labor market attachment of low-income and welfare populations at the national, state, and local levels; (2) evaluate welfare reform and learn the effects of specific policies, such as time limits and sanctions; and (3) meet reporting requirements under TANF and aid in the administration of welfare programs. There are two data sources for measuring employment and incomes of the disadvantaged: survey data and administrative. Surveys have been the mainstay of evaluating welfare programs and of monitoring changes in income and employment for decades. These include national surveys—such as the U.S. Censuses of Population, the Current Population Survey (CPS), the Survey of Income and Program Participation (SIPP), the National Longitudinal Surveys (NLS), and the Panel Study of Income Dynamics (PSID)—and more specialized surveys that
OCR for page 276
Studies of Welfare Populations: Data Collection and Research Issues gather data for targeted groups, such as current or former welfare recipients, and at the state or local level.1 Although survey data continue to be important, the use of administrative data sources to measure income and employment has grown dramatically over the past 30 years. Data on wages and salaries from state Unemployment Insurance (UI) systems, for example, have been used to measure the earnings and employment of individuals that participated in state AFDC/TANF programs, manpower training, and other social programs. Data on earnings (and employment) from Social Security Administration (SSA) records have been linked with the records of welfare and social program participants. What type of data one uses to measure income and employment among current and past welfare participants and welfare-eligible households may have important consequences for implementing and evaluating recent welfare reforms. Recent debates between the states and the federal government, for example, over employment targets and associated sanctions mandated under PRWORA hinged crucially on exactly how the fraction of a state’s caseload that is employed would be measured. Furthermore, the conclusions of several recent assessments of the impacts of welfare reform and caseload decline appear to depend on how income and employment of welfare leavers and welfare-eligible populations are measured.2 In this paper we assess the strengths and weaknesses of using survey or administrative data to measure the employment and income of low-income populations. We review a number of studies, most of which have been conducted in the past 10–15 years,3 that assess the comparability of income and employment measures derived from surveys and administrative records. Clearly the primary criterion for evaluating data sources is their accuracy or reliability. Ideally one would compare the income and employment measures derived from either surveys or administrative data sources with their true values in order to determine which source of data is the most accurate. Unfortunately this ideal is rarely achieved. One seldom, if ever, has access to the true values for any outcome at the individual level. At best, one only can determine the relative differences in measures of a particular outcome across data sources. In this paper, we try to summarize the evidence on these relative differ- 1 Often these samples are gathered in the context of evaluations of specific welfare or training programs. 2 See, for example, studies by Primus et al. (1999), Cancian et al. (1999), and Rolston (1999) for a flavor of how this debate hinges on measurement issues. 3 Several earlier studies compared employment measures for low-income populations across alternative data sources, most notably the study by Greenberg and Halsey (1983) with data from the SIME/DIME Experiments. Given changes over time in such things as Unemployment Insurance coverage and response rates in surveys, we focus on the most recent studies available to maximize the relevance of our findings for the measurement of these outcomes for current and future studies.
OCR for page 277
Studies of Welfare Populations: Data Collection and Research Issues ences and the state of knowledge as to why they differ. These studies point to several important dimensions along which surveys and administrative records differ and, as such, are likely to account for some, if not all, of the differences in the measures of income and employment derived from each. These include the following: Population Coverage: Surveys generally sample the population while administrative data typically cover the population of individuals or households who are enrolled in some program. In each case issues arise about the sizes of samples at state or substate levels and sample designs that may limit the issues that can be examined. Reporting Units: Different data sources focus on individuals, households, tax-filing units, or case units. Differences in reporting units hinder the ability to move across data sources to obtain measures of income and complicate efforts to evaluate the differential quality of income data across data sets. Furthermore, differences in reporting units may have important consequences for the comprehensiveness of income measures, an issue especially relevant when attempting to assess the well-being, and changes in the well-being, of disadvantaged populations. Sources of Income: Data sources differ in the breadth of the sources of individual or household income they collect. Surveys such as the CPS and, especially, the SIPP, attempt to gather a comprehensive set of income elements, including labor earnings, cash benefits derived from social programs, and income from assets. In contrast, administrative data sources often contain only information on a single type of income (as in the case of UI earnings) or only those sources of income needed for the purposes of a particular record-keeping system. Measurement Error: Different data sources may be subject to different sources of measurement problems, including item nonresponse, imputation error, and measurement error with respect to employment and income (by source). Furthermore, issues such as locating respondents, respondent refusals, and sample attrition are important in conducting surveys on low-income populations. Incentives Associated with Data-Gathering Mechanisms: Data sources also may differ with respect to the incentives associated with the gathering of information. In the case of surveys, respondents’ cooperation may depend on a comparison of the financial remuneration for a survey with the respondent “burden” associated with completing it. In the case of administrative data, the incentives relate to the administrative functions and purposes for which the information is obtained. What is important is attempting to anticipate the potential for and likelihood of biases in measures of income and employment that may result from such incentives. The importance of various strengths and weaknesses of different data sources for measuring employment and income generally will depend on the purpose to
OCR for page 278
Studies of Welfare Populations: Data Collection and Research Issues which these measures are put. We note five considerations. First, when conducting an experimental evaluation of a program, the criteria for judging data sources is whether they yield different estimates of program impact, which generally depends on differences in income (employment) between treatment and control groups. In this case, errors in measuring the level of income between treatment and control groups could have little effect on the evaluation. Alternatively, suppose one’s objective is to describe what happened to households who left welfare. In this case, researchers will be interested in the average levels of postwelfare earnings (or employment). We discuss results from Kornfeld and Bloom (1999) where UI data appear to understate the level of income and employment of treatments and controls in an evaluation of the Job Training Partnership Act (JTPA), but differences between the two groups appear to give accurate measures of program impacts. Depending on the question of interest, the UI data may be suitable or badly biased. Second, surveys, and possibly tax return data, can provide information on family resources while UI data provide information on individual outcomes. When assessing the well-being of case units who leave welfare, we often are interested in knowing the resources available to the family. When thinking about the effects of a specific training program, we often are interested in the effects on the individual who received training. Third, data sets differ in their usefulness in measuring outcomes over time versus at a point in time. UI data, for example, make it relatively straightforward to examine employment and earnings over time, while it is impossible to do this with surveys unless they have a longitudinal design. Fourth, sample frames differ between administrative data and surveys. Researchers can not use administrative data from AFDC/TANF programs, for example, to examine program take-up decisions because the data only cover families who already receive benefits. Surveys, on the other hand, generally have representative rather than targeted or “choice-based” samples. Fifth, data sources are likely to have different costs. These include the costs of producing the data and implicit costs associated with gaining access. The issue of access is often an important consideration for certain sources of administrative data, particularly data from tax returns. The remainder of this paper is organized as follows: We characterize the strengths and weaknesses of income and employment measures derived from surveys, with particular emphasis on national surveys, from UI wage records, and from tax returns. For each data source, we summarize the findings of studies that directly compare the income and employment measures derived from that source with measures derived from at least one other data source. We conclude the paper by identifying the “gaps” in existing knowledge about the survey and administrative data sources for measuring income and employment for low-income and
OCR for page 279
Studies of Welfare Populations: Data Collection and Research Issues welfare-eligible populations. We offer several recommendations for future research that might help to close these gaps. USING SURVEY DATA TO MEASURE EMPLOYMENT AND INCOME In this section, we discuss the strengths and weaknesses of measuring income and employment status for low-income populations using survey data. Most of our analysis focuses on the use of national surveys—CPS and SIPP in particular—because of the availability of several high-quality studies that compare their income and employment measures to other data sources. Where available, we also summarize studies that assess income and employment measurement with more targeted surveys. TABLE 9–1 Key Features of Selected National Surveys That Report Employment and Income Status of Individuals and Households Feature Current Population Survey (CPS) Survey of Income and Program Participation (SIPP) Panel Study of Income Dynamics (PSID) National Longitudinal Survey of Youth, 1979 (NLSY79) Nationally representative sample? Yes Yes Only at sample inception in 1968 No, but representative for cohorts covered at sample inception Primary unit of analysis Household Household Household Individual Longitudinal data? No Yes Yes Yes Typical sample size 60,000 households 21,000 households 8,700 households 11,400 individuals Capacity for state and local Analysis For all but small states For large states only Limited Limited Coverage of income sources Broad Very broad Broad Very broad Accuracy of earnings dataa 97% 92% — — Accuracy of AFDC datab — — Timeliness of data Several months 2+ years 2-year lag 1–2 year lag aFor 1990, See Table 9–3. bAFDC=Aid to Families with Dependent Children, for 1990, see Tables 9–2 and 9–3.
OCR for page 280
Studies of Welfare Populations: Data Collection and Research Issues The key features of the national surveys for the purposes of this paper are summarized in Table 9–1. Potential Strengths The CPS and SIPP are vital data sets for understanding the functioning of low-wage labor markets and the effects of antipoverty programs. These data get high marks on many of the concerns mentioned in the introduction. They have a national sampling frame covering program participants and nonparticipants that make these data valuable for developing a broad perspective on developments in low-wage labor markets. An example of this type of study is Primus et al. (1999), which uses CPS data to show that AFDC/TANF and Food Stamp Program participation rates have declined considerably faster than poverty rates between 1993 and 1997. They further report that incomes of poor single mothers fell between 1995 and 1997 (after rising between 1993 and 1995), and that the safety net is lifting fewer children from poverty than in the past. Concerns arise with this study, some of which are mentioned in the text that follows. Nonetheless, the CPS and the SIPP are the only data sets that would allow analysts to address the important issues that Primus et al. examine on a national scale. The other national data sets that have been used to analyze the employment and income status of low-income populations are the National Longitudinal Survey (particularly the National Longitudinal Survey of Youth 1979) and the PSID. Both of these data sets have the additional feature that they are longitudinal surveys so that one can obtain information on earnings and employment status over time for the same person (and household).4 The PSID has surveyed, until very recently, its respondents and the “splitoffs” of initial respondent households on an annual basis since 1968. Similarly, until 1994 the NLSY79 conducted annual surveys of a random sample of individuals who were 14–21 years of age in 1979. Both of these surveys gathered detailed information on labor market earnings and employment status of respondents, earnings and some employment information on other adult household members, and some information on other sources of income, including income from various public assistance programs. One of the advantages of longitudinal data sets such as SIPP, PSID, and NLSY is that they allow one to monitor the entry into and exit from welfare or other social programs and the factors related to welfare dynamics, including changes in earnings and family structure. The CPS, SIPP, and PSID, in addition to having nationally representative samples, focus on households as the unit of analysis, and include information on all adult household members.5 Given the general presumption that families pool 4 Each wave of the SIPP is a longitudinal survey with between 2.5 and 4 years of data on the residents of a sample housing unit. Surveys to these respondents are asked every 4 months. 5 The NLSY79 focuses on the original respondent, but it gathers a considerable amount of information on the respondent’s spouse and/or cohabiting partner.
OCR for page 281
Studies of Welfare Populations: Data Collection and Research Issues resources, data sets that focus on families or households (and include information on cohabiting partners) are valuable. A calculation in Meyer and Cancian (1998) illustrates the usefulness of having data on family, as well as individual, incomes. Their study examines the economic well-being of women in the 5 years after leaving AFDC. They show that in the first year upon exit from AFDC, 79 percent of the women have incomes below the poverty line, but when family income is considered, a smaller number, 55.5, have income below the (correspondingly larger) poverty line. After 5 years, 64.2 percent of the women still have incomes below the poverty line, while only 40.5 percent of the broader family unit had income below the poverty line. The nationally representative surveys provide information on multiple sources of income, especially in the SIPP, either through separate questions or prompting of specific income sources. By asking specific questions about, for example, welfare receipt or food stamps, the data identify participants and (eligible) nonparticipants, so the data can be used to study program entry effects. The national surveys also measure income and employment in a comparable fashion both over time and across geographical locations, though in January 1994 the way that earnings information was elicited in the CPS was changed (Polivka, 1997).6 Another strength of the nationally representative surveys is that questions can be modified to reflect changing circumstances. For example, the U.S. Census Bureau periodically conducts cognitive interviews of respondents to the CPS in order to assess how they responded to different CPS income- and welfare-related questions. Such studies are used to determine which of the CPS questions were confusing and how respondents interpreted questions. Results from these cognitive interviews are used to improve the way questions are asked, with the goal of improving the quality of the data on key variables such as income and program participation.7 Typically, this sort of sophisticated assessment can only be done on large-scale, national surveys. To summarize, there are several potential strengths of using survey data to measure income and employment. These include the following: Surveys can provide representative samples for specific populations and generally include data for other family members. 6 Previously, earnings had to be reported in weekly amounts, and amounts over $2,000 per week were truncated. Now earnings can be reported over any interval and the data (to Bureau of Labor Statistics) are not truncated. Studies that use repeated cross-sections of the CPS that span 1994 risk misinterpreting results if they fail to account for the redesign. Polivka provides adjustment factors for earnings (at the 10th, median, and 90th percentiles) reported prior to 1994 to make the series comparable. She also shows that top-coded values that are imputed using a Pareto distribution do a good job of fitting the distribution of data that are not top coded. 7 See Bogen et al. (1997) and Bogen (1998).
OCR for page 282
Studies of Welfare Populations: Data Collection and Research Issues Surveys typically provide demographic data and data on other characteristics of households (such as educational attainment). They also may gather detailed information on many distinct income sources. National surveys provide consistent information across states and localities. Surveys can be flexible, so their developers can control what information is collected about income and employment, and this information can be improved over time Potential Weaknesses Three general concerns arise with the nationally representative surveys that keep them from being the solution, or “core” data, for understanding the effects of welfare reform. The most important issue is that sample sizes and sampling frames are such that these data cannot be used to examine certain subpopulations of interest, such as welfare recipients in a particular state (perhaps with the exception of the largest states, such as California, New York, and Texas). A distinguishing feature of welfare reform is that program responsibility now largely rests with states and even counties within a state. The nationally representative data sets do not have sample designs and sample sizes that allow analysts to examine behavior at a level that corresponds to where program decisions are being made. Second, there appear to be systematic changes in the coverage of low-in-come populations in the CPS. Studies have found that AFDC and Food Stamp Program benefits and the number of recipients in the CPS have declined over time relative to estimates of participants from administrative records. This issue of coverage is a serious concern for studies that use the CPS for measuring the income of welfare populations.8 In Table 9–2, we reproduce comparisons of aggregate AFDC/TANF and Food Stamp Benefits Program between CPS and administrative data sources from the Primus et al. (1999) study. It shows there has been a sharp decline between 1990 and 1997 in the percentage of AFDC/ TANF and Food Stamp Program benefits reported in the CPS compared to amounts reported in administrative data.9 The reduction in coverage of AFDC/ 8 Roemer (1999) suggests the reduction in coverage could be related to PRWORA—the March 1997 survey did not use state-specific labels for TANF benefits in 14 states that had abolished AFDC. Benefit estimates were 4.5 percentage points lower than the benchmark in states that had abolished AFDC than in states that had not. The delivery mechanism of benefits in some circumstances (for example, through employers), an enhanced sense of stigma, and caseload reductions that exacerbate recall errors may also contribute to underreporting. 9 Primus et al. adjust the CPS data proportionately to account for the decline in benefits over time, but the value of this adjustment depends on the patterns of discrepancies in the data. Unfortunately, we know little about the factors associated with the underrepresentation of program participants in the CPS or the SIPP.
OCR for page 283
Studies of Welfare Populations: Data Collection and Research Issues TABLE 9–2 AFDC/TANF and Food Stamp Aggregate Benefits Paid Based on Administrative Data Compared to Estimates from Current Population Survey (CPS) (in billions of dollars) AFDC/TANF Benefits Food Stamp Benefits CPS Data Administrative Data Ratio (%) CPS Data Administrative Data Ratio (%) 1990 14.259 18.855 75.6 10.335 13.556 76.2 1991 15.554 20.804 74.8 12.373 16.551 74.8 1992 15.362 22.258 69.0 13.394 20.014 66.9 1993 17.540 22.307 78.6 15.010 22.253 67.5 1994 17.145 22.753 75.4 15.317 22.701 67.5 1995 15.725 21.524 73.1 14.542 22.712 64.0 1996 13.494 19.710 68.5 14.195 22.440 63.3 1997 10.004 15.893 62.9 12.274 19.570 62.7 SOURCE: Primus et al. (1999:65), which in turn gives the sources, as Health and Human Services and U.S. Department of Agriculture administrative records and Center on Budget and Policy Priorities tabulations of CPS data. TANF (or family assistance) benefits also is consistent with Roemer’s (2000: Table 3b) calculations from the CPS for 1990 through 1996. Interestingly, the apparent decline in AFDC/TANF coverage does not show up in the SIPP, though the SIPP appears to capture only about three-quarters of aggregate benefits. Polivka (1998) compares the monthly average number of AFDC recipients in the March CPS to the monthly average reported to the Department of Health and Human Services (prior to quality control). She finds there has been a modest decrease in the proportion of total months on AFDC as measured in the CPS. The ratio of the CPS estimated to the administrative count (excluding Guam, the Virgin Islands, and Puerto Rico) is 83.0 (1989), 86.7 (1990), 86.0 (1991), 82.5 (1992), 84.2 (1993), 78.5 (1994), 75.5 (1995), and 79.6 (1996). The timing of the drop in the ratio corresponds to changes in the March CPS survey instrument. Taken together, the Primus et al. (1999) and Polivka (1998) results suggest that the decline in benefits reported in the CPS results from both a reduction in the coverage of families receiving AFDC and from an underrepresentation of benefits conditional on receipt, though the second factor seems quantitatively more important than the first. The third potential weakness of national surveys is that there is little or no “cost” to respondents of misreporting of income, employment, or other circumstances.10 10 Shroder and Martin (1996), for example, show subsidized housing (broadly defined) is badly reported on surveys, including the American Housing Survey (and presumably the SIPP). An underlying problem is that the phrase “public housing” means different things to different people, ranging from only projects to any kind of subsidized housing.
OCR for page 284
Studies of Welfare Populations: Data Collection and Research Issues Some specific potential weaknesses associated with the PSID and NLSY79 are of potential relevance for obtaining information on the income and employment status of low-income populations. Most notable is the fact that they are not, by design, representative of the general population over time. Both data sets began with samples that were representative of their targeted groups—young adults in the case of the NLSY79 and the national population as of 1968 in the case of the PSID—but are not designed to be representative of the national population, or even of the age group covered in the NLSY79, in subsequent years. This feature can result in biased measures of summary statistics on income and employment vis-à-vis the nation as a whole in more recent years. The other feature of the NLSY79 and PSID relevant for assessing the income and employment status of low-income populations is their respective sample sizes. The original sample for the NLSY79 was 12,686 young men and women, from which approximately 90 percent of the original sample remains today. The original sample in the PSID was 5,000 U.S. households in 1968 and, because of its growth through the accumulation of additional households through splitoffs from original households, it contained more than 8,700 in 1995. Although these are not small sample sizes, the sizes of low-income samples at a point in time are relatively small compared to both the CPS (which contains some 60,000 households at a point in time) and most waves of the SIPP (which, in its larger waves, contains data on 21,000 households). The sizes of the low-income or welfare subsamples in the NLSY79 and PSID for even the largest states are generally too small to derive reliable measures on income and employment, let alone other outcomes. To summarize, there are two primary potential weaknesses with using national survey data to measure income and employment of low-income populations. They are the following: Sample sizes in national surveys often are small for studies that focus on welfare or low-income populations, or that wish to examine specific targeted groups, such as current or former welfare recipients. There appears to be falling coverage (of both recipients and benefits) in national surveys. Direct Assessments of Income and Employment Measures from Survey Data Moore et al. (1997) conducted a general survey of income reporting in the CPS and SIPP, and Roemer (2000) assesses trends in SIPP and CPS income reporting between 1990 and 1996.11 A central finding in Moore et al. (1997) and 11 There are no comprehensive assessments of the quality of income and employment measurements for either the NLSY79 or the PSID. Roemer (1999) and Nelson et al. (1998) update the CPS
OCR for page 285
Studies of Welfare Populations: Data Collection and Research Issues Roemer (2000) is that there is underreporting of many types of income in surveys. The reasons for this and, hence, solutions in the design of effective surveys are complex. The magnitudes of CPS and SIPP underreporting for selected years are given in Tables 9–3a and 9–3b, taken from the two papers. (Note that differences may be the result of flawed benchmarks rather than flawed surveys.) Surveys of Income Reporting in the SIPP and CPS The understatement of certain types of income, such as interest and dividend receipts, is probably not critical for low-income populations because low-income families typically receive small amounts of income from these sources. Based on the evidence presented in Tables 9–3a and 9–3b, it appears that wages and salaries are fairly accurately reported in the CPS, although less accurately in the SIPP. But Moore et al. (1997) note that 26.2 percent (35,205,000 out of 134,135,000 total weighted cases) of the wage and salary “responses” in CPS surveys are imputed from cases where the respondent did not give an answer, replied “don’t know,” or refused to answer the question. They also report that 7 to 8 percent of households refuse to participate in the CPS, so imputations and imputation quality is clearly a critical element in survey quality. The apparent accuracy of wage and salary reporting in Tables 9–3a and 9–3b does not fully resolve concerns that we have about data accuracy for low-income populations, because we do not know much about the characteristics of families that underreport their incomes. If, for example, most of the underreporting of income occurs among the disadvantaged, the findings of Moore et al. (1997) and Roemer (2000) on wage and salary reporting in the CPS and SIPP may be of little comfort. Roemer, for example, shows there are significantly more aggregate dollars reported below family income of $25,000 in the SIPP relative to the March CPS. He suggests that the SIPP does a better job than the CPS of capturing the incomes of low earners and a worse job of capturing the incomes of high earners. Learning more about the nature of underreporting would appear to be a high priority for future research. Matching Studies of Wage and Salary Income Roemer (2000) examines the accuracy of CPS wage and salary reports by matching CPS data to Internal Revenue Service (IRS) tax returns in selected years for the first half of the 1990s. The sample is limited to nonjoint returns and selected joint returns where each filer matches a March CPS person. The sample is restricted further to observations with no imputed wages in the CPS. He finds that in the middle of the income distribution (from $15,000 to $150,000), at least calculations to 1996. Roemer (2000) also provides a nice discussion of adjustments that need to be made to compare aggregate SIPP and CPS totals to National Income and Product Account data.
OCR for page 305
Studies of Welfare Populations: Data Collection and Research Issues A fifth concern relates to the incidence and accuracy of tax filing by individuals and households, especially among low-income populations. This concern takes two forms: (1) whether people file any tax return, and (2) if they file, whether they report all sources of income to the IRS (or state taxing authorities).26 We consider each in turn. If large fractions of low-income taxpayers do not file tax returns, then tax return data have very limited value. Unfortunately, there is not a lot of information on the filing propensities of people with low income. Information from the early 1990s (Scholz, 1994) suggests that 14 to 20 percent of those entitled to the earned income tax credit at the time failed to receive it, meaning that they failed to file tax returns.27 Later, we discuss one recent study on the tax filing propensities of a low-income population that sheds some preliminary light on this issue. Among filing units, it is also possible that their members do not report all of their sources of income on their tax returns. For example, individuals may fail to file income received as independent contractors. Although firms or individuals who use independent contractors are obligated to report payments to such contractors to the IRS, failures to do this generally are difficult to detect. Again, we know little about the incidence of underreporting of various income sources for low-income populations. To summarize, using tax return data to measure income and employment has several potential weaknesses. These are the following: Gaining access to tax returns is difficult. The data provide limited information on demographic and other characteristics. Some low-income workers may not file, despite being eligible for the earned income tax credit, or may not report all their income. Comparison of Income Reporting from UI Wage and IRS Tax Filings Data for a Low-Income Population In a recent study of the EITC for a sample of assistance units on the California caseload, Hill et al. (1999) compared UI wage data with linked data from the 26 If one is just interested in enumerating the population (as opposed to knowing incomes associated with families and individuals within the population), IRS data appear to be comprehensive. Sailer and Weber (1999) report that the IRS population count is 95.4 percent of the Census population count. The consistency is fairly good across gender, age, and state. Unfortunately, for many of the people enumerated, the IRS does not know anything about them other than that they exist. 27 Cilke (1998) uses a CPS-IRS exact match file to examine the characteristics of people who are not required to file tax returns and actually did not file tax returns. The entire paper is presented as proportions, however, so it does not provide information on the absolute number of low-income families with earnings who fail to file.
OCR for page 306
Studies of Welfare Populations: Data Collection and Research Issues sample members’ IRS tax returns. The study used data from the California Work Pays Demonstration Project (CWPDP), which was conducted in four counties (Alameda, Los Angeles, San Bernardino, and San Joaquin) starting in 1992. The data consisted of two sets of assistance units drawn from the caseloads in these counties. One set, which is used for the sample in Table 9–8, consisted of a random sample drawn from a caseload at a particular date in 1992. Although this sample is representative of the caseload at that time, recall that the study by Bane and Ellwood (1983) showed that random samples from the existing caseload of AFDC are disproportionately made up of assistance units that are “welfare dependent.” The second set of assistance units, which is the sample used for Table 9–9, is a random sample of new entrants to the caseload in 1993. Bane and Ellwood (1983) and others have found that a significant proportion of new entrants remain on welfare for only a relatively short period.28 Furthermore, Gritz and MaCurdy (1991) find that most new entrants exit from AFDC to employment. We also break both samples up into female-headed households (Aid to Families with Dependent Children-Family Group AFDC-FG cases) and two-parent households (AFDC-U). We report on annual earnings information for the year after the samples were drawn, that is, 1993 for the random sample of the caseload and 1994 for the new entrants sample.29 The first two lines of each panel of each table give estimates of the employment rates of each sample of AFDC recipients. As expected, employment rates of the point-in-time caseload (Table 9–8) are lower than the sample of new entrants (Table 9–9). Employment rates of one-parent cases (AFDC-FG) are lower than the employment rates of two-parent cases (Aid to Families with Dependent Children-Unemployed Parent [AFDC-U]). What is striking and not necessarily expected, however, is that the implied employment rates using UI data and using tax return data are nearly identical. From Table 9–8, employment rates of the point-in-time AFDC-FG caseload were 26 percent using UI data and 22 percent using tax return data. The corresponding rates for AFDC-U cases were 31 percent for both data sources. Employment rates were 37 percent using UI data for the new entrant sample and 33 percent using tax returns. Employment rates were 48 percent using UI data for the AFDC-U new entrants and 49 percent using tax returns. 28 For example, Bane and Ellwood (1983) estimate that 65 percent of new entrants leave the caseload in 2 years. 29 Through an interagency agreement between the California Department of Social Services (CDSS) and the state’s taxing authority, the Franchise Tax Board (FTB), UI wages and wages and adjusted gross income (AGI) from tax returns were merged by the FTB. The researchers were able to specify computer runs on these merged files. Assistance units in the study could, and did, leave AFDC after they were enrolled in this study. Nonetheless, wage and income data from UI records and tax returns were available for all of the original assistance units in the CWPDP study.
OCR for page 307
Studies of Welfare Populations: Data Collection and Research Issues TABLE 9–8 Random Sample from Caseload in 1992, Information for Tax Year 1993 Those Filing Tax Returns Full Sample AFDC-FG cases % of households with UI earnings 26 % of households that filed tax returns 22 Average UI earnings ($) of adults in household 4,514 1,242 Average adjusted gross earnings ($) on tax returns 10,589 2,378 Average wage & salary earnings ($) on tax returns (Line 7) 9,748 2,189 Average income ($) reported to AFDC 1,222 360 % of households with No UI earnings, but filed tax return 5.89 % of households with UI earnings, but filed no tax return 11.41 % of households for which AGI<UI wages 12.61 % of households for which AGI=UI wages 78.59 % of households for which AGI>UI wages 8.80 % of households for which AGI<UI wages, for UI wages > 0 3.39 % of households for which AGI>UI wages, for AGI > 0 40.47 Self-employment income ($) reported on tax returns Fraction of filers reporting any 0.06 Average amount reported 357 AFDC-U Cases % of households with UI earnings 31 % of households that filed tax returns 31 Average UI earnings ($) of adults in household 5,223 1,792 Average adjusted gross earnings ($) on tax returns 8,482 2,595 Average wage & salary earnings ($) on tax returns (line 7) 7,554 2,311 Average income ($) reported to AFDC 2,513 894 % of households with no UI earnings, but filed tax return 7.07 % of households with UI earnings, but filed no tax return 8.21 % of households for which AGI<UI wages 9.26 % of households for which AGI=UI wages 78.39 % of households for which AGI>UI wages 12.05 % of households for which AGI<UI Wages, for UI Wages > 0 3.97 % of households for which AGI>UI wages, for AGI > 0 40.53 Self-employment income ($) reported on tax returns Fraction of filers reporting any 0.12 Average amount reported 562 SOURCE: Hill et al. (1999). Although tax return data and UI data would give similar perspectives about employment patterns of the 4-county California sample, it is clear that each sample covers workers that the other misses. For example, in the top panel of Table 9–8 (AFDC-FG cases from the point-in-time sample), roughly one-quarter
OCR for page 308
Studies of Welfare Populations: Data Collection and Research Issues TABLE 9–9 Random Sample of New Entrants to AFDC Caseload in 1993, Information for Tax Year 1994 Those Filing Tax Returns Full Sample AFDC-FG cases % of households with UI earnings 37 % of households that filed tax returns 33 Average UI earnings ($) of adults in household 6,769 2,868 Average adjusted gross earnings ($) on tax returns 13,185 4,342 Average wage & salary earnings ($) on tax returns (Line 7) 12,575 4,141 Average income ($) reported to AFDC 1,625 709 % of households with no UI earnings, but filed tax return 8.34 % of households with UI earnings, but filed no tax return 13.55 % of households for which AGI<UI wages 15.69 % of households for which AGI=UI wages 71.31 % of households for which AGI>UI wages 13.00 % of households for which AGI<UI wages, for UI wages>0 4.21 % of households for which AGI>UI wages, for AGI>0 39.88 Self-employment income ($) reported on tax returns Fraction of filers reporting any 0.04 Average amount reported 95 AFDC-U cases % of households with UI earnings 48 % of households that filed tax returns 49 Average UI earnings ($) of adults in household 8,516 5,138 Average adjusted gross earnings ($) on tax returns 12,970 6,360 Average wage & salary earnings ($) on tax returns (Line 7) 11,421 5,601 Average income ($) reported to AFDC 3,264 1,831 % of households with no UI earnings, but filed tax return 10.45 % of households with UI earnings, but filed no tax return 7.94 % of households for which AGI<UI wages 11.77 % of households for which AGI=UI wages 64.71 % of households for which AGI>UI wages 23.51 % of households for which AGI<UI wages, for UI Wages>0 6.12 % of households for which AGI>UI wages, for AGI>0 46.83 Self-employment income ($) reported on tax returns Fraction of filers reporting any 0.11 Average amount reported 512 SOURCE: Hill et al. (1999). of people (5.89/22) who filed tax returns had no corresponding UI record.30 Over 40 percent (11.41/26) of those with positive UI earnings did not file taxes.31 Of 30 We took great care in the analysis to make sure the comparison samples did not have changes in marital status and had a full four quarters of UI data (including zero quarters). 31 Households with low earnings are not obligated to file tax returns. For example, a married
OCR for page 309
Studies of Welfare Populations: Data Collection and Research Issues those with both UI and tax return earnings, more than 40 percent reported more earnings on tax returns than would be expected based on UI data. Similar figures apply to each other group, though for AFDC-U cases, only about 20 percent of the cases with UI earnings do not file tax returns. The fact that across all four groups (two samples, and AFDC-FG and AFDCU cases), tax return income exceeded UI income in at least 40 percent of the cases with positive earnings from both sources, is consistent with households from this welfare-based population having earnings that are not from covered employment. The fact does not seem to be explained by people leaving welfare (through changes in family structure). Among AFDC-FG cases, only 1 to 13 percent of these households had no months on AFDC during the tax reference year and between 56 and 83 percent were on welfare for 9 to 12 months during that year. There is also little evidence that self-employment income plays an important role in earnings differences between tax return and UI income. Based on comparisons between UI and tax return data, we offer several tentative conclusions: Tax return and UI data appear to give very similar information on employment rates of the four-county California caseload. There are good reasons, however, to think that both data sources will lead to underestimates. UI data will miss independent contractors and possibly other “flexible workers.” Tax return data will miss families who do not file tax returns. The two data sources appear highly complementary. Each appears to capture a significant number of families that the other misses. Using them together, therefore, should result in more accurate measures of the employment experiences of the caseload than using either separately. Tax return data have a broader definition of income and, if the household unit is married, will cover both spouses and hence are likely to offer more accurate income information. UI data are much easier to access than tax returns. RECOMMENDATIONS Taking into account all of the features of a data source, including not only its accuracy but also its cost and ease of access, it appears that no single source can be declared “preferred.” The inability to find a preferred data source is inevitable given the differences in the desired uses of data, the constraints imposed by budgets for data collection, and the access limitations to data. The fact that UI couple, is not required to file if their income is below the standard deduction and two exemptions ($12,200 in 1997), regardless of how many children they have. Hill et al. (1999) also show that most of these non filers had very low levels of UI earnings ($2,500 or less in annual covered earnings).
OCR for page 310
Studies of Welfare Populations: Data Collection and Research Issues wage data are inexpensive, timely to obtain, and available at the state level, for example, implies that they will continue to be a focal data set for state-level evaluations of welfare reform. But our review raises a number of serious questions about UI data. In the remainder of this paper, we highlight selected issues that we believe need further attention in the hopes of encouraging future research on at least some of them. Certain questions related to welfare reform can only be answered with nationally representative data sets, such as the CPS or SIPP. While Moore et al. (1990) and Roemer (1999a) conclude that income, especially labor earnings, are measured well in the CPS and SIPP, there are, in our view, several important questions that remain with respect to income and employment measurements for low-income populations with national surveys. The questions are as follows: First, none of these studies, to our knowledge, focus on the reporting of income by disadvantaged, welfare-eligible, and/or welfare-prone populations. Second, as noted in Primus et al. (1999), participation in welfare programs is underreported in the CPS (and the SIPP). Moreover, this underreporting appears to have increased over time. This is a troubling problem, especially as one looks to the future when TANF programs become state specific, with different names. Recommendation 1: We would like to see further work on the sources of anti-poverty program underreporting and its origins in nationally representative survey data. Plans are under way for some of the needed work. Professor Hotz is a principal investigator on a project recently approved by the U.S. Census Bureau to match data from UI wage records and administrative data on AFDC/TANF participation for the California subsamples of several waves of the SIPP.32 The work of this project should yield some more recent information on both the welfare participation underreporting and income reporting issues. This study—or comparable ones done with matches of the SIPP with administrative data for the subsamples from other states—also may provide some insight into the impact of changes in family structure on income reporting for welfare leavers by exploiting the (limited) panel structure of the SIPP. Further research also is needed on the use of UI wage records to measure the income of low-income and welfare-prone populations. While the Kornfeld and Bloom (1999) evaluation suggested that UI wage data and survey data produced similar estimates of the impact of a social program (i.e., JTPA-funded training programs) on earnings and employment, their study also found that average earnings of JTPA-eligible individuals were consistently lower than those based 32 The other investigators on this project are in collaboration with David Card, Andrew Hildreth, and Michael Clune at University of California-Berkeley and Robert Schoeni at RAND.
OCR for page 311
Studies of Welfare Populations: Data Collection and Research Issues on survey data. Furthermore, the study by Hill et al. (1999) also found that UI wage data produced substantially lower estimates of earnings than did tax returns data for a welfare-based population drawn from the California AFDC caseload. Learning more about the quality of this data source for measuring income is extremely important because UI wage data presumably will continue to be a core resource in state and local evaluations of the effects of welfare reform. Several issues related to UI wage data appear to need further scrutiny. First, the studies by Burgess and his coauthors raises important concerns about the “coverage” of UI and tax returns, particularly for the low-income population. Recommendation 2: It would be extremely useful to follow the helpful lead of the various Burgess studies to closely examine the coverage and trends in coverage of low-income populations with UI data. Such an examination could be aided by using a match of UI data with respondents in a national survey, such as the SIPP, so that one could learn more about the demographic characteristics of individuals (and households) that report labor market earnings on a survey that are not recorded in UI wage records data. States may be able to augment UI data used for evaluation of welfare reform by collecting supplemental information on the degree to which employers are designating workers as independent contractors. Additional work at the state level to assess the overall coverage of UI data also would be valuable. Second, more work is needed to understand the extent to which UI wage data provide a misleading measure of the earnings available to low-income households. This problem arises in short- and long-term follow-up analyses of earnings for welfare samples drawn from state caseloads. One can use UI data to measure subsequent earnings for individuals who were in assistance units as long as they remain on welfare. However, as noted by Rolston (1999), one may not be able to accurately measure household income after assistance units leave the rolls because it is difficult to keep track of the identities of household members. The evidence provided in the Meyer and Cancian (1998) and Hill et al. (1999) studies suggest that this may be a serious problem. Recommendation 3: To learn more about family well-being, it will be necessary to continue to rely on targeted follow-up surveys to monitor samples of welfare leavers. Unfortunately surveys are expensive. We recommend that a pilot study be undertaken to devise a survey that is designed just to obtain Social Security numbers of other adults in a household, which can then be used to obtain UI wage earnings for these family members. It might be useful for state TANF agencies to analyze the methods that their JTPA agencies use to gather follow-up earnings data on terminees from
OCR for page 312
Studies of Welfare Populations: Data Collection and Research Issues their programs. Such follow-up assessments are required under JTPA, and many states have contracted with firms and/or universities to gather these follow-up data. Tax returns data also may be useful to learn more about whether the discrepancies between UI wage data and income measures from tax returns noted in that study are the result of differences in family composition and the “composition” of income reported on tax returns. A third issue relates to the possibility that wage earnings are missed because individuals move out of the state from which UI wage data are drawn or because workers earn part of their income in other states. Again, comparisons of UI wage data with data from federal tax returns may help us to assess the importance of this problem and, more importantly, the biases that it imparts on measures of individual and household income. To learn more, it may be useful to take a closer look at what is known about the interstate mobility of disadvantaged and welfare-prone populations, such as the work done on movements of welfare populations in response to “welfare magnets,” as in Meyer (1999) and the citations therein, and the implications this mobility has for the coverage of low-income workers in UI data. REFERENCES Abraham, Katherine G., James R.Spletzer, and Jay C.Stuart 1998 Divergent trends in alternative wage series. Pp. 293–324 in Labor Statistics Measurement Issues, J.Haltiwanger, M.Manser, and R.Topel, eds. National Bureau of Economic Research Studies in Income and Wealth, Volume 60. University of Chicago Press. Baj, John, Sean Fahey, and Charles E.Trott 1992 Using unemployment insurance wage-record data for JTPA performance management. In Chapter 4 of Research Report 91–07. Chicago: National Commission for Employment Policy. Baj, John, Charles E.Trott, and David Stevens 1991 A Feasibility Study of the Use of Unemployment Insurance Wage-Record Data as an Evaluation Tool for JTPA: Report on Project Phase 1 Activities. Chicago: National Commission on Employment Policy, January. Bane, Mary Jo, and David T.Ellwood 1983 The Dynamics of Dependence: The Routes to Self-Sufficiency. Prepared for the U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation. Cambridge, MA: Urban Systems Research and Engineering, Inc. Blakemore, Arthur E., Paul L.Burgess, Stuart A.Low, and Robert D. St. Louis 1996 Employer tax evasion in the unemployment insurance program. Journal of Labor Economics 14(2):210–230. Bogen, Karen, Meredith Lee, Julia Klein Griffiths, and Anne Polivka 1997 Income Supplement—Summary and Recommendations from Cognitive Interviews, Unpublished paper, Bureau of the Census, Bureau of Labor Statistics, September.
OCR for page 313
Studies of Welfare Populations: Data Collection and Research Issues Bogen, Karen 1998 Once Upon a Time, There Was Welfare Reform: Evaluating the New March CPS Welfare-Related Questions: Results from the 1998 Respondent Debriefing. Unpublished paper, Bureau of the Census, June. Bollinger, Christopher R., and Martin H.David 2001 Estimation with response error and non-response: Food stamp participation in the SIPP. Journal of Business and Economic Statistics 19(a):129–141. Bound, John, and Alan B.Krueger 1991 The extent of measurement error in longitudinal earnings data: Do two wrongs make a right? Journal of Labor Economics 9(1):1–24. Burgess, Paul L., Arthur E.Blakemore, and Stuart A.Low 1996 Using statistical profiles to improve unemployment insurance tax compliance. Research in Employment Policy 1:243–2711. Cancian, Maria, Robert Haveman, Thomas Kaplan, and Barbara Wolfe 1999 Post-Exit Earnings and Benefit Receipt Among Those Who Left AFDC in Wisconsin. Institute for Research on Poverty, Special Report No. 75. Madison, WI: University of Wisconsin. Cilke, Jim 1998 A Profile of Non-Filers. OTA Paper #78, Office of Tax Analysis, U.S. Department of Treasury, Washington, DC. Coder, J., and L.S.Scoon-Rogers 1996 Evaluating the Quality Income Data Collection in the Annual Supplement to the March Current Population Survey and the Survey of Income and Program Participation. SIPP Working Paper 96–04. Coder, John 1992 Using administrative record information to evaluate the quality of the income data collected in the SIPP. Pp. 295–306 in Proceedings of Statistics Canada Symposium 92: Design and Analysis of Longitudinal Surveys, Ottawa: Statistics Canada. Goodreau, K., H.Oberheu, and D.Vaughan 1984 An assessment of the quality of survey reports of income from the Aid to Families with Dependent Children (AFDC) program. Journal of Business and Economic Statistics 2:179– 186. Greenberg, David, and Harlan Halsey 1983 Systematic misreporting and the effects of income maintenance experiments on work effort: Evidence from the Seattle-Denver experiment. Journal of Labor Economics 1:380– 407. Gritz, R.M, and T.MaCurdy 1991 Patterns of Welfare Utilization and Multiple Program Participation Among Young Women. Report to the U.S. Department of Health and Human Services under Grant 88-ASPE 198A. Halsey, Harlan 1978 Validating income data: Lessons from the Seattle and Denver income maintenance experiment, pp. 21–51 in Proceedings of the Survey of Income and Program Participation Workshop, U.S. Department of Health, Education and Welfare, Washington, DC. Hill, Carolyn, V.J.Hotz, Charles Mullin, and John Karl Scholz 1999 EITC Eligibility, Participation and Compliance Rates for AFDC Households: Evidence from the California Caseload. Report submitted to the California Department of Social Services, April.
OCR for page 314
Studies of Welfare Populations: Data Collection and Research Issues Houseman, Susan N. 1999 Flexible Staffing Arrangements: A Report on Temporary Help, On-Call, Direct-Hire Temporary, Leased, Contract Company, and Independent Contractors Employment in the United States, August. Available: http://www.dol.gov/asp/futurework/conference/staffing/intro.htm [September 7, 2001] Internal Revenue Service 1996 Federal Tax Compliance Research: Individual Income Tax Gap Estimates for 1985, 1988, and 1992. Publication 1415 (Rev. 4–96). Washington, DC. Kornfeld, Robert, and Howard S.Bloom 1999 Measuring program impacts on earnings and employment: Do unemployment insurance wage reports from employers agree with surveys of individuals? Journal of Labor Economics 17(January):168–197. Lamas, E., T.Palumbo, and J.Eargle 1996 The Effect of the SIPP Redesign on Employment and Earnings Data. SIPP Working Paper 9606. Lamas, E., J.Tin, and J.Eargle 1994 The Effect of Attrition on Income and Poverty Estimates from the Survey of Income and Program Participation (SIPP). SIPP Working Paper 190. Marquis, K.H., and C.J.Moore 1990 Measurement errors in SIPP program reports. Pp. 721–745 in Proceedings of the Bureau of the Census 1990 Annual Research Conference. Meyer, Daniel R., and Maria Cancian 1998 Economic well-being following an exit from Aid to Families with Dependent Children. Journal of Marriage and the Family 60(2):479–492. Meyer, Bruce D. 1999 Do the Poor Move to Receive Higher Welfare Benefits? Unpublished paper, Northwestern University Economics Department, October. Moore, J., K.Marquis, and K.Bogen 1996 The SIPP Cognitive Research Evaluation Experiment: Basic Results and Documentation. Bureau of the Census, January. Moore, Jeffrey C., Linda L.Stinson, and Edward J.Welniak, Jr. 1997 Income Measurement Error in Surveys: A Review. Statistical Research Report. U.S. Census Bureau. Nelson, Charles T., Marc I.Roemer, Daniel H.Weinberg, and Edward J.Welniak, Jr. 1998 Fifty Years of United States Income Data from the Current Population Survey. Unpublished paper, Housing and Household Economics Statistics Division, Bureau of the Census. December. Polivka, Anne P. 1997 Using Earnings Data from the Current Population Survey After the Redesign. Unpublished paper. Bureau of Labor Statistics. 1998 Note on the Possible Effects of Welfare Reform on Labor Market Activities: What Can Be Gleaned from the March CPS. Unpublished paper, Bureau of Labor Statistics. December 1. Primus, Wendell, Lynette Rawlings, Kathy Larin, and Kathryn Porter 1998 After Welfare: A Study of Work and Benefit Use After Case Closing in New York State. Revised interim report submitted to the Office of the Assistant Secretary for Planning and Evaluation, U.S. Department of Health and Human Services, December. 1999 The Initial Impacts of Welfare Reform on the Incomes of Single-Mother Families. Washington, DC: Center for Budget and Policy Priorities. Rockefeller Institute of Government.
OCR for page 315
Studies of Welfare Populations: Data Collection and Research Issues Rodgers, Willard L., Charles Brown, and Greg J.Duncan 1993 Errors in survey reports of earnings, hours worked, and hourly wages. Journal of American Statistical Association 88(December):1208–1218. Roemer, Marc 1999 Assessing the Quality of the March Current Population Survey and the Survey of Income and Program Participation Income Estimates, 1990–1996. Unpublished paper, Income Statistics Branch, Bureau of the Census, June 16. 2000 Reconciling March CPS Money Income with the National Income and Product Accounts: An Evaluation of CPS Quality. Unpublished paper, Income Statistics Branch, Bureau of the Census, August 10. Rolston, Howard 1999 The Income of Former Welfare Recipients. Unpublished paper, Administration on Children and Families, U.S. Department of Health and Human Services, September 21. Sailer, Peter, and Michael Weber 1999 The IRS population count: An Update. Pp. 85–89, In Turning Administrative Systems into Information Systems. Scholz, John Karl 1994 The earned income tax credit: Participation, compliance and anti-poverty effectiveness. National Tax Journal (March):59–81. Shroder, Mark, and Marge Martin 1996 New Results from Administrative Data: Housing the Poor, or, What They Don’t Know Might Hurt Somebody. Unpublished paper, Office of Policy Development and Research, U.S. Department of Housing and Urban Development. Smith, Jeffrey 1997 Measuring Earnings Levels Among the Poor: Evidence from Two Samples of JTPA Eligibles. Unpublished paper, Department of Economics, University of Western Ontario, June. Stevens, David W., Liping Chen, and Jinping Shi 1994 The Use of UI Wage Records for JTPA Performance Management in Maryland. Unpublished paper, The Jacob France Center at the University of Baltimore, September 19. U.S. Department of Commerce, Bureau of the Census 1998 SIPP Quality Profile, 1998. SIPP Working Paper Number 230, Third Edition. Wisconsin Department of Workforce Development 1999 Differences Between AFDC and W-W Leavers Survey Data for January–March 1998 and Wisconsin’s UI Wage Records for 1998. DWD MEP Folio Brief 01–99, October 19. Yen, W., and H.Nelson 1996 Testing the Validity of Public Assistance Surveys with Administrative Records: A Validation Study of Welfare Survey Data, unpublished manuscript, May 1996.
Representative terms from entire chapter: