National Academies Press: OpenBook

Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers (1991)

Chapter: EVALUATION OF ESTIMATES OF PROGRAM PARTICIPANTS FROM CPS AND SIPP

« Previous: Other Variables for Related Policy Issues
Suggested Citation:"EVALUATION OF ESTIMATES OF PROGRAM PARTICIPANTS FROM CPS AND SIPP." National Research Council. 1991. Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers. Washington, DC: The National Academies Press. doi: 10.17226/1853.
×
Page 46
Suggested Citation:"EVALUATION OF ESTIMATES OF PROGRAM PARTICIPANTS FROM CPS AND SIPP." National Research Council. 1991. Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers. Washington, DC: The National Academies Press. doi: 10.17226/1853.
×
Page 47
Suggested Citation:"EVALUATION OF ESTIMATES OF PROGRAM PARTICIPANTS FROM CPS AND SIPP." National Research Council. 1991. Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers. Washington, DC: The National Academies Press. doi: 10.17226/1853.
×
Page 48
Suggested Citation:"EVALUATION OF ESTIMATES OF PROGRAM PARTICIPANTS FROM CPS AND SIPP." National Research Council. 1991. Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers. Washington, DC: The National Academies Press. doi: 10.17226/1853.
×
Page 49

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

DATABASES FOR MICROSIMULATION: A COMPARISON OF THE MARCH CPS AND SIPP 46 enforcement, child care tax credits, and programs to encourage employment. Neither SIPP nor CPS is designed specifically to address such initiatives, but, again, SIPP has the advantage in terms of the scope of the data that are included. For example, SIPP has regularly asked about participation in job programs and has included a module on day care needs and expenses in every panel. Many of these new policy interests involve a longitudinal component, and there has thus been growing interest in studying the dynamics of program participation on a subannual basis. Longitudinal surveys such as the PSID have supported extensive analysis of movement in and out of poverty and reliance on income support programs on a year-to-year basis over long spans of time. However, until the ISDP and SIPP, there has been no comparable database for analysis of short-term program dynamics. Yet such analysis is important to understand turnover and recidivism in caseloads, to determine whether multiple program participation tends to be concurrent or serial in nature, and to assess the effectiveness of program innovations such as transitional Medicaid and child care support in moving families from welfare to the work force. Obviously, the CPS cannot contribute to these types of analyses; the SIPP can. EVALUATION OF ESTIMATES OF PROGRAM PARTICIPANTS FROM CPS AND SIPP Up to this point, this chapter has addressed problems for modeling and analysis of income support programs from errors associated with specific aspects of the design and operation of the March CPS and SIPP surveys. This section adopts a more global perspective, reviewing estimates from these two surveys of characteristics of AFDC and food stamp recipients compared with data from administrative records. Although microsimulation models simulate program participation from among a pool of simulated eligible units, and some models ignore reported recipiency of benefits altogether in selecting participants, the extent to which the input data accurately portray the recipient population has obvious implications for the quality of the simulations. Most of the available comparisons evaluate one or another but not both surveys against administrative records. (The aggregate comparisons, reviewed above, that found more complete reporting of total recipients and benefits in SIPP versus CPS are an exception.) Hence, it is not possible to make definitive assessments of the relative merits of the two surveys in characterizing the participant population. In addition, although administrative data are used as the target in comparisons with survey estimates, observed differences do not always indicate errors in the survey data. The differences ma y be due to errors in the administrative records or to conceptual distinctions between the two sources. Allin and Doyle (1990) compared SIPP estimates of reported food stamp participants (including those simulated to be eligible and not eligible for the program) with data from the Integrated Quality Control System (IQCS), which

DATABASES FOR MICROSIMULATION: A COMPARISON OF THE MARCH CPS AND SIPP 47 comprises monthly samples of case records for the food stamp, AFDC, and Medicaid programs. Their comparisons were limited to cases with incomes at or below 250 percent of the poverty threshold for the month of August 1984: for that period, the SIPP sample comprised 1,272 and the IQCS sample 6,932 unweighted food stamp recipient units. Allin and Doyle found significant differences between the SIPP and the IQCS on several dimensions. SIPP estimated higher proportions than the IQCS of food stamp units containing disabled members (10 versus 7%); headed by a male (36 versus 30%); receiving earned income (27 versus 18%); with total income above the poverty level (17 versus 7%); and with zero countable assets (84 versus 77%) or with countable assets over $3,000 (2 versus 0%). SIPP estimated lower proportions than the IQCS of food stamp units comprising one person only (26 versus 32%); containing no school-age children (48 versus 53%); and having medical expense deductions (1 versus 3%). The estimated average child care deduction in SIPP was only 72 percent of that in the IQCS, and the average estimated medical expense deduction in SIPP was only 41 percent of the average in the IQCS. Finally, the SIPP food stamp participant population included 11 percent who appeared to be ineligible for the program on the basis of their assets or income levels. The authors suggest and, in some cases, are able to provide supporting evidence for the various factors that may account for the differences they find. For example, the Census Bureau's imputation procedures very likely account for over one-fourth of the seemingly ineligible participant cases. However, more work is clearly needed to understand their results. Unpublished tabulations of average monthly reported AFDC participants from the March 1984 and 1988 CPS, which the Urban Institute prepared for the panel, provide a basis for evaluating the CPS portrayal of the AFDC population in comparison with data from the 1983 and 1987 IQCS (representing averages of 12 monthly samples for each year). The following are unweighted sample sizes for average monthly AFDC cases:16 16 In total, there were 22,621 AFDC unit-months from the March 1984 CPS versus 72,940 unit-months from the 1983 IQCS, and 21,454 AFDC unit-months from the March 1988 CPS versus 65,570 unit-months from the 1987 IQCS. The effective sample sizes are somewhere between the smaller numbers cited in the text and the larger numbers just cited, because many unit-months represent the same cases. The author constructed tabulations for the total population of units reporting AFDC participation in the CPS from separate tabulations prepared by the Urban Institute for units simulated to be eligible for and reporting participation in AFDC and units simulated to be ineligible for but reporting participation in AFDC.

DATABASES FOR MICROSIMULATION: A COMPARISON OF THE MARCH CPS AND SIPP 48 1983 1987 CPS 1,885 1,788 IQCS 6,078 50,4065 Again, there are important differences between the CPS and the administrative data, similar to those found for the food stamp recipient population in the comparisons involving SIPP. Of AFDC units in 1983, the CPS estimated higher proportions than the IQCS for units comprising four or more persons (33 versus 28%); including three or more children (29 versus 25%); headed by someone not the head (or spouse) of the entire household (19 versus 4%);17 with positive gross income (33 versus 10%); and with earnings (17 versus 5%). Of AFDC units in 1987, the CPS estimated higher proportions than the IQCS for units comprising four or more persons (34 versus 28%); including three or more children (32 versus 26%); headed by someone not the head (or spouse) of the entire household (18 versus 3%); with positive gross income (37 versus 12%); and with earnings (20 versus 7%). In addition, for 1987 the CPS estimated a lower proportion than the IQCS of AFDC units headed by a married person (11 versus 13%). Finally, the AFDC participant population estimated by the CPS for both 1983 and 1987 included 14 percent who were simulated to be ineligible for the program. The differences noted above for the recipient population, such as the much higher proportions of AFDC and food stamp recipient units with earnings in CPS and SIPP in comparison with the IQCS, extend as well to the eligible population. That is, the simulated pool of eligible units typically contains far more cases of program units with earnings and other characteristics that are less typical of the caseload as portrayed in the administrative data and fewer cases with characteristics that resemble the caseload. In turn, this pattern has made it difficult for the microsimulation models to calibrate the simulated participant population to match administrative controls. For example, the TRIM2 1983 baseline simulation, after the calibration process had been completed, estimated 17 This large difference between the CPS and IQCS (observed also for 1987) suggests that caseworkers may not accurately categorize household composition for the IQCS (on the assumption that a sizable proportion of AFDC units are indeed subfamilies within larger households).

DATABASES FOR MICROSIMULATION: A COMPARISON OF THE MARCH CPS AND SIPP 49 8 percent of the simulated AFDC caseload with earnings, compared with 5 percent in the IQCS. The TRIM2 1987 baseline simulation, after calibration, estimated fully 13 percent of the simulated caseload with earnings, compared with only 7 percent in the IQCS.18 Similar problems have characterized food stamp simulations. Runs of the MATH model with a March 1981 CPS file projected to known population controls for August 1984 produced fewer cases of eligible units in single- parent families than there were such cases reported as beneficiaries in the IQCS. Runs of a model with 1984 SIPP data produced no better results (Doyle and Trippe, 1989: Table 7). Allin and Doyle (1990) reported better results for SIPP with a revised definition of household type for simulated food stamp units. However, the revised definition represented an arbitrary choice, and it is not clear whether the problem of too few cases of certain kinds of eligible units stems from population undercoverage, poor definition of the recipient unit within the household, or other factors.19 The models calibrate their baseline simulations of program participants in order to provide credibility and create what is assumed to be an improved database for simulations of alternative programs. However, the calibration almost always produces participation rates that are biased to a greater or lesser degree and that may bias the results for simulations of alternative programs, if the assumption that the biases are consistent across scenarios does not hold.20 Clearly, more work is needed to understand the reasons for the differences between survey and administrative record estimates of program caseload characteristics. 18 The March 1988 CPS used for the 1987 baseline had an unusually large number of states with fewer simulated eligible units than reported participants, making it more than usually difficult to match the administrative controls for cases with earnings and at the same time match controls for total AFDC recipients by state; see Giannarelli (1990). 19 One possible explanation, noted above, concerns imputation of variables such as income and assets. To the extent that a proportion of seemingly ineligible participant units are so categorized because of erroneous imputations, adopting a more appropriate imputation procedure could expand the eligible pool, make the calibration process easier, and very likely improve the quality of the simulations. In this regard, it should be noted that the AFDC-ineligible participants simulated by TRIM2 for 1983 and 1987 are similar to actual participants (from the IQCS) on such characteristics as proportion of units headed by a single woman or by a married person. The most pronounced differences relate to income: the ineligible participants have much higher proportions with positive income and earnings than eligible participants from the CPS; the latter group in turn are better off than the actual beneficiaries from the IQCS. (Of course, failure to report income to caseworkers may result in the IQCS underestimating participants' economic status.) 20 The calibrated participation rates are often, although not always, used for alternative program simulations.

Next: THE DIVISION OF LABOR FOR PRODUCTION OF DATABASES FROM CPS AND SIPP »
Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers Get This Book
×
Buy Paperback | $100.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

This volume, second in the series, provides essential background material for policy analysts, researchers, statisticians, and others interested in the application of microsimulation techniques to develop estimates of the costs and population impacts of proposed changes in government policies ranging from welfare to retirement income to health care to taxes.

The material spans data inputs to models, design and computer implementation of models, validation of model outputs, and model documentation.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!