Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
ALTERNATIVE MODEL DESIGNS: PROGRAM PARTICIPATION FUNCTIONS AND THE ALLOCATION OF ANNUAL TO 113 MONTHLY VALUES IN TRIM2, MATH, AND HITSM virtually no studies that attempt to validate participation functions externally through comparing simulations of program changes with actual outcomes14 or through ex post forecasting of current law with an earlier database, as was done in the panel's validation experiment. The panel considered including alternative participation functions in its experiment, but for several reasons did not pursue this idea. In the course of conducting its experiment, the panel did, however, learn a good deal about the current simulation and calibration of AFDC participation in TRIM2 (see the appendix to this chapter). The problems that came to light certainly support the need to perform validation studies of the participation function. They also support the need to conduct in-depth investigations of the quality of the March CPS and the administrative program data. APPENDIX: PERFORMANCE OF THE AFDC CALIBRATION PROCESS IN TRIM2 Two baseline files examined in the course of the validation experiment conducted by the Panel to Evaluate Microsimulation Models for Social Welfare Programs (see Cohen et al., Chapter 8, in this volume) provide information about the performance of the TRIM2 simulation of participation in the AFDC program and calibration to administrative targets. These files were a 1983 baseline produced from the March 1984 CPS and a 1987 baseline produced from the March 1988 CPS. With regard to the variables used as controls, application of the TRIM2 AFDC participation and calibration functions produced good results for national and state estimates of participating units, particularly in 1983. However, the results were less satisfactory for estimates of the percentage of the national caseload with earnings. Specifically, ⢠Total simulated participating units were 99.4 and 99.9 percent, respectively, of the administrative control numbers for 1987 and 1983. ⢠In 1987 simulated participants were within plus or minus 1.9 percent of the administrative number in 24 states, including 6 of the 10 largest states, and within plus or minus 3.9 percent in 41 states, including 8 of the 10 largest; in only 2 states did the simulated number differ from the target by 10 percent or more. ⢠In 1983 simulated participants were within plus or minus 1.9 percent of the administrative number in 34 states, including all of the 10 largest, and within plus or minus 3.9 percent in 45 states; in no state did the simulated number differ from the target by 8 percent or more. 14 The only such study known to us is that of Beebout and Haworth (1989), which examined the accuracy of the MATH forecasts of participation under the 1977 Food Stamp Reform Act (see also Cohen, Chapter 7, in this volume).
ALTERNATIVE MODEL DESIGNS: PROGRAM PARTICIPATION FUNCTIONS AND THE ALLOCATION OF ANNUAL TO 114 MONTHLY VALUES IN TRIM2, MATH, AND HITSM ⢠However, simulated participants included 12.7 and 7.8 percent, respectively, with earnings in 1987 and 1983 compared with 7.7 and 5.4 percent in the administrative data for those 2 years (expressed another way, simulated participants with earnings were 164 and 145 percent, respectively, of actual participants with earnings in 1987 and 1983). The explanation for these results has to do with discrepancies between the simulated data on eligible units and the administrative data on recipients. First, the simulated eligible population from the March CPS invariably contains a higher proportion with earnings than are shown in the administrative dataâ15 percent versus 8 percent in 1987 and 13 percent versus 5 percent in 1983. These differences ma y stem from errors in the March CPS (see Citro in this volume), such as greater undercoverage of welfare-eligible groups, which are less likely to have earnings, and also from errors in the administrative data, such as underreporting of earnings. Some part of the difference ma y be accurate in that eligible units with earnings may be less likely to apply for AFDC benefits than other units. Second, simulation of eligibility by state invariably results in several states with fewer simulated eligible units than reported participants. Again, population undercoverage may in part account for this finding. Moreover, apparently due in large part to sampling variability, the number of such states varies across years. In 1987 there were eight states with fewer simulated eligible units than participants; in 1983 there were only two such states.15 (Examination of other baseline files showed three such states in 1982, eight in 1985, and five in 1986; see Giannarelli [1990].) The calibration process endeavored to satisfy the constraints both for participants in each state and for the proportion with earnings. To match state-level controls for states with high simulated participation rates, it was necessary to select all or most of the eligible units in these states. Inevitably, this meant that the simulated participating caseload in these states included a higher proportion of units with earnings than shown by the administrative data; consequently, the national controls for participants with earnings could not be matched. The problem was more severe in 1987 than 1983 because of the larger number of states with very high participation rates. For variables not used as controls, application of the AFDC participation function and calibration process produced mixed results. Thus, total dollars of AFDC benefits are not part of the calibration, and, in fact, the 1987 and 1983 calibrated baseline simulations do not closely match the profile of benefits by state: 15 There were another five states in 1987 and seven states in 1983 with simulated participation rates of 90 to 99 percent.
ALTERNATIVE MODEL DESIGNS: PROGRAM PARTICIPATION FUNCTIONS AND THE ALLOCATION OF ANNUAL TO 115 MONTHLY VALUES IN TRIM2, MATH, AND HITSM ⢠The total dollar benefits simulated represented 95.1 and 97.3 percent, respectively, of actual benefits paid out in 1987 and 1983. Both percentages are lower than the corresponding percentages of simulated to actual participating units, and the difference is greater for 1987, which had a higher proportion of simulated participants with earnings whose benefits on average are lower than those of other participants. ⢠The state profile of the difference between actual and simulated benefits is very flat in 1987. After correcting for the national underestimate of benefits, simulated benefits were within plus or minus 1.9 percent of actual benefits in only 9 states, including 2 of the 10 largest, and within plus or minus 3.9 percent in only 15 states, including 2 of the largest. In 19 states simulated benefits differed from actual benefits by plus or minus 10 percent or more, and in 11 states simulated benefits differed from actual benefits by plus or minus 15 percent or more. ⢠The state profile of the difference between actual and simulated benefits is flat in 1983 as well. After correction for the national underestimate of benefits, simulated benefits were within plus or minus 1.9 percent of actual benefits in only 8 states, including 4 of the 10 largest, and within plus or minus 3.9 percent in only 18 states, including 5 of the largest. In 13 states simulated benefits differed from actual benefits by plus or minus 10 percent or more, and in 7 states simulated benefits differed from actual benefits by plus or minus 15 percent or more. Tabulations of various characteristics at the national level showed that the calibration was reasonably successful in reproducing the profile of the caseload shown in administrative records in many, but not all, instances. The problem for the calibration is that the simulation of AFDC eligibility from the March CPS produces two distinct groups: those eligible units who reported participation in AFDC to the CPS and those who did not. The group of eligible reporters rather closely fits the stereotypical pattern for AFDC of the large welfare family headed by a single woman with little other income or earnings. The administrative data are close to the profile of eligible reporters on many dimensions, although they do not exhibit quite as high a proportion of cases fitting the stereotype. At the other extreme, eligible units not reporting AFDC include a much higher proportion of smaller units headed by a married couple with some other income and earnings (see Table 1). Eligible reporters accounted for about half the total eligible population in both 1987 and 1983. However, for reasons that could include differential undercoverage and reporting errors in the CPS, they accounted for only about two-thirds of the total number of participants in each year needed to match the
ALTERNATIVE MODEL DESIGNS: PROGRAM PARTICIPATION FUNCTIONS AND THE ALLOCATION OF ANNUAL TO 116 MONTHLY VALUES IN TRIM2, MATH, AND HITSM TABLE 1 Characteristics of Simulated Units Eligible for and Reporting AFDC and Simulated Units Eligible for but Not Reporting AFDC (TRIM2 Model) and Actual Participants, 1987 Characteristic Percent of of Unit Eligible Reporters Eligible Nonreporters Participants Large unit: four or more persons 34 19 28 Youngest child age 10 or older 19 31 21 Headed by male 9 33 13 Headed by married couple 8 26 13 Headed by someone not the parent or stepparent of 6 26 7 youngest child Headed by someone not the head or spouse for the whole 19 31 3 household Headed by black 42 33 41 Headed by white, non-Hispanic 37 48 40 Unit with zero gross income 74 63 88 Unit with zero earnings 88 81 92 NOTE: Data for eligible reporters come from tabulations of the March 1988 CPS data file, TRIM2 baseline version, for units simulated by TRIM2 to be eligible for AFDC that also reported receipt of AFDC benefits to the CPS. Weighted count is 2,430,000 average monthly units. Data for eligible nonreporters come from tabulations of the March 1988 CPS data file, TRIM2 baseline version, for units simulated by TRIM2 to be eligible for AFDC that did not report receipt of AFDC benefits to the CPS. Weighted count is 2,361,000 average monthly units. Data for participants come fro m tabulations of the 1987 Integrated Quality Control System data file, containing samples of participating AFDC units from monthly case records. Weighted count is 3,710,000 monthly units. administrative data.16 Hence, even if the TRIM2 AFDC participation function automatically assigned participant status to all eligible reporters, which it does not, it would still have to draw from the pool of eligible nonreporters to reach the target caseload. The calibration process in 1987 produced a simulated participating caseload that resembled the administrative data on the dimensions of unit size, age of youngest child, sex, marital status, and race of head but not on the dimensions of relationship of head to youngest child, relationship of the 16 The March CPS includes additional AFDC reportersâabout 10 percent of the administrative totalâwho are simulated to be ineligible for the program by TRIM2 and hence are excluded from participation.
ALTERNATIVE MODEL DESIGNS: PROGRAM PARTICIPATION FUNCTIONS AND THE ALLOCATION OF ANNUAL TO 117 MONTHLY VALUES IN TRIM2, MATH, AND HITSM unit head to the household head, gross income, and earnings. The calibration was more successful in 1983, deviating substantially from the administrative profile of the AFDC caseload only on the dimensions of relationship of the unit head to the household head and gross income (see Table 2). TABLE 2 Characteristics of Simulated AFDC Participants (TRIM2 Model) and Actual Participants, 1987 and 1983 Characteristic Percent of of Unit 1987 Participants 1983 Participants Simulated Actual Simulated Actual Large unit: four or more persons 29 28 29 28 Youngest child age 10 or older 22 21 23 22 Headed by male 16 13 17 15 Headed by married couple 13 13 14 15 Headed by someone not the parent or stepparent of youngest child 12 7 10 11 Headed by someone not the head or spouse for the whole household 26 3 27 4 Headed by black 37 41 41 44 Headed by white, non-Hispanic 42 40 41 41 Unit with zero gross income 70 88 74 90 Unit with zero earnings 87 92 92 95 NOTE: Data for simulated participants come from tabulations of the March 1988 and March 1984 CPS data files, TRIM2 baseline version, for units simulated by TRIM2 to be eligible for and participate in AFDC. Weighted counts are 3,703,000 average monthly units for 1987 and 3,623,000 average monthly units for 1983. Data for participants come fro m tabulations of the 1987 and 1983 Integrated Quality Control System data files, containing samples of participating AFDC units from monthly case records. Weighted counts are 3,710,000 monthly units (1987) and 3,501,000 monthly units (1983). The 1983 file excludes cases for Hawaii. (The total count, with those cases, is 3,628,000.) The discrepancies between the March CPS profile of simulated AFDC eligible units and the administrative profile of actual participants, which make it difficult to calibrate the simulation of participation in the AFDC program, undoubtedly reflect problems with the March CPS data. However, particularly for those dimensions on which both eligible reporters and nonreporters of AFDC differ from actual participants (notably, relationship of the unit head to
