Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
EVALUATIONS OF MICROSIMULATION MODELS: LITERATURE REVIEW 265 However, without exchanging these modules in the two models, it is not clear how much difference this makes. Haveman and Lacker do point out that it is sometimes possible to perform the equivalent of a sensitivity analysis by simply examining the competing algorithms. For example, they argue that PRISM's estimates of outlays in private defined benefit plans would be nearly 50 percent higher than those from DYNASIM as a result of indexing nominal-valued constants to different series. They admit to the limitations of this approach though. Summing up, Haveman and Lacker found both models to be impressive and highly innovative. However, they believed that this impressiveness has an associated cost since the enormous complexity of these models makes them, effectively, black boxes. While the input into them can be understood and judged, and the projections are illuminating, the assumptions and other inputs that are used to produce the projections cannot be understood. The interaction of the complex relationships, transition matrices, time-triggered status changes, and so on is so complicated that little feel is possible for why the resulting projections are what they are. Consequently, there is little justification for a reviewer to believe the predictions of one model more than those of another. Haveman and Lacker recommended three approaches for investigating microsimulation models: backcasting, sensitivity analyses where certain modules would be switched between the models, and the analysis they performed for PRISM and DYNASIM. While Haveman and Lacker's research is not a validation analysis, their careful scrutiny of DYNASIM and PRISM, in conjunction with an external validation, would greatly expedite determination of the cause of any discrepancies between the models and the truth. In addition, work of this sort provides modelers with obvious examples of alternative methodologies that can be used in a sensitivity analysis. Therefore, while this analysis is not a validation analysis per se, it is an important addition to validation. ICF, INC. (1987) ICF, Inc. (1987) compared aggregate estimates from TRIM2 with those for HITSM, in projecting the costs of various provisions of President Ford's welfare reform bill (H.R. 1720). Both models were based on the March 1986 CPS data. In addition, estimates for both models were compared with AFDC quality control data, which served as a surrogate for the truth. In terms of the total cost, TRIM2 and HITSM's estimates were within 8 percent of each other. With respect to the individual provisions, (1) TRIM2 estimated that the mandatory unemployed parent (UP) program would cost about $370 million in 1985, while HITSM estimated that it would cost $422 million; (2) TRIM2 estimated that elimination of the work expense deduction and use of $100 and 25 percent earnings disregard would cost $232 million, while HITSM estimated that it would cost $164 million; (3) TRIM2 estimated
EVALUATIONS OF MICROSIMULATION MODELS: LITERATURE REVIEW 266 that elimination of the child care disregard would cost $93 million, as opposed to HITSM's estimate of $34 million; and (4) TRIM2 estimated that elimination of the deeming of grandparent's income would cost $129 million, while HITSM estimated the cost at $228 million. According to ICF, Inc., the difference in estimates of the costs of the UP provision was due to TRIM2's underestimate of the average AFDC UP benefits. HITSM estimated this to be $496, AFDC quality control data reported $504, and TRIM2 estimated it as $448. The difference in estimates of the costs of the work expense and earnings disregard provision resulted from the fact that TRIM2 estimated a larger number of earners as receiving AFDC benefits than HITSM did. Again, the AFDC quality control data supported the HITSM figure. The difference in estimates of the cost of the child care disregard was due to differences in the estimated average number of units per month benefiting from child care. TRIM2 estimated it to be 84,000, while HITSM estimated it as 41,000. The AFDC quality control figure was 47,000. The difference in estimates of the cost of eliminating the deeming of grandparent's income resulted from the different methods used to define age in the two models. As a result, HITSM counted more children than TRIM2 did. ICF, Inc., determined that the different estimates were due to the fact that TRIM2 and HITSM's current law simulations were substantially different. For example, TRIM2 reported about 45 percent more earners receiving AFDC than HITSM did under current law. To understand the source of these differences, ICF examined the differences in the two models and found that a major difference is that the HITSM estimates were adjusted for underreporting of income in the CPS data while the TRIM2 estimates were not. Also, the models made use of different forms of imputation and statistical matching to augment the CPS. Finally, the models controlled their simulation results to replicate different control totals from administrative data. HITSM controlled to nationwide totals of participants and benefit levels and for the number of children, benefit size, and number of adults. TRIM2 controlled to state-by-state participation totals for the large states, as well as unit type and proportion of the caseload with earnings for the United States. Clearly, if this comparison had involved those responses that benefit from the controlling that TRIM2 does, the comparisons would have been more favorable for TRIM2. ICF, Inc., concluded that in different applications each of the models has its relative strengths and weaknesses. For example, TRIM2 is likely better for state-level analyses because it includes a state-by-state participation function. Because HITSM can more closely replicate actual data on the number of units with earnings deductions and AFDC UP benefit amounts, it may be better than TRIM2 for analyses of changes in earnings disregards and in AFDC UP benefits. 5