Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
EVALUATIONS OF MICROSIMULATION MODELS: LITERATURE REVIEW 258 showed that ignoring differentials in taxable earnings and assuming that average taxable earnings will increase at the same rate as average earnings in general is not valid, assuming the interactions given in DYNASIM. They then identified the source of the discrepancy as a shift in the age distribution of the population. Their hope was that this finding could be used to improve the actuarial model. Hendricks and Holden mention that this is one of several advantages that DYNASIM provides modelers. GENERAL ACCOUNTING OFFICE (1977) The report An Evaluation of the Use of the Transfer Income ModelâTRIMâto Analyze Welfare Programs (U.S. General Accounting Office, 1977) was probably the first attempt to determine the magnitude of the errors present in TRIM. Although the primary use of TRIM and its successor TRIM2 is to analyze program alternatives, the actual effects of most alternatives are never known because most are never enacted. Even for existing programs, the primary amounts that the model estimates (i.e., the number of people eligible for a given transfer program and the associated costs) are hard to substantiate because of the lack of administrative data. Even though GAO tested TRIM, it was not able to determine to its satisfaction the model's accuracy. The GAO's report noted, however, that the accuracy or reliability of TRIM's estimates should not be overlooked because of this lack of administrative data. In addition, the inability to make this kind of comparison increases the need for information about the range of variability of the estimates generated by TRIM. GAO's efforts at model validation were hampered by the poor quality and lack of documentation of TRIM, which made it difficult to understand the model's assumptions, capabilities, and limitations. The general approach that GAO used to overcome the problem of a lack of administrative data or other comparison values was a sensitivity analysis of TRIM. GAO examined the impact on TRIM of (1) adjusting for income underreporting, (2) alternative procedures for determining appropriate filing units, (3) aging the database, (4) simulating the assets test for each transfer program, and (5) simulating participation in each transfer program. GAO analyzed the effect of variance in the input data set on the variability of TRIM's estimates. The GAO report noted that estimates acquired from sample survey data might differ from figures that would have been obtained if there had been data on every person rather than a sample of persons. Because of this the Census Bureau describes the estimates it makes from the CPS, for example, in terms of standard errors. But, TRIM's estimates, which also are based on these data, do not contain any similar indicator of variability. GAO stated that such an indicator is warranted. GAO cited a report by Beebout et al. (1976) that investigated the effect on TRIM's estimates of income underreporting and found that by adjusting
EVALUATIONS OF MICROSIMULATION MODELS: LITERATURE REVIEW 259 for it the model's estimates of the number of food-stamp-eligible households were reduced by nearly 10 percent. GAO also examined the use of alternative procedures for determining the appropriate definition of tax and transfer program filing units. It discovered that simulated costs and caseloads for the supplemental security income (SSI) program varied substantially, depending on the procedure used. It also determined that estimates for the Aid to Families with Dependent Children (AFDC) program had a 3.5 percent difference for the number of units and a 4.8 percent difference in program costs. GAO also found that aging the database had a measurable effect on TRIM's estimates (see also Beebout, 1974). The March 1973 CPS was projected out 2 years by using several TRIM aging options, which were input into the model. In addition, the 1975 March CPS was input, with the output serving as the âtruth,â and the various outputs were compared. Even with only 2 years of aging, one particular demographic aging option, D5, was strongly preferred to the remaining three options in estimating the total numbers of households, families, persons, and children. With respect to estimating income distribution, three aging methods were compared to the March 1975 CPS, this time with a preference for a different demographic aging option as well as an economic aging option. Finally, the impact of the aging process on tax and transfer program estimates was investigated. GAO concluded that such estimates will vary depending on the aging option that is selected. GAO also examined the impact of using various assets tests on estimates of the caseload and benefits for each transfer program as well as assumptions about participation in such programs. GAO concluded that the assumptions made in the participation simulation procedures for AFDC, SSI, and general assistance were highly subjective. The procedures appeared, to GAO, to be ad hoc processes aimed at matching a known level of program participation without concern for why a person chose to participate or not to participate. The procedures were thought to yield accurate estimates for existing programs for current or past years, since all the necessary data were available. But any of the procedures would be meaningless in evaluating wholesale program revisions (since participation behavior might change also) or in estimating future participation in existing programs (since data would not be available for comparison). In addition, GAO pointed out that TRIM had a limited ability to accommodate changes in the unemployment rate and no ability to account for people's behavioral responses to proposed changes in the structure and benefits of transfer programs. Finally, the GAO report discussed the high cost of performing a sensitivity analysis of TRIM. Using the model to simulate a transfer program necessitates specifying a considerable amount of input data, which makes preparation for a TRIM simulation relatively complicated, particularly for someone who is not an experienced TRIM analyst.