Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers (1991)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

A VALIDATION EXPERIMENT WITH TRIM2 283 variability. Third, there is the hope that superior versions of microsimulation models will have an omnibus superiority robust to choice of loss function. We examined a variety of outputs from TRIM2. Table 2 provides definitions for these responses. The first 10 defined variables are categorical variables that were studied in both their original form and a dichotomized (0– 1) version. An advantage of the dichotomized version is that, since its mean is a percentage, analysis of projections as estimates of change is simplified, there being numerous ways to analyze changes in the distribution of a categorical variable. Two outputs of great interest are the aggregate responses of total participants and total benefits. ANALYSIS OF RESULTS Total Variability Due to Use of Alternative Module s The primary question we address is: What differences in the outputs result from use of alternative components in the three modules? In other words, do the alternatives make a difference? For estimates of change in the total number of AFDC participants from 1983 to 1987, the estimates from the 16 versions of TRIM2 range (in units of 1,000) from −213 to 293, while the comparison value from the quality control files is 98 (i.e., QC87 −QC83=98). For estimates of change in total benefits, the estimates from the 16 versions of TRIM2 range (in millions) from $1,578 to $3,742, while the comparison value is $2,499. These results indicate that, in using alternatives that were thought a priori to have similar success in modeling AFDC, there are versions that disagree about whether the number of participants goes up or down and by an amount that was at least twice the size of the observed change of 100,000 people. Also, these alternatives provide estimates of change in total payouts that range from increased costs of $1.6 billion to $3.7 billion. Moreover, the estimates of change are with a forecast horizon of only 4 years. Table 3 provides other ranges for estimates of both change and level for other variables of interest. Note that the comparison value for change was outside the range provided by the 16 versions of TRIM2 for the following variables: (1) percentage of households where youngest child is less than 5 years old, (2) percentage of households where race/ethnicity of head is black non-Hispanic, and (3) percentage of households with no adult earnings. Also, it was not uncommon for versions of TRIM2 to incorrectly identify the direction of change; this occurs for 7 of the 10 variables examined. (Note that if a difference is not significantly different from 0, which may be true for some of these estimates, it is irrelevant whether the estimated sign is in the wrong direction.) For estimates of level there are also three variables for which the comparison value (as given by the QC87 value) is not included in the range of values supplied by the 16 versions of TRIM2. These are (1) percentage of 

A VALIDATION EXPERIMENT WITH TRIM2 284 TABLE 2 Definitions of Selected Output Variables From TRIM2 Response Variable Definition Individual Type of unit AFDC unit type. The responses are basic, unemployed parent, and incapacitated parent. The dichotomized version studied was 1 for basic, 0 otherwise. Number of adults Adults eligible to be aided in AFDC unit. The responses are 0, 1, and 2. The dichotomized version studied was 1 for two and 0 otherwise. Number of children Children eligible to be aided in AFDC unit. The responses are 0, 1, 2, 3, and more than 3. The dichotomized version studied was 0 for more than two children, 1 otherwise. Number aided Total persons eligible to be aided in AFDC unit. The responses are 0, 1, 2, 3, 4, and more than 4. The dichotomized version studied was 0 for more than three, 1 otherwise. Age of youngest child Age of youngest child in AFDC unit. The responses are 0–1, 2–4, 5–9, 10–14, and 15– 18. The dichotomized version studied was 1 for less than 5, 0 otherwise. Race of head Race/ethnicity of AFDC unit head. The responses are white non-Hispanic, black non- Hispanic, Hispanic, and other non-white. The dichotomized version studied was 1 for black non-Hispanic, 0 otherwise. Adult earnings Monthly earned income of adults in AFDC unit in 1987 dollars. The responses are none, $1–$199, $200–$499, and more than $499. The dichotomized version studied was 0 for 0 earnings, 1 for positive earnings. Marital status Marital status of AFDC unit head. The responses are never married, married spouse present, married spouse absent or separated, married spouse in the armed forces, married spouse absent (other), widowed, and divorced. The dichotomized version studied was 1 for no spouse, 0 otherwise. Sex of head Sex of AFDC unit head. Female was coded as 1, male as 0. Age of head Age of AFDC unit head. The responses are 14–16, 17–19, 20–24, 25–29, 30–39, 40– 49, 50–59, and 60 and over. The dichotomized version was 1 for less than 20, 0 otherwise. Gross income Gross monthly income of AFDC unit members in 1987 dollars. The responses are none, $1–$199, $200–$499, and more than $499. Relationship Relationship of AFDC unit head to household head. The responses are head/spouse and other. Size of benefit Size of benefit in 1987 dollars. Aggregate Total participants Average monthly number of AFDC participants (in thousands). Total participants with earnings Average monthly number of AFDC participants with earnings (in thousands) Benefits Annual benefits (in $ million) to AFDC participants. 

A VALIDATION EXPERIMENT WITH TRIM2 285 households where there are more than two children, (2) percentage of households where the number receiving assistance is greater than three, and (3) percentage of housholds where the head of household is less than 20 years old. In general, Table 3 demonstrates that potentially important differences were obtained for ma ny of the responses studied when alternate modules were used, although the actual policy implications of ma ny of the differences depend on the intended application of the estimates. TABLE 3 Observed Differences for Various Estimates from TRIM2 Validation Experiment Variable IQCS87 Level Range of Level, IQCS87 Minus Range of Change, Runs 1–16 IQCS83 Runs 1–16 Minus 83 Base Type of AFDC unit (% 90.40 87.99, 91.90 1.20 −0.45, 3.52 basic) No. of adults (% 2) 9.58 6.32, 10.43 −0.65 −3.41, 0.42 No. of children (%>2) 25.98 26.36, 28.21 0.58 0.29, 1.79 Total no. in unit (%>3) 28.20 28.72, 31.10 0.13 −0.27, 1.46 Age of youngest child 53.58 51.96, 55.25 −0.09 −2.76, −0.64 (%<5) Race of head (% black 41.30 39.50, 42.75 −2.84 −1.85, 0.01 non-Hispanic) Earnings of adults (% 92.31 91.22, 94.88 −2.26 −0.91, 1.67 none) Marital status of head 86.82 86.01, 90.42 1.61 −0.45, 3.60 (% no spouse) Sex of head (% female) 86.77 82.96, 86.89 1.30 −0.50, 3.53 Age of head (%<20) 5.73 6.51, 7.30 −1.18 −1.51, −0.62 NOTE: Differences among model runs and between a model r un and the IQCS ma y not be statistically significant. These observations are not meant to imply any weakness of TRIM2 relative to other models, microsimulation or not, whose goal is to provide estimates of the above quantities. No such conclusions can be drawn, given that this analysis does not include other modeling approaches. What this experiment does demonstrate is the large amount of variability that results from reasonable changes to the basic model.4 In other words, the alternatives that were investigated made a difference. 4These ranges should not be interpreted as any type of confidence interval.