National Academies Press: OpenBook
« Previous: Analysis of Variance Methods for Sensitivity Analysis and External Validation
Suggested Citation:"Nonparametric Analysis." National Research Council. 1991. Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers. Washington, DC: The National Academies Press. doi: 10.17226/1853.
×
Page 291
Suggested Citation:"Nonparametric Analysis." National Research Council. 1991. Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers. Washington, DC: The National Academies Press. doi: 10.17226/1853.
×
Page 292
Suggested Citation:"Nonparametric Analysis." National Research Council. 1991. Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers. Washington, DC: The National Academies Press. doi: 10.17226/1853.
×
Page 293
Suggested Citation:"Nonparametric Analysis." National Research Council. 1991. Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers. Washington, DC: The National Academies Press. doi: 10.17226/1853.
×
Page 294

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

A VALIDATION EXPERIMENT WITH TRIM2 291 the 16 versions. After all, there were reasons for the choices that were made in the current version of TRIM2. On the other hand, it can be argued that, since the alternatives were fairly exchangeable a priori, the overall mean is more representative of the current state of knowledge than version 1. A problem with this argument arises if some of the alternatives are not equally reasonable a priori, in which case the overall mean could be misrepresentative of the current state of knowledge and have quite different properties from version 1. As a result, effects that brought the overall mean closer to the comparison values might not do the same for version 1. This raises two points. First, this analysis is most relevant when the alternative modules are essentially equally credible, where it is not clear a priori whether one alternative is better than another. For the most part this was true of the present experiment, though MONTHS is generally considered to be superior to old MONTHS. In addition, in acknowledgment of the influential effect that the results from any of the 16 versions could have on the analysis, it is often of interest to repeat this type of analysis using some form of robust analysis of variance to reduce the impact of a large interaction effect, or possibly a clearly inferior alternative module. Nonparame tric Analysis Another way of communicating the relative strengths of the various model versions is to simply display in a table the differences between the various estimates and the comparison values for each version and each response of interest. A slight problem with this is that there is no standardization of the discrepancies. A similar analysis that has a natural standardization is to replace the differences for each response by the ranks of the absolute differences over the 16 model versions. These ranks of estimates of change are provided in Table 5; the analogous information for level is provided in Table 6. In addition to the 16 model versions, the 1983 IQCS data are included for comparison. (Therefore, the largest difference received rank 17.) Since there are 17 ranks for every variable, the average rank across runs is 9. The bottom rows of Tables 5 and 6 include the average rank for each model across responses. Of course, these averages are highly dependent on the response variables included in the analysis; therefore, these averages should be interpreted carefully, with full cognizance of the effect of variable selection. Tables 5 and 6 support six hypotheses, which ma y or may not survive further scrutiny. First, this analysis seems to indicate that, while some model versions have advantages in estimating certain responses, there does not appear to be a clearly superior version of TRIM2. There is certainly a great deal of variability from one response variable to another. Without considering subject matter information, it is difficult to identify much structure. Second, full aging, used in versions 4, 8, 12, and 16, appears to be the worst of the four types of aging in matching the comparison values, buttressing

TABLE 5 Ranks of Model Discrepancies From Comparison Values for Estimates of Change Response Variable Run Identification 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 QC83 Race of head 15 4 9 17 12 2 8 14 10 3 11 13 7 1 5 6 16 Earnings of adult 1 4 3 9 14 15 13 17 6 2 5 7 11 10 12 16 8 Marital status of head 13 2 9 14 16 1 11 12 17 8 4 5 15 6 3 7 10 Sex of head 8 1 10 17 9 2 11 16 14 12 4 7 15 13 3 5 6 Age of head 6 1 14 16 8 2 13 15 5 3 11 10 12 4 7 9 17 Type of AFDC unit 6 2 14 16 10 1 15 17 11 4 8 12 9 3 7 13 5 No. of adults 7 1 14 17 8 2 15 10 9 5 10 12 6 3 11 13 4 No. of children 2 15 17 16 4 11 14 13 1 10 12 9 3 7 8 6 5 Total no. in unit 10 16 3 7 8 14 1 11 13 17 2 9 12 15 6 5 4 Age of youngest child 3 16 14 12 4 17 15 13 2 10 8 6 5 11 9 7 1 Total participants 3 5 14 17 2 8 13 16 4 9 10 15 1 12 7 11 6 Total participants with earnings 3 2 6 15 12 11 14 17 4 1 5 9 10 7 13 16 8 A VALIDATION EXPERIMENT WITH TRIM2 Total benefits 1 12 9 14 4 15 8 11 2 13 5 10 7 16 3 6 17 Average rank 6.0 6.2 10.5 14.4 8.5 7.8 11.6 14.0 7.5 7.5 7.3 9.5 8.7 8.3 7.2 9.2 8.2 292

TABLE 6 Ranks of Model Discrepancies From Comparison Values for Estimates of Level Response Variable Run Identification 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 QC83 Race of head 4 15 7 3 6 16 8 5 12 2 13 14 11 1 9 10 17 Earnings of adult 10 6 7 1 3 5 2 12 9 4 8 11 14 13 15 17 16 Marital status of head 4 5 10 13 6 8 11 12 1 3 15 16 2 7 14 17 9 Sex of head 16 11 8 1 17 15 7 5 12 10 4 3 14 13 6 2 9 Age of head 3 5 13 17 1 4 12 16 6 8 15 14 2 7 10 11 9 Type of AFDC unit 16 4 7 10 17 3 9 11 15 13 2 5 14 12 1 6 8 No. of adults 2 4 14 17 3 7 15 16 9 5 10 12 8 1 11 13 6 No. of children 2 9 11 10 1 6 8 7 5 16 17 15 4 13 14 12 3 Total no. in unit 7 14 5 3 6 12 4 2 15 17 11 9 13 16 10 8 1 Age of youngest child 5 15 12 9 4 16 14 11 17 3 7 10 13 2 6 8 1 Total participants 4 5 14 17 2 8 13 16 3 9 10 15 1 12 7 11 6 Total participants with earnings 7 8 2 6 4 3 5 14 9 1 10 12 13 11 15 17 16 A VALIDATION EXPERIMENT WITH TRIM2 Total benefits 4 8 13 16 2 9 11 14 1 12 6 10 5 15 3 7 17 Average rank 6.5 8.4 9.5 9.5 5.5 8.6 9.2 10.8 8.8 7.9 9.9 11.2 8.8 9.5 9.3 10.7 9.1 293

A VALIDATION EXPERIMENT WITH TRIM2 294 the results seen above. Also, not aging and demographic aging generally are relatively successful. Third, adjusting the data for undercoverage (with the admittedly suboptimal method used) appears to have a slight advantage over not adjusting, which can be seen by comparing the average ranks for run 1 versus 9, 2 versus 10, and so forth. In addition, adjustment seems to have a moderating influence, with all eight versions using adjustment having rather similar performance. Thus, use of adjustment tends to obscure the differences between aging routines and MONTHS and old MONTHS, with all eight versions having, roughly, average performance. Fourth, MONTHS appears to be slightly superior to old MONTHS, but the advantage is small. Fifth, the 1983 quality control estimates for change, which assume no change from 1983 to 1987, performed better than the average model version (the average rank being 9 for Table 5). Under some circumstances the comparison values for the beginning of the period of study provide an interesting challenge to the estimates from the various model versions. If the study had examined a situation in which a substantial policy change had occurred, use of the old quality control data provides a standard that a reasonable model should exceed. Namely, the model should perform better than an estimate based on the assumption of no change over the period. This type of comparison is extremely informative in providing a naive estimate of how well one can do with a very simple model. (Other less naive estimates also should be considered for this purpose. Certainly, if the same or smaller errors are achievable by using a model that is less expensive to run and maintain, the worth of the more expensive model is harder to justify.) Also, this comparison provides an estimate of how much variability is natural to the problem, which can be compared to the variability left unexplained by the model versions. This particular experiment, as discussed further below, examined a situation in which the policy change was modest but the economic changes were relatively large. Under these circumstances the comparison with an estimate of no change is less important, because the noise is in some sense too large a fraction of the signal. Under a test where there was a larger policy change, this result would suggest that versions that failed to outperform the assumption of no change (and other naive estimators) are seriously flawed. Even in this unfair test many of the model versions here easily outperformed the assumption of no change. Finally, it is important to notice that in Table 5 the best model is version 1, the current version of TRIM2, lending credence to the theory that all of the modules were not equally plausible a priori and that proper choices had been made as to which alternatives should be included in TRIM2. The above nonparametric analysis has some limitations. Since it is descriptive, significant differences between models cannot be identified. This can be remedied by using Friedman's test (see, e.g., Lehmann, 1975), which could be applied directly to Tables 5 and 6. A problem with this approach is that it

Next: Analysis of Categorical Data »
Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers Get This Book
×
Buy Paperback | $100.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

This volume, second in the series, provides essential background material for policy analysts, researchers, statisticians, and others interested in the application of microsimulation techniques to develop estimates of the costs and population impacts of proposed changes in government policies ranging from welfare to retirement income to health care to taxes.

The material spans data inputs to models, design and computer implementation of models, validation of model outputs, and model documentation.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!