National Academies Press: OpenBook
Suggested Citation:"DESIGN OF EXPERIMENT ISSUES." National Research Council. 1991. Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers. Washington, DC: The National Academies Press. doi: 10.17226/1853.
Page 301

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

A VALIDATION EXPERIMENT WITH TRIM2 301 owing to the likely correlation between states for some response variables. The limitations that result from examining TRIM2 at one point in time help demonstrate that model validation should not be an occasional examination of a model. Rather, it should be a continuous process that accumulates knowledge about potential weaknesses, the size of errors, and situations in which the model is fallible. Another limitation of the present experiment was that, in addition to simulating just one time period, just one change in the law was simulated. Had time and resources permitted, we would like to have simulated alternative policy scenarios—for example, a mandated AFDC minimum benefit nationwide, which would have represented a major policy change. Of course, we could not have conducted an external validation of alternative policy scenarios because they were never enacted. We could have conducted sensitivity analyses of alternative policies and obtained information relevant to the question of whether, in fact, microsimulation model estimates of differences between two policies are less variable than estimates for a particular policy because of sources of error similarly affecting both policies. A final limitation of the experiment was that we examined only one model, owing to the time and resource constraints that we, the sponsoring agencies, and the agencies' contractors operated under. It would have been desirable to expand the experiment to include other major models in use today, such as MATH, HITSM, DYNASIM2 (Dynamic Simulation of Income Model 2), PRISM (Pension and Retirement Income Simulation Model), and MRPIS (Multi-Regional Policy Impact Simulation) model. We should note, however, that all of these models are not directly comparable, because they are often used for different purposes and, by design, cannot necessarily produce estimates of the same quantities. Therefore, adding the model class as another factor in, say, an analysis of variance, may not generally be feasible. However, in cases in which models are directly comparable, at least for some estimates, one could certainly expand our basic experiment in this direction. If the problems of comparability could be overcome, there are major advantages to be gained through comparing the effectiveness of different global modeling strategies. For example, the question of the relative advantages of static versus dynamic aging could be addressed in this way. Comparing several models with the truth (or a reasonable surrogate), along with a comprehensive analysis of model differences as exemplified by Haveman and Lacker (1984), will often yield great insight into the strengths and weaknesses of the various models. DESIGN OF EXPERIMENT ISSUES As model validation of microsimulation models is pursued further, some issues of experimental design that were ignored for the most part here will have to be

Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers Get This Book
Buy Paperback | $100.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

This volume, second in the series, provides essential background material for policy analysts, researchers, statisticians, and others interested in the application of microsimulation techniques to develop estimates of the costs and population impacts of proposed changes in government policies ranging from welfare to retirement income to health care to taxes.

The material spans data inputs to models, design and computer implementation of models, validation of model outputs, and model documentation.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook,'s online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!