The Role of Microsimulation as a Policy Analysis Tool
In this part of our report we turn from a broad investigation of the role of information in the policy process to examine with both a narrower and a deeper focus the role of microsimulation modeling as a tool for analysis of social welfare policy issues. Our sponsors—the Office of the Assistant Secretary for Planning and Evaluation (ASPE) in the U.S. Department of Health and Human Services and the Food and Nutrition Service (FNS) in the U.S. Department of Agriculture—asked us to address the basic question of how the agencies should use microsimulation modeling in the 1990s and future decades to support their policy-oriented missions. They further asked us to evaluate the role played by microsimulation models in the legislative debates of the past and to consider advances in databases, research knowledge, computing technology, and statistical evaluation methods that could improve the ability of microsimulation models to support the legislative debates in the future.
We address some of the questions about the historical role of microsimulation modeling in our review of the policy process in Part I. We describe the ''first revolution" in the use of numbers to guide policy choices, which fostered the growth in importance over the past 20 years of formal modeling tools as a source of detailed information for legislative decision making. Unfortunately, underinvestment in needed data, research, and model development in the 1980s led to declining, or at best, stagnant capabilities of microsimulation and other types of models to support the policy debate. We have identified two particular areas in which new investment is sorely needed: investment to improve the quality and relevance of the input data used by models and policy research generally, and investment to add value to the outputs of models through systematic
validation, documentation, and effective presentation of the results to decision makers. We note particularly the need for thorough evaluation of the quality of the information produced by models, both because policy makers need to be aware of the extent and sources of uncertainty in the estimates and because policy analysis agencies need validation results to develop cost-effective strategies for investment in future model development. Thus, we call for a "second revolution," in which both the numbers and associated measures of their quality inform the legislative process and contribute to the development of improved policy analysis tools.
Although our recommendations in Part I pertain generally to the entire range of models, in almost every instance they apply with particular force to microsimulation. The complexity and large scale of most models of this type make it both more than usually difficult and more than usually important to effect improvements in model inputs and to develop systematic programs for validation of model outputs.
Our discussion in this part draws most heavily on experience with the major models for income support programs, which have long been central to our sponsors' interests; however, we also consider models and modeling issues specific to health care, retirement income, and tax policy. (The health care policy area, in particular, is rapidly becoming of critical importance to decision makers.) Among the models that we reviewed are
TRIM2 (Transfer Income Model 2), MATH (Micro Analysis of Transfers to Households), and HITSM (Household Income and Tax Simulation Model), which are static models of income support and tax programs;
DYNASIM2 (Dynamic Simulation of Income Model 2) and PRISM (Pension and Retirement Income Simulation Model), which are dynamic models of retirement income programs;
the submodel added to PRISM to simulate alternatives for financing long-term care of the elderly;
the tax policy model maintained by the Office of Tax Analysis; and
MRPIS (Multi-Regional Policy Impact Simulation), which is a hybrid income support and tax policy model that uses microsimulation, input-output, and cell-based techniques.
We first describe the components and operational steps involved in microsimulation, briefly review the important stages in the development of microsimulation models for policy analysis both in the United States and in other countries, and present our general findings about the utility of the microsimulation approach (Chapter 4).
In the remainder of Part II we address in detail the critical issues in microsimulation models: databases (Chapter 5); design principles and practices, and possible expansions in model capabilities (Chapter 6); computing environments for models (Chapter 7); special problems in health care, retirement
income, and tax policy modeling (Chapter 8); validation of model results (Chapter 9); documentation and archiving of models (Chapter 10); and the structure of the microsimulation modeling community, including the potential of microsimulation for use in social science research (Chapter 11). Each chapter includes recommendations. The Appendix to Part II provides background information on the main features of the microsimulation models listed above, the primary databases used by the models, and technical terms used in microsimulation modeling.
The reader will note that our recommendations for future development of microsimulation models are quite often general in nature and include calls for further study. In particular, we are not able to provide much guidance about desirable changes to microsimulation model design and capabilities—for example, whether model development should emphasize the incorporation of added behavioral elements and whether current static models should adopt more dynamic approaches—because we have found that there is simply no literature evaluating microsimulation models. Very few rigorous studies have been carried out to assess the sensitivity of models to alternative assumptions or specifications or to compare the accuracy of model projections against known values. (As part of its work, the panel conducted a limited validation study of the TRIM2 model, combining an external validation with sensitivity analysis.) Likewise, there have been almost no attempts to assess the variance in model outputs due to such sources as sampling error. Hence, despite obtaining much helpful information and ideas from knowledgeable people in the microsimulation and research communities, we cannot responsibly support specific recommendations about the priority of adding this or that capability or making this or that design change to microsimulation models. We also do not provide recommendations directed to specific models, due both to the paucity of validation studies and to our belief that a broad-based assessment of microsimulation modeling is more important than a detailed evaluation of any particular model. Our frustration on both matters underscores our view that an overriding priority for policy analysis agencies must be to provide adequate resources and support for systematic evaluation efforts that can guide cost-effective development of microsimulation models in the future.