Microsimulation Models: Then and Now
BASIC ELEMENTS OF MICROSIMULATION MODELS
Defined very simply, the microsimulation approach to evaluating alternative legislative proposals involves modeling the impact of government programs at the level at which they are intended to operate. That is, instead of modeling the impact of program changes on aggregates, such as the national economy or demographic subgroups of the population, microsimulation looks at the impact on individual decision units, which may be families in the case of income support programs, hospitals and doctors in the case of health care cost reimbursement programs, or corporations in the case of changes to corporate-based taxes. Given the diversity of the population, the complexity of most government programs, and all of the factors that need to be taken into account in developing an appropriate microlevel comparison of current policy with one or more hypothetical alternatives, the microsimulation approach inevitably entails a large number of steps.
In Figure 4-1, we sketch the major operations involved in microsimulation modeling, using income-support program models, such as Micro Analysis of Transfers to Households (MATH) and Transfer Income Model 2 (TRIM2), as the prototype. Not every microsimulation model includes all of the steps that are diagrammed, nor do all models implement the steps in the order shown; nonetheless, the chart helps to convey the complexity that characterizes most microsimulation models.
The chart begins with the operations involved in generating the main database for input to a microsimulation model. For many models, this database is a household survey, such as the March income supplement to the Current Population Survey (CPS). For other models, the main database is a set of administrative records, such as the Statistics of Income (SOI) sample of tax returns. In either case, the originating agency, such as the Census Bureau, goes through a series of operations to generate a microdata file. For a household survey such as the March CPS, these operations include survey and questionnaire design, data collection, data processing, data adjustments (such as imputing values for missing responses and weighting the records to represent the population), and preparing the data for public release. All of these steps have an impact on the quality and utility of the data for microsimulation purposes.
Despite the many operations completed by the originating agency, most microsimulation models go through a number of additional steps to create a database that is suitable to the model's purposes. These steps typically include converting the file to a format that suits the model's hardware and software (e.g., converting a character file to a binary format or converting a household-family-person record structure to a family-person structure) and generating recodes of variables. An important set of recedes for many models is to determine the members of a "filing unit," that is, those people within a household or family who constitute the unit eligible for benefits from a program such as Aid to Families with Dependent Children (AFDC) or the unit that will file a tax return. Models such as TRIM2 and MATH develop a large number of filing unit recodes in a single operation in order to provide flexibility for analysts in specifying policy alternatives. Modelers also typically adjust the database in a variety of ways to make the data more useful for their purposes. Such adjustments include correcting income amounts for underreporting—using control totals from outside data sources such as the National Income and Product Accounts—and imputing missing variables needed in simulations (e.g., imputing various kinds of expenses to the March CPS, such as child care payments that are allowable deductions for programs like AFDC). More elaborate adjustments sometimes involve matching entire data files to the main database, either through exact matching or statistical matching techniques. The major tax models routinely match SOI and CPS records to create a richer database for the entire population; the major retirement income models start with exact-match files containing CPS data and Social Security Administration records of earnings histories. (Exact matches make use of unique identifiers, such as social security numbers, that are present in two or more files to match records for the same people. When exact matching is not possible, statistical matches can come into play, combining records for people who resemble each other on a set of variables contained in two or more files.)
All of these adjustments make use of other data sources that also reflect many operations on the part of the originating statistical or administrative
agency to collect the data and put them into a form suitable for policy analysis and public release.
Another set of steps that is often, although not always, part of generating a suitable database for modeling, involves projecting, or "aging," the data forward in time. At best, the input data available to modelers are 1 year out of date; usually, they are several years old. At the same time, policy changes are usually proposed for implementation at least 1 year into the future, and Congress typically requires cost estimates for 5 years from the anticipated implementation date. For some policy proposals, such as changes to the social security benefit structure, the desired projection period may extend 30 years or more into the future. Hence, analysts need to make the database more closely resemble anticipated future conditions.
The major income-support program and tax models use "static" aging techniques to project the database. This method involves reweighting the data records to match outside control totals on selected characteristics. Typically, the models adjust the weights to agree with population projections by such characteristics as age, race, sex, and household composition: for example, changing the weights of people aged 24-35 in 1990 to represent the number of people who are expected to be in that age category in, say, 1995. (The population projections themselves are usually the output of cell-based models.) The models may also scale income amounts to agree with inflation forecasts of macroeconomic models and adjust employment status.
The major retirement income models use "dynamic" aging techniques: the model itself generates the expected future composition of the population by applying probabilities for birth, death, marriage, employment status change, and other processes to the individual records; Figure 4-2 sketches the basic steps typically involved in dynamic aging. These models "grow" the people in the database, for example, simulating an unmarried adult age 24 in 1990 to turn age 25 and marry in 1991, to turn age 26 and have a child in 1992, and so on. The aging portion of dynamic models (unlike that of static models) is central to their structure and operation.
After file conversion, data adjustment, and aging, the modeling approach follows a series of steps that involve what one might think of as the model itself. The first step in this sequence typically is to simulate current programs (create a baseline), by using the portion of the model that endeavors to replicate the program accounting rules, and to add to the data records values for eligibility status, the benefit to which the unit is entitled, and so on. A critically important part of the "baseline" simulation involves simulating the participation decision for those units that are determined to be eligible for program benefits. Often, the baseline file is created as soon as a new database is obtained and brought on line so that it is ready for use when the agencies request runs on alternative programs.
An integral part of creating a baseline data file involves adjusting or
"calibrating" one or more aspects of the simulation so that the simulated values agree as closely as possible with available control totals. For example, the TRIM2 modelers calibrate the baseline simulation of participation in AFDC so that the number and characteristics of units selected to participate in the program match administrative reports of average monthly recipients by state and a few characteristics of the national caseload. The projected database produced by dynamic models is also typically calibrated to outside economic and population projections. Considerable judgment by an analyst enters into the determination of how much effort to devote to the calibration process, because it is rarely possible to achieve complete agreement with all of the desired controls.
The next step is to simulate one or more program alternatives according to the specifications provided by agency analysts. It is often possible to simulate alternatives, such as a change in the amount of an allowed deduction, simply by resetting a parameter switch in the model; in other cases, the model code must be rewritten to accommodate one or more features of the proposed change. If the model simulates behavioral responses to program changes, such as the impact on labor supply of changing the AFDC benefit amount or the subsequent feedback effects of a labor supply response, these components of the model would be the next steps. In practice, however, because the complexities of simulating behavioral responses and second-round effects are an order of magnitude greater than the previous steps, these capabilities are infrequently or only very crudely implemented in today's microsimulation models.
The final step involves tabulating the output for the baseline program and the various proposed alternatives that were simulated. Typical output includes tables of "gainers and losers" under each alternative in comparison with the baseline file. At that time, an analyst may also make "out-of-model adjustments" to the results. For example, analysts in the Office of the Assistant Secretary for Planning and Evaluation (ASPE) project TRIM2 output to the appropriate future years at this stage, instead of using the TRIM2 set of static aging routines. Another type of out-of-model adjustment could include changing the overall level of participation for one or more simulations on the basis of the analyst's assumptions about behavioral response.
DEVELOPMENT OF MICROSIMULATION MODELING FOR POLICY ANALYSIS
Although microsimulation techniques were originally developed by social science researchers, they have been used much less for basic research purposes than for policy analysis. There are some exceptions, notably the field of family demography, which has a 25-year history of development and application of microsimulation models. (We review this history in Chapter 11, as part of a discussion of the potential of microsimulation models to make useful contributions to a broad range of social science research problems that, in turn, could
well improve their utility for policy analysis.) In this section we give a capsule overview of the key stages in the development of microsimulation modeling as a tool for social welfare policy analysis on the part of the federal government. Our summary is not exhaustive: we cover some of the major models and their applications, which are referred to throughout our report.1
Before beginning this overview, we note that microsimulation techniques have also had widespread application for policy analysis on the part of state governments and on the part of government agencies abroad. A recent survey indicated that half the states use microsimulation models, in addition to other techniques, to estimate revenues and assess the impact of tax policy changes (Peat Marwick, 1989). State agencies have also used microsimulation models to analyze a range of social welfare policies.
With regard to microsimulation modeling efforts in other countries, 2 the (preunification) Federal Republic of Germany became active in model development in the early 1970s. German microsimulation models include the Wohngeldmodell, first developed in 1974-1975, which has been used extensively to analyze housing allowance schemes; the Frankfurt model, which was developed over the 1970s primarily for academic research use but also provided an analysis of alternative public pension policies; the BAFPLAN model, first developed in 1976-1977, which has been in continuous use since that time to evaluate public training assistance programs; and the APF model, which was developed in 1984-1985 to analyze family allowance plans.
Canada has also been an active developer and user of microsimulation models since the late 1960s. Recently, Statistics Canada developed the Social Policy Simulation Database/Model (SPSD/M), which is a static model of Canadian tax and transfer programs. The model, which is publicly distributed, is implemented on a personal computer and designed for ease of use by nontechnical analysts. Research and public interest groups, as well as government agencies, have worked with the model.
In Sweden, the Industrial Institute for Economic and Social Research developed a microsimulation model of the corporate sector, the MOSES model, in the mid- to late 1970s. Recently, Statistics Sweden has begun exploring the development of microsimulation models for analysis of public policies affecting
the household sector. The Hungarian Statistical Office recently completed development of a multipurpose microsimulation model, which to date has been used to analyze income tax and family allowance programs. The model is also intended to generate databases, by means of correcting and merging survey data, that will enable the office to cut back on the frequency of specific surveys. In addition, the Hungarian Statistical Office is planning to develop microsimulation models of the health care system and the corporate sector.
Finally, most members of the Organization for Economic Cooperation and Development (OECD), even if they are not heavily involved in the use of microsimulation techniques otherwise, maintain tax policy microsimulation models. Recently, a working group of the OECD began a program to coordinate development of improved microsimulation models for public policy analysis on the part of interested member countries.
Origins: 1950s to mid-1960s
In the 1950s, Guy Orcutt, an economist, developed the concept of analyzing national economic and social policies by simulating behavior for individual decision units, using microlevel databases and probabilistic (Monte Carlo) techniques (Orcutt, 1957; Orcutt et al., 1961).3 Such "microsimulation" models, in his view, would provide a richer and much more realistic tool for policy analysis than could the macroeconomic models of the national economy on which he had previously worked. Orcutt concentrated his development efforts on fully dynamic models. In the early 1960s, researchers at the Brookings Institution, encouraged by analysts in the Treasury Department, developed a static, cross-sectional microsimulation model to analyze the impact of changes in federal tax policies on revenues and various categories of taxpayers (Pechman, 1965).
Development of Static Models: Late 1960s to Early 1980s
In the late 1960s, Gail Wilensky and her colleagues on the staff of President Johnson's Commission on Income Maintenance Programs (the Heineman Commission) developed the first federally sponsored operational microsimulation model for analyzing alternative welfare policies. The Reforms in Income Maintenance (RIM) model was used to simulate many variations of a negative income tax and other welfare reform plans (McClung, 1970; Wilensky, 1970). Subsequently, RIM was used by the U.S. Department of Health, Education, and
Welfare (HEW) and the U.S. Office of Economic Opportunity to simulate welfare policy options considered during the debate on President Nixon's Family Assistance Plan.
In the early 1970s, analysts and programmers at the Urban Institute, under contract to HEW, attempted to rewrite RIM to make the model more user friendly and cost-effective. Failing to do so, they abandoned the effort and developed the Transfer Income Model, the first working version of which was released in 1973. TRIM, like RIM, was a cross-sectional model that used static aging techniques. The Office of the Assistant Secretary for Planning and Evaluation (ASPE) in HEW provided support to the Urban Institute to maintain and further develop TRIM and commissioned simulations of a number of welfare reform schemes. At about the same time, Social Security Administration staff developed their own in-house static microsimulation model, the Simulated Tax and Transfer System (STATS) model, to analyze the distributional impacts of alternative social security policies. STATS was used to produce estimates for the debate that led to the enactment of the Supplemental Security Income program for the elderly and disabled in 1972.
In 1974 analysts involved in the development of TRIM left the Urban Institute to join Mathematica Policy Research, Inc., where they developed a new static tax and transfer policy model based on TRIM: the Micro Analysis of Transfers to Households model. During 1974-1976, the MATH model was used, under contract to the Food and Nutrition Service of the U.S. Department of Agriculture, to analyze more than 200 variations of proposed reforms to the food stamp program; the effort culminated in the 1977 Food Stamp Reform Act. Subsequently, in the mid- to late 1970s, Mathematica administered a subscription service for the MATH model: sponsor agencies funded a program of maintenance, development, documentation, training, and dissemination for the model. At its height, agencies in the Departments of Agriculture, Energy, Health and Human Services, Labor, and Treasury, along with the Congressional Budget Office (CBO) and the Congressional Research Service, were subscribers, receiving regular updates of the model, database, and documentation.
Also in the early to mid-1970s, analysts at the Brookings Institution continued their efforts to apply microsimulation techniques to analyzing the aggregate incidence, by income class, of the entire U.S. tax system—federal, state, and local—as well as the tax burden implications of proposed changes in the tax system. Their databases—originally referred to as MERGE files (see Minarik, 1980)—represented statistical matches of samples of income tax returns from the Internal Revenue Service (IRS) with household surveys such as the March CPS. Meanwhile, the Treasury Department's Office of Tax Analysis brought a version of the tax policy model in-house and undertook continuing development, maintenance, and application of the model for policy analysis purposes. A little later, in the late 1970s and early 1980s, ASPE organized a TRIM users' forum and then a broader based microsimulation users' forum to
provide an opportunity for agency and contractor analysts to exchange ideas and information related to the use of microsimulation modeling for policy analysis purposes.
Early in 1977 the Carter administration undertook a major effort to design and enact a welfare reform program that included job training and employment components, known as the Program for Better Jobs and Income (PBJI). Both the Departments of Labor (DOL) and Health and Human Services (HHS) were involved: DOL sponsored the development of modules in the MATH model to simulate acceptance rates for a public service jobs program and labor supply responses to the proposed welfare changes; ASPE staff in HHS constructed their own model to simulate PBJI. The model, known as KGB after the authors' last names (Kasten-Greenberg-Betson), included a jobs component and simulation of labor supply responses (see Betson, Greenberg, and Kasten, 1980). The development of KGB was completed on a crash basis in 5 weeks. Both MATH and KGB estimates were used in the policy process as the two departments hammered out a proposal satisfactory to President Carter. However, the initiative failed in Congress, and by 1980 the KGB developers had left ASPE, and the model had fallen into disuse.
Also in the late 1970s, the Urban Institute received support from ASPE to redesign TRIM in order to effect major improvements in its operating efficiency and usability. (A primary reason for the development of KGB had been the belief that TRIM, as then implemented, was too difficult and costly to modify.) Somewhat later, the MATH model was also redesigned to improve computational efficiency.
Meanwhile, in the late 1970s and early 1980s, proposals were floated to develop comprehensive microsimulation models for the health care sector. However, except for limited capabilities to simulate the Medicaid program that were added to models such as MATH, health care policy analysts were using cell-based models. For example, ASPE used the Health Financing Model, which is primarily cell based, to cost out alternative national health insurance schemes considered by the Carter administration.
Development of Dynamic Models: 1970s to Early 1980s
Dynamic microsimulation models proved more difficult to develop and apply than their static cousins. Orcutt and his colleagues at the Urban Institute received funding in 1969 to develop the Dynamic Simulation of Income Model (DYNASIM), the first version of which was completed in 1975. DYNASIM was designed to simulate a wide range of demographic and socioeconomic life events and their interactions with government policies over time. Interest waned in using the model to analyze welfare policies, but interest grew in
applying DYNASIM to generate earnings histories and examine the long-range implications of social security policies.
In the late 1970s, the Urban Institute received funding from DOL and CBO to effect a major redesign of DYNASIM. The goals of the redesign were twofold: first, to make the model much cheaper to run with large samples and able to be moved to government computers; second, to focus the model on retirement issues, including improving labor force histories and the social security module and incorporating a private pension simulation module. Over a 3-month period in 1980, Lewin/ICF, Inc, developed a primitive version of the Pension and Retirement Income Simulation Model (PRISM) for the President's Commission on Pension Policy, to analyze the effects of alternative ways of integrating public and private pension systems.
Retreats and Advances: The 1980s
During the early 1980s, use of the static tax and transfer program models, such as TRIM2 and MATH, languished, because of constrained resources for model development and use, and because the Reagan administration was primarily concerned with cutting back welfare programs and entitlements. This period saw the development of benefit-calculator models, which run on samples of welfare program administrative records and can readily simulate changes that reduce eligibility and benefits. However, ASPE took advantage of the improved cost-efficiency of TRIM2 to have the Urban Institute improve a number of modules, including a module to allocate yearly income and employment values to monthly values (to match program eligibility rules) and modules to simulate state and federal income taxes. CBO also brought TRIM2 in-house and used the model for studies that helped pave the way for later welfare reform initiatives (see, e.g., U.S. House of Representatives, 1985). The MATH model was also used for a study sponsored by the Congressional Research Service to sort out the effects of the 1981 Omnibus Budget Reconciliation Act cutbacks in welfare programs from the effects of the recession on the low-income population (Citro and Beebout, 1984).
In the early to mid-1980s, PRISM was further refined and enhanced (e.g., a macroeconomic model link was added in 1982-1983). It was used for a number of analyses of retirement income, including simulation of the effects of alternative private pension vesting plans and of the major changes to the social security program that were enacted under the Reagan administration. At the same time, DYNASIM2 was used to analyze the implications of retirement policy changes, including legislation to limit mandatory retirement and the 1983 Social Security Act Amendments. DYNASIM2 was also used for other studies, such as the effects on welfare program costs of alternative rates of teenage childbearing.
In the mid-1980s, Lewin/ICF, Inc., developed the Household Income and
Tax Simulation Model (HITSM), as a proprietary model—an indicator, perhaps, of reviving interest in microsimulation techniques for policy analysis. Several years later, the same firm developed the Health Benefits Simulation Model (HBSM), the first major microsimulation model for analyzing alternative health care policies as they affect the household sector. The primary database for HBSM is the 1980 National Medical Care Utilization and Expenditures Survey. In the mid-1980s, the Brookings Institution, with foundation and HHS support, worked with Lewin/ICF, Inc., to develop a submodel in PRISM to analyze alternative schemes for financing long-term health care of the elderly. Subsequently, with funding from ASPE, the long-term care financing submodel was revised and developed as a public-use model.
Over the 1980-1988 period, ASPE made a major investment in a model to simulate the second-round effects of policy changes, such as the effects on regional employment of welfare or tax reform: the Multi-Regional Policy Impact Simulation (MRPIS) model. It was developed by the Social Welfare Research Institute at Boston College and included microsimulation, input-output, and cell-based components. Meanwhile, the Office of Tax Analysis and the congressional Joint Committee on Taxation continued to develop and maintain tax policy models that received extraordinarily heavy use in the policy debate culminating in the Tax Reform Act of 1986. At the same time, ASPE analysts used TRIM2 to analyze the distributional effects of tax reform proposals. In the late 1980s, TRIM2 was used heavily in the policy debate that led to the Family Support Act of 1988. Subsequently, TRIM2 was used to evaluate alternative child care tax credit schemes. The Medicaid module in TRIM2 was revised and applied to analyze the impact of expanding Medicaid coverage of the low-income population.
Throughout the 1980s, the MATH model was used to simulate a range of proposals to modify the food stamp program. For the fiscal 1984 budget, both MATH and the food stamp benefit-calculator model were used to simulate alternative proposals to simplify the program rules while minimizing the effects on current recipients. Those two criteria proved impossible to satisfy at the same time, so that the effort failed in Congress (see Carlson, 1989). Subsequently, MATH was used to simulate the impact of changes in the minimum wage on the food stamp program and the scale of the program if there had been no changes since 1980.
In the mid- to late 1980s, the Food and Nutrition Service (FNS) supported efforts by Mathematica Policy Research, Inc., to investigate the potential for using data from the new Survey of Income and Program Participation (SIPP) as a microsimulation model database. Mathematica built the FOSTERS (Food Stamp Eligibility Routines) model for the food stamp program with data from the 1984 and 1985 SIPP panels. This model was used to simulate congressional proposals to raise the asset value of vehicles that program recipients can own
and still remain eligible; the high price tag estimated for this change deterred congressional action.
Finally, in the late 1980s, both ASPE and FNS sponsored studies to evaluate aspects of the TRIM2 and MATH models, including the study by this panel.
In this once-over of the history of microsimulation modeling in the United States, we have by no means captured the full extent of the contribution of microsimulation models to the policy analysis function; we have glossed over many important policy uses and also many models developed for more limited purposes.4 However, we hope to have conveyed the flavor of the wide-ranging and important uses that microsimulation models have had in the policy process over the past 20 years.
ROLE AND CURRENT STATUS OF MICROSIMULATION: FINDINGS
On the basis of our review of the experience to date in using microsimulation models for policy analysis, and considering the comparative merits of other kinds of models, we present our general findings about the current role and status of the microsimulation approach. The rest of Part II provides findings and recommendations about the various aspects of microsimulation models that we have addressed—beginning with their data inputs and concluding with their potential uses for basic social science research.
The microsimulation modeling approach to estimating the impact of proposed changes in government programs offers important conceptual and operational benefits to the policy process. Microsimulation models operate at the level of the individual decision unit, by taking into account the diverse characteristics and circumstances of the relevant population, whether it be low-income families, taxpayers, or health care providers. They obtain input from microlevel databases of individual records, mimic the way in which government programs apply to the individuals described in those records, and maintain the outputs of simulated variables for current and alternative programs on each of the individual records. As a result, the models have the capability to respond to important information needs of the policy process:
First, microsimulation models can simulate the effects of very fine-grained as well as broader policy changes. For example, a microsimulation tax model can estimate the effects of a proposed change to the tax code that applies only to taxpayers with certain kinds or levels of income or expenses, as well a proposed increase or decrease in tax rates across-the-board.
Second, microsimulation models can simulate the impact of proposed changes that involve complicated interactions among more than one government program. For example, a microsimulation model of income support programs can simulate the net effects of a proposed change to AFDC that also alters the calculation of food stamp benefits.
Third, microsimulation models can simulate the effects of proposed changes on subgroups of the population, in addition to aggregate estimates of program costs and caseloads. For example, a microsimulation model of physicians' services can simulate the effects of changes in Medicare fee schedules on different types of medical specialties and geographic areas; or a microsimulation model of health insurance programs can provide detailed distributional information about the effects of changes in insurance coverage and benefits on specific types of families.
Besides offering flexibility in examining alternative programs, microsimulation models—in common with many other modeling techniques—provide a framework that ensures consistency of estimates across a wide range of proposals.
The orientation of microsimulation models to the individual decision unit is conceptually attractive. It is, after all, individual parents who decide whether or not to apply for AFDC or to take or quit a job; it is individual taxpayers who decide to itemize or not to itemize deductions or to move assets from taxable to nontaxable instruments in response to tax law changes; it is individual doctors who decide to increase or decrease the number of diagnostic tests of patients in light of government cost reimbursement policies. Although individuals may behave like other parents or taxpayers or doctors with similar characteristics, accurately portraying their individual circumstances and the factors entering into their decisions appears crucial to analyzing the complex issues in social policy.
We conclude that no other type of model can match microsimulation in its potential for flexible, fine-grained analysis of proposed policy changes. Large-scale macroeconomic models, which are designed to estimate the aggregate effects of policy and program changes, such as the implications for the deficit and for national economic growth of a President's proposed budget, rival microsimulation models in size and complexity. However, these models use entirely different data and modeling techniques: evaluation of systems of simultaneous equations estimated with aggregate time series (such as the relationship of public to private spending and investment). Their outputs are for aggregates, such as all consumers in the nation (or in a region or state), and they are in no way able to estimate the impact of changes in particular programs
on particular groups, such as the effects on the working poor of mandating the AFDC unemployed-parent program in all states.
Simpler macrolevel models, which estimate a single equation on the basis of a few aggregate time series, are often applicable to analyses of particular programs. For example, such a model might estimate growth in AFDC costs and caseloads on the basis of changes in unemployment, inflation, and average benefit level. However, single-equation time-series models are very limited in scope and do not provide any real capability for analyzing complex program alternatives or for sorting out the detailed effects of program changes.
Cell-based models, which develop estimates for subgroups or "cells" that make up the population of interest (for example, an AFDC model might comprise cells for case type by state), can provide more detailed information on policy effects than macroeconomic models, but they, too, are limited in comparison with microsimulation models. Cell-based models, whether they contain several thousand or only a handful of cells, make the critical assumption that all elements within a subgroup will behave in the same way. Should a policy change affect members within cells in different ways, or should policy makers want information for different groupings, a cell-based model must be rebuilt.
Microeconometric multiple-regression models, which produce estimates of the impact of a set of variables on some aspect of individual economic behavior, resemble microsimulation models in their use of microlevel data and their ability to provide disaggregated as well as aggregated results. For example, regression models of welfare program participation—which might include explanatory variables for family size and type; family income and expected benefit level; age, race, and sex of family head; and other characteristics—can be run on a microlevel database to produce participation probabilities for individual families. In turn, these probabilities can be aggregated for subgroups or for the total population. However, the key variable for analyzing the impact of a proposed program change with such a model, namely, expected benefit level, must be supplied by some other means. Indeed, some microsimulation models use a regression-based approach to determine program participation, after they have calculated program eligibility and expected benefits, by applying the detailed program operating rules to each family's record.
A Range of Policy Analysis Tools
Microsimulation models are by no means the only useful tool for policy analysis. Indeed, the policy analysis community benefits from having available a wide range of modeling tools to answer a variety of questions and provide alternative perspectives. As we just noted, microsimulation models are distinguished by their capability for fine-grained analysis of proposed policy changes; however, this capability is not always needed. There are many policy issues and questions for which it is neither necessary nor advisable to crank up
a full-scale microsimulation model, particularly in view of the costs of large databases and complex modeling routines.
When policy makers are primarily concerned with the impact of proposed policy changes at an aggregate level—such as the interrelationships among federal spending and revenues, on one hand, and economic aggregates such as the inflation rate, on the other—it seems perfectly appropriate and cost-effective to use aggregate modeling techniques. And in cases for which limited distributional information is desired, it may be most cost-effective to carry out the estimation with a cell-based spreadsheet model.
However, in those many instances in which the policy process is likely to require detailed information for each of a number of proposed alternatives, microsimulation is potentially the only approach that can satisfy the analytic needs. Hence, we believe that microsimulation models merit continued support on the part of federal agencies as an important, if not the only, tool for estimating the impact of proposed program changes.
The capability for detailed analysis provided by microsimulation models comes at a price. Although we support the use of microsimulation models for policy analysis, it is important to recognize that the complex nature of such models entails costs. Microsimulation models are highly complex for a number of reasons: they typically require large amounts of data; they must model many complex features of government programs; and they are pressed to provide more and more elaborately detailed information.
Because of their complexity, microsimulation models can be resource intensive to develop and apply, and difficult to understand and evaluate. Moreover, because microsimulation models must usually meld together a variety of data and research results of varying degrees of quality and, in the process, make many unsupported assumptions, there are potentially serious implications for the quality of the resulting estimates. And there are likely to be compounding effects of the errors introduced at each of the many steps in the simulation process.
Indeed, we are gravely concerned that the history of microsimulation model development to date has witnessed too many instances in which costs have proved disproportionately large in comparison with benefits. In our view, the tendency to pile complexity upon complexity has all too often led to a situation in which the modeling task—whether it be for development or application—incurs added time and cost; in which it is difficult for the analyst, let alone the decision maker, to evaluate the quality of the output; and in which the model, instead of providing a capability for timely, flexible response to changing policy needs, becomes sluggish and inflexible in operation.
A typical response in the past to the problems posed by the complexity of
microsimulation models has been to pare back the capabilities of the model or to focus new development on the model's ''accounting" functions that mimic program rules and leave aside other, more difficult aspects, such as modeling behavioral response. However understandable in many instances, these kinds of choices limit the usefulness of the models for the policy debate.
In our review, we accorded high priority to identifying strategies with the potential to improve the quality, flexibility, accessibility, and overall cost-effectiveness of the next generation of microsimulation models without compromising their ability to provide the fine-grained policy information that is their prime reason for being. We believe that such strategies exist: for example, new computer technologies are very promising in this regard. An important implication of our recommendations is that policy analysis agencies must be willing, over the next few years, to allocate a higher percentage of available resources to investment in microsimulation models rather than to current applications (unless, of course, overall budgets can be increased). As we discuss further below, investments are urgently needed to improve the data, research, and computational inputs to models. Investment is even more urgently needed to evaluate the quality of model outputs and to build capabilities into models that will facilitate systematic validation in the future.
The overall uncertainty of the estimates produced by existing microsimulation models is virtually unknown at this time. Although in theory the microsimulation models in use today provide better estimates of distributional impacts and at least as good estimates of overall costs and caseloads as other kinds of models, it is not known if this theory is true in fact. There is very little evidence with which to assess the validity of microsimulation model results, that is, how well they compare with actual policy outcomes. In addition, there are almost no measures available of the degree of uncertainty or variability in the estimates or the major sources of variation. However, we suspect that the level of uncertainty, given the large number and varying quality of microsimulation model inputs, is high.
We believe that analysts and policy makers can have considerable confidence in the quality of the computer models per se, that is, in the accuracy with which the computer code replicates the model specifications. Microsimulation modelers have long made a practice of devoting time and resources to computer model verification, through such activities as examining individual test cases to determine that the code to simulate a program change is working properly. Another check against egregious errors in the computer code is the long-standing practice of analysts from various agencies, in both Congress and the executive
branch, to get together periodically over the course of developing major legislation to compare models' outputs and to search vigorously for explanations of discrepancies.
However, very little systematic study has been conducted of the quality of the estimates produced by microsimulation models during their 20-year history of use in the policy process. The dearth of analysis includes external validation studies that compare model output with measures of truth; internal validation studies that assess the sensitivity of model results to the input data, the specifications for individual modules and their interactions, and other components of the simulation process; and studies that assess the variance of model estimates due to sampling error in the primary database and other sources.
Microsimulation models are not alone in lacking systematic validation of their outputs. As we note in Chapter 3, information about the uncertainty in estimates of the effects of proposed policy changes is largely absent from the policy debate, regardless of what type of modeling tool has been used. The conditional nature of almost all policy analyses makes the task of validation difficult. The many different factors involved in most policy analyses are also a hindrance to validation, as is the resistance of decision makers to information about uncertainty. Given the highly complex nature of microsimulation models, it is perhaps not surprising that the validation literature for their outputs is so scant. Yet we believe strongly that the impediments to model evaluation can and must be overcome. Otherwise, policy makers will continue to make decisions based on numbers that may be of highly variable quality, and the agencies that provide support to decision makers will lack information on the most cost-effective ways to invest in improved microsimulation models for the future. Given the high costs of microsimulation model development, it is particularly important to have good information on which to base investment decisions. We are encouraged that, recently, both ASPE and FNS have supported major microsimulation model evaluation studies, in addition to that undertaken by this panel (see, e.g., Doyle and Trippe, 1989; Kormendi and Meguire, 1988); as discussed further in Chapter 9, much more work needs to be done.
There are serious questions about the adequacy of the data sources used to construct microsimulation model databases. Much of the computer code and sizable fractions of staff resources for current microsimulation models are devoted to reprocessing and manipulating available input data, not only to produce databases that are more efficient to process, but also to try to compensate for deficiencies in data content and quality. Examples of important deficiencies for modeling income support programs include underreporting of income receipt and undercoverage of population subgroups, particularly low-income minorities,
in household surveys such as the March CPS. SIPP was designed to address some of these problems, but it does not currently have sufficient sample size and is not timely enough to be a satisfactory substitute. For data on health care, there are serious gaps, difficulties in linking available data sources together, and problems with timeliness. For data on retirement income and tax policy, impediments to linking survey and administrative data cause serious problems for models. In our view, improvements in data quality, together with a shift in the data production function to place more responsibility for producing useful databases on the originating agencies, represent high priorities that promise substantial dividends in terms of reduced cost and improved relevance and quality of model estimates. Again, although we here emphasize the linkage of data quality and microsimulation modeling, we point out that all analytical approaches to the development of policy estimates rise and fall with the quality of the data.
There are serious questions about the underlying base of research knowledge that supports modeling individual behavior and other model capabilities. Although predicated on the desirability of simulating individual decisions as they are affected by and affect government programs, current microsimulation models are very limited in this regard. This statement applies not only to models that are avowedly "benefit calculators," such as the administrative records-based models of AFDC and food stamp recipients, but also to models that simulate program effects for the broad population. Except for the basic decision of whether to participate in a new or modified program, the models rarely simulate other behavioral responses, such as the response of income support beneficiaries to work incentives. They also rarely simulate second-round effects of a policy change, such as the impact of raising or lowering health care benefits on consumption of medical services and, consequently, on employment in the health care sector in relation to the rest of the economy.
An important factor in this paucity of behavioral components in microsimulation models in addition to high cost and complexity is the weakness of the underlying research knowledge base. There are no generally agreed-upon estimates of key behavioral relationships, and the form of the available parameter estimates is often not readily suited to implementation in a microsimulation context. We do not anticipate rapid progress in ameliorating this situation, given constrained budgets for research and aspects of the academic research culture that militate against the kinds of research that can most benefit the policy analysis process. However, we offer a number of recommendations for the agencies to spur the production of policy-relevant research. We also recommend model design and development practices that we believe are most cost-effective for incorporating new research knowledge.
The adequacy of the computer hardware and software technologies used to implement current microsimulation models is questionable. The major social welfare policy microsimulation models that are widely used today are designed for mainframe, batch-oriented computing environments that represent yesterday's technology and limit the models in important ways. Computing costs for a single simulation run are much lower for today's models than for the models of the 1960s and 1970s. However, other costs, such as the combined staff and computer costs of rewriting portions of the model code—often needed to simulate innovative policy proposals—remain high. The current computing environment for microsimulation modeling discourages experimentation, either substantively or for validation purposes, and puts barriers in the way of direct access by analysts to the models.
Some model developers have explored the potential of microcomputer technology to support more flexible and accessible models with promising results. Other hardware configurations, such as some combination of linked micro and mainframe computers, may also provide improved capabilities. New developments in software, such as graphical user interfaces and computer-assisted tools for design of software, are also very promising. We strongly recommend that agencies position themselves to build the next generation of microsimulation models around new computer hardware and software technologies that can enhance the cost-effectiveness of this important class of policy analysis tools.
Microsimulation Modeling Community
The current structure of the microsimulation modeling community is costly. Several aspects of the interrelationships among the policy analysis agencies that use microsimulation models, their modeling contractors, and academic researchers are troubling. One set of problems stems from the highly decentralized and fragmented nature of policy analysis in the federal government. While having positive features, the involvement of many different agencies frequently imposes costs of duplication of effort and often isolates groups of analysts who could benefit from a higher level of communication and an exchange of ideas and viewpoints. Our suggestions of useful ways to enhance interagency cooperation are oriented to microsimulation, although the problems in this area also affect policy analysis based on other types of modeling tools.
Another set of problems stems from the very circumscribed nature of the community that is actively involved in developing and applying microsimulation models. As in the past, there are today a handful of private firms that operate the major microsimulation models for social welfare programs on behalf of their federal agency clients. The agencies, which typically have only a few or no staff who are able to use the models themselves, are very dependent on
their contractors for support. In our observation, these firms have performed responsibly and capably in responding to agencies' needs. Nonetheless, we believe that it would be beneficial to expand access and use of the models on the part of agency analysts. It would also be useful to expand access and use of the models on the part of academic researchers, who in most disciplines have played a relatively minor role heretofore in applying, refining, and evaluating this class of models. Having more people who are knowledgeable about microsimulation models and adept in using them can only help in the development of improved models and in the vital process of validating model results.
In sum, we believe that microsimulation models are important to the policy process, and we anticipate that the need for the kinds of detailed estimates that they can best generate will grow in future years. However, because of the lack of evidence with which to assess the performance of the current models and the limitations of available databases and research knowledge, we cannot responsibly advocate substantial investments that would expand the capabilities of existing models in any specific direction. We strongly support allocating sufficient resources to the current models to evaluate their capabilities, maintain them, and improve them as appropriate and cost-effective. The validation and maintenance functions, together with incremental improvement, are critical to the ultimate objective of developing a new generation of microsimulation models after investments in data, research, and computing technology have borne fruit. Maintaining a cadre of knowledgeable and experienced users and producers of the current models will enable new models to be built much more expeditiously and efficiently. We urge the relevant agencies to make the investments that are required to ensure that a new generation of models is developed in a timely manner to meet the policy needs of the future.