Click for next page ( 133


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 132
Development of Projection Models As the concern about retirement income security and the costs to provide it increases over the next few years, decision makers will look to projection models to estimate the likely costs and other effects of alternative policy proposals. Such models will require good data and a solid base of research knowledge with which to project demographic and economic trends and behavioral responses to policy changes and other factors. Outputs from analytical models of the behavior of individuals and employers that receive general consensus should be inputs to projection models. Otherwise, when policy proposals are expected to elicit a behavioral response, projection models will fall short in estimating the likely effects. In the preceding two chapters, we identified critical deficiencies in both data and research. As we discuss below, available projection models are also deficient in many respects. The question is how to allocate limited resources for developing better capabilities for retirement-income-related policy analysis. As we stress throughout this report, the U.S. Department of Labor (DOL) and other relevant agencies should give priority to effecting needed improve- ments in data and analytical research before making significant investments in broad-based projection models. Such investments will be misplaced until there is better understanding of key behaviors such as employer decisions about hiring and compensation policies and individual decisions about savings and consump- tion and better data with which to project trends in those behaviors and other factors. Until efforts to improve data and research bear fruit, we recommend that agencies make limited investments in special-purpose models that use the best available data to answer policy makers' specific questions. At the same time, agencies should consider the kinds of broad-based projec 132

OCR for page 132
DEVELOPMENT OF PROJECTION MODELS 133 lion modeling capabilities that it will be important to develop in the future. In this chapter, we address: types of projection models that could be useful; the dimensions of models that should be considered in planning for improved projec- tion capabilities; the capabilities and limitations of existing models; long-term issues in developing models of employers and employer behavior; long-term issues in developing improved individual-level microsimulation models; issues in developing validation capabilities in new and current models; and strategies for projection modeling in the near term with existing models or one-time spe- cial-purpose models (including improvements to the model maintained by the Social Security Office of the Actuary). CURRENT MODELS AND THEIR USES Types of Models Projection models vary widely in scope, complexity, and modeling strategy. Some of the major types are time-series models, cell-based models, microsimu- lation models, macroeconomic models, and computable general equilibrium mod- els. Some models incorporate multiple approaches: for example, in the Macro- economic-Demographic Model (MDM), a macroeconomic growth model interacts with cell-based models of population growth, family formation, the labor market, and other model components. Other models reflect primarily one approach but make use of the outputs of other kinds of techniques; see Appendix C for descriptions of some models. Times-Series Models Time-series models project aggregate historical data into the future on the basis of a function, which can be simple or complex, of the behavior of the data in previous periods. For example, future labor force partici- pation rates for older men can be projected to continue to decline on the basis of trend data from 1950 to 1995. However, the longer the projection period, the more hazardous it is to assume that past trends will continue unchanged into the future. Indeed, for the example just noted, there is currently debate about whether the trend toward early retirement among men has peaked. Cell-Based Models Cell-based models divide the population into groups, such as 5-year age categories by sex, and develop separate projections for each group for example, separate projections of life expectancies, retirement age probabilities, and so forth. The separate projections may be based on time-series analysis, modified by assumptions about the likely behavioral effects of policy changes (e.g., the effects on retirement age of reductions in initial Social Security benefits). Because a cell-based model reflects differences among groups, the aggregate outcome it projects may differ greatly from the outcome of a simple time-series projection for the population as a whole.

OCR for page 132
34 ASSESSING POLICIES FOR RETIREMENT INCOME The Office of the Actuary in the Social Security Administration (SSA) uses cell-based techniques to develop 75-year projections of the balance in the trust funds. Such techniques are also used in actuarial models of employer pension fund balances and in models used by the Census Bureau, SSA, and others to develop population projections. Microsimulation Models Microsimulation models use individual records for large samples of the population, estimating outcomes for each individual on the basis of that person's characteristics and aggregating the results. Such mod- els usually employ stochastic or probabilistic methods to generate distributions for such outcomes as retirement decisions. DYNASIM2 and PRISM are two dynamic microsimulation models that have been used to evaluate alternative Social Security and employer pension policies. They use dynamic aging techniques to project their samples forward in time. In dynamic aging, characteristics of persons in the sample are altered year by year through the application of transition probabilities to simulate who in the sample will live, die, marry, divorce, stay on the same job, take another job, retire, and so on. Policy simulations are then performed on the samples for 1 or more years. The transition probabilities may come from a variety of sources, including time- series analysis (e.g., for life expectancy) and behavioral analysis (using simple or complex analytical models). Thus, for example, a behavioral equation in DYNASIM2 relates labor force participation to age, race, sex, education, disabil- ity, marital status, presence of children, and spouse's earnings and includes a complex error term by which labor force participation changes for an individual are linked across time and with changes in hours and wages. Another microsimulation projection technique is called static aging, in which the characteristics of the sample are altered by adjusting the sample weights to reflect the expected distributions in the projection year before conducting policy simulations. This method is typically employed for short-range projections (e.g., of tax liabilities or welfare program participants in the next 5 years) and not for the longer range projections that are often required for retirement-income-related policy modeling. Microsimulation models can estimate the effects of more detailed policy changes for a larger number of groups than is possible with cell-based models. In addition, Burtless (1996) argues that microsimulation provides an organizing framework. It requires model developers and users to think through all of the behavioral links and interactions. It also helps them identify gaps in research knowledge and data and set priorities for filling in key gaps. However, micro- simulation models have costs as well as benefits, which must be evaluated vis-a- vis other modeling approaches in the context of users' information needs. In particular, they tend to be complex and difficult to use. Both microsimulation and cell-based models can be developed for organiza- tions (e.g., employers) as well as for individuals. However, with the exception of

OCR for page 132
DEVELOPMENT OF PROJECTION MODELS 135 the Pension Insurance Management System (PIMS) model recently developed for the Pension Benefit Guaranty Corporation (PBGC) (see Appendix C), we know of no policy projection models of retirement-income-related behavior of employers. Macroeconomic Forecasting Models Macroeconomic forecasting models are used to make projections for the U.S. economy. They use systems of simul- taneous equations, estimated with historical time series, to forecast the effects of aggregate factors, such as rising inflation or changes in the federal budget deficit, on aggregate outcomes, such as gross national product or unemployment. Computable General Equilibrium (CGE) Models CGE models estimate the longer run effects of proposed policy changes on prices and quantities in markets, taking into account behavioral responses and feedback effects between supply and demand. For example, an increase in marginal tax rates may decrease work effort, thereby leading to reduced labor supply, thereby leading to higher wage rates and increased labor supply until a new equilibrium is reached. As generally implemented, CGE models are macro rather than micro in nature, in that they are not usually disaggregated on many dimensions. However, unlike macroeco- nomic forecasting models, CGE models are based on explicit theories of the behavior of households and employers rather than on relationships among aggre- gates. Dimensions of Models There are three key dimensions on which to evaluate projection models: the degree to which the model provides accurate estimates of policy out- comes, in that the estimates approximate what would occur if a proposed policy change were implemented; the amount of information provided in the model output on the uncer- tainty of the model estimates and their sensitivity to key assumptions; and the degree to which the model incorporates best current professional judg- ment about the underlying behavior for example, the best judgment about the appropriate functional form and parameter values with which to estimate changes in personal savings in response to changes in employer pensions or Social Secu- rity. Obviously, it is highly desirable that models be accurate, provide estimates of uncertainty and sensitivity to key assumptions, and incorporate best profes- sional judgment (or be able to use alternative specifications to reflect important disagreements). But determining the accuracy of model outputs is a difficult task at best. Also, models have to date rarely included the capabilities with which to

OCR for page 132
136 ASSESSING POLICIES FOR RETIREMENT INCOME estimate uncertainty, conduct sensitivity analyses, or incorporate alternative model specifications; consequently, little model validation work has been done. With recent advances in computing technology and statistical methods, it is now possible to effect significant improvements in model capabilities in these areas (see "Validation" below). Other dimensions of projection models include: scope, in terms of the types of policies that a model can simulate and the kinds of outcome measures for which it can provide information; detail, in terms of how well a model can represent all of the provisions of current law and estimate the effects of changes to one or more provisions; extent of disaggregation of model outputs, that is, the extent to which the model can estimate the distributions of outcomes for groups of interest, variously defined, in addition to average tendencies; time period, whether the model can develop projections for 5 years or 75 years; extent of feedback loops, which permit estimating not only the first-order effects of a policy change, but also the second-order effects, taking into account behavioral responses to the new policy; ability to link to other models, either by supplying outputs for other mod- els to use as inputs or by accepting other models' outputs as inputs; amount of elapsed time that is required to obtain outputs from the model on one or more policy options, including options that are close variants of current policies and options that differ markedly in one or more respects; extent of openness and transparency, that is, how usable and understand- able the model is to people other than the developers; ease of use and modification as needed; and extent of portability, from one computing hardware and software environ- ment to another. The first five dimensions listed above scope, detail, disaggregation, time period of projections, and feedback loops have to do with the breadth and depth of model capabilities. Other things being equal, it is desirable to have a model that can simulate a broad range of policy options and, for each option or combi- nation of options, estimate a broad range of outcomes. Other things being equal, it is also desirable to have a model that can evaluate proposals for fine tuning one or more aspects of complicated sets of program provisions, that can develop projections over long time horizons, and that can provide distributions of outputs for particular groups of interest in addition to population means. As an example, in considering a policy change that increases or decreases employer pension benefits, decision makers are likely to be interested in the effects not only for all workers, but also for workers in different industries, age groups, and earnings categories and by size-of-employer categories. Finally, a projection model is

OCR for page 132
DEVELOPMENT OF PROJECTION MODELS 137 more useful to the extent that it can estimate the second-round effects of policy changes, which, because of behavioral responses, are likely to differ from the initially estimated effects. For example, changes designed to increase employer pension coverage, if they also increase employer costs, could lead employers to opt out of pension plans and hence reduce coverage over time. Desirable though these dimensions are, there are costs to increasing the breadth and depth of model capabilities. For example, models that simulate a wide range of policy outcomes and options are likely to be time and resource intensive to develop and apply. Also, modeling of feedback effects is quite difficult, both operationally and because of the uncertainty of behavioral re- sponses to policy changes. Hence, it may be cost-effective to have different models for answering questions about particular types of policy changes, as well as different models for estimating means versus distributions, effects over shorter versus longer periods, and second-order as distinct from first-order effects. In- deed, the various types of modeling strategies listed earlier, such as cell-based and microsimulation approaches, reflect choices about how much and what kinds of detail to include in models. With a range of models, it becomes critical to consider ways to facilitate links between them (the sixth dimension listed). The last four dimensions listed response time, openness and transparency, ease of use, and portability have to do with operational features that determine how readily a model can be applied during the course of a policy debate, modi- fied as needed to incorporate additional capabilities, and validated and improved. These dimensions, which cut across model types, also have to do with how large a user community develops for a model, which, in turn, is a major factor in how widely a model is used and how rigorously it is evaluated. Features of models that facilitate a higher level of performance on these dimensions include adequate documentation, public availability of the model to analysts, a user-friendly interface that allows people who are not programmers to operate the model, and other design elements that permit ready modification of the model to simulate newly proposed policy options in a consistent manner and to facilitate multiple runs and specifications for model validation purposes. Many of these features (e.g., real-time response mode or a graphical, point-and-click user-friendly interface) appear more obtainable when a model is designed for microcomputer or workstation computing environments than when it is designed for mainframe or supercomputing environments. It clearly seems important for models to provide timely response and to be open, easy to use, and portable, whatever their type or breadth and depth of capabilities. However, the costs of achieving high levels of performance on these dimensions must also be considered. As an example, it is typically time and labor intensive to produce good documentation, although, with forethought, it is pos- sible to build features into models to facilitate the documentation process. Also, the nature of an agency's information requirements may dictate trade-offs be- tween, say, the ease of use of a model and its capabilities. For example, a

OCR for page 132
138 ASSESSING POLICIES FOR RETIREMENT INCOME particular set of applications may require computing power that is not available from microcomputers or workstations (e.g., CGE models may require super- computers to attain the level of desired detail of inputs and outputs). In this case, the only modeling option is to invest in a model that has limited access. An alternative is to implement a simpler version of the model on a microcomputer and a more elaborate version or a version that will process larger numbers of sample cases on a supercomputer. Assessment There are many projection models available today that relate in some way to retirement income security. Some of these models were developed for very narrowly targeted applications, such as actuarial cell-based models that are de- signed to project the balance of assets and liabilities for a specific company's pension plan. Although these kinds of models serve important purposes, we have not reviewed them because of their lack of generality. Also, our review does not extend to health care financing projection models, although such models are very important to a full assessment of retirement income security, nor to tax revenue models, such as that used by the U.S. Treasury Department.1 Rather, we have looked primarily at models that can address somewhat broader retirement-in- come-related policy questions, for example, the effects of tax policy changes on accumulations in Individual Retirement Accounts (IRAs) and pension accounts over people's working lives. Our overall conclusions about existing models are mixed. Some models appear to have useful features for some kinds of applications but, on balance, we conclude that existing models are not adequate to fully inform the retirement income policy debate. For some important topics, there are currently no available relevant models. Specifically, there are no models of employer behavior with regard to benefit offerings or demand for older workers that could provide input for individual- based models of labor supply and retirement asset accumulation. Rather, existing employer-based models address such issues as pension funding effects on share prices and employer pension policy within a corporate financial setting. The new PIMS microsimulation employer-based model has been developed for the PBGC, but it has a limited focus. PIMS estimates corporate bankruptcies among em- ployers with defined benefit pension plans on the basis of simulating key eco- nomic variables and employer characteristics. It also simulates plan participants and funding status. Finally, it simulates the implications of projected bankrupt 1See Office of Technology Assessment (1994) for an evaluation of models that were used in the recent health care reform policy debate.

OCR for page 132
DEVELOPMENT OF PROJECTION MODELS 139 cies of companies with underfunded plans for the PBGC fund that insures such plans. For individuals, there are microsimulation models that are broad in scope and provide the capability for detailed, disaggregated analysis. They could be very useful for retirement-income-related policy analysis, but they generally suf- fer from several deficiencies: they have not been adequately validated and are not designed to facilitate validation; they are based on old or limited data sets; they do not reflect best current professional judgment about underlying behavior; they have little facility for implementing alternative behavioral specifications when there are important disagreements; their provision for feedback loops be- tween policy changes and behavioral responses is weak; they generally have little provision for ready exchange of inputs and outputs with other models; they are not well documented; and they are not open and transparent or readily portable, although they now are generally implemented with personal computing technol- ogy. CGE models that have been developed to answer such questions as the intergenerational benefits and costs of alternative Social Security financing sys- tems are impressive in their treatment of the interactions of the household, gov- ernment, and private sectors of the economy and in their use of up-to-date infor- mation and research knowledge. However, these models deal primarily with "big picture" issues in a highly stylized manner and primarily for the Social Security system, not the much more heterogeneous world of employer pensions. Also, to date, these models have not addressed the limitations of the life-cycle model of savings and consumption (see Chapter 3), and they are usually structured as equilibrium models without unemployment, particularly long-term unemploy- ment, among older workers. These models do not provide distributional detail, nor can they address the effects of very specific policy changes or combinations of changes. Finally, there are as yet no links between these models and cell-based or microsimulation projection models that would provide additional specificity. A positive trend is that newer models generally follow design principles that encourage use and validation. For example, PIMS, the employer-based model, operates on personal computers and has a design that facilitates model validation. However, newer models, such as PIMS, SSASIM, the Solvency and Individual Retirement model (SIR), and the model under development by Wolf et al. (1995), tend to be limited in scope. SSASIM is a cell-based model similar in structure to the SSA actuarial cost model that uses Monte Carlo methods to characterize uncertainty about forecasts of Social Security trust fund balances. SIR is a cell- based model similar to the SSA actuarial cost model that provides limited distri- butional information for different types of workers. Wolf is developing a micro- simulation model primarily for simulating kinship networks and disability status of the elderly. Because individual-level microsimulation models potentially offer very im- portant capabilities for retirement-income-related policy analysis, we review as

OCR for page 132
140 ASSESSING POLICIES FOR RETIREMENT INCOME pects of two of the existing dynamic models DYNASIM2 and PRISM in Appendix D. The discussion focuses on some weaknesses of these models, not to single them out for criticism, but to support our case for important areas of future development for this type of model.2 LOOKING TO THE FUTURE We recommend priority attention to data collection and research and analytical modeling over the next few years rather than to significant investments in broad- based projection models. However, we believe it is important to establish goals for projection model development and, in some instances, to move forward on design work, so that implementation can progress expeditiously once invest- ments in data and research bear fruit. Specifically, the federal government should establish goals for the development of employer models (using cell-based or microsimulation techniques) and for a new general-purpose individual-level microsimulation model. Other retirement income modeling tools are needed as well, but it is clearly important to have the projection capabilities that employer models and a new individual-level microsimulation model can provide. Employer Models Employer hiring and compensation policies are critical to retirement income security, most obviously affecting the extent of pension coverage and benefits, but also affecting Social Security benefit levels and the opportunities to contrib- ute to such earnings-linked savings vehicles as IRAs. Employers also often provide other important benefits, such as retiree health care and disability insur- ance, as well as opportunities for part-time, post-retirement work. A full assess- ment of the likely effects of many retirement-income-related policy proposals will not be possible without considering what is happening to and with employ- ers. Shifts in employment across sectors that have historically differed in wage and benefit levels can affect retirement income security, as can employer re- sponses to changes in policies and other factors that alter incentives (e.g., pension simplification may increase the probability that small businesses will set up pension plans). Employer-based projection models are clearly needed for much retirement income policy analysis. In some instances, decision makers will want informa 2For other assessments of microsimulation models, see Citro and Hanushek (1991); the papers cited in Cohen (1991a); Employee Benefit Research Institute (1980a, 1980b, 1981); Reno (1993:29- 31); Ross (1991); and U.S. General Accounting Office (1986). Burtless (1989) and Grossman and Watts (1982) focus on behavioral elements of microsimulation models. Anderson (1996) and Hollenbeck (1995) review microsimulation and other projection models that are relevant for retire- ment income policy modeling.

OCR for page 132
DEVELOPMENT OF PROJECTION MODELS 141 lion about how many employers will take a certain action within a specified time period (e.g., set up a pension plan) and how many workers will be affected. Such information could most likely come directly from an employer model that in- cludes such characteristics as industry and size. In other instances, decision makers will want to know the longer term implications for retirement income flows (e.g., from pension accumulations) for different groups of workers; for this information it will most likely be necessary to produce outputs from employer models that can feed into individual-level models for workers and their families. At present, there are no models that can project the distribution of employer pension and related benefit offerings or the distribution of employer demand for workers. Moreover, there is little basis on which to develop such models in the near term: the needed data and research knowledge are simply lacking (see Chapters 3 and 4~. Resources spent on model development cannot be cost- effective until more complete data are available (e.g., about the distribution of employer benefits) and until there is much better understanding of the factors that influence hiring and compensation decisions for different kinds of employers. Gaining such understanding will require considerable elaboration and testing of alternative theories about employer behavior. The development of employer- based projection models is an important goal for the Department of Labor and other agencies, but one that can only be realized over the long term. It is important to keep this long-term model development goal in mind as decisions are made about investments in employer-related data and research. In this regard, it could be useful to consider alternative projection model designs and strategies and their requirements. One question is whether to start with a cell-based design that classifies employers on a limited number of important dimensions or to opt for a microsimulation model. A cell-based model would be easier to implement and have fewer data requirements than a microsimulation model, but it would have more limited capability for analysis and for simulation of behavioral response. It could also be useful to bring together researchers and people with experi- ence in developing projection models to consider possible model designs on the basis of anticipating what research results may show. For example, what would be the design and data requirements for a projection model if employer costs are taken to be the key factor in hiring and compensation policies? What would be the requirements if employee preferences are also deemed to be an important factor? As new data about employers become available and are analyzed, DOL could sponsor workshops to discuss research findings and their implications for the development of employer-based projection models. Clearly, to make progress toward the goal of model development, it will be important for DOL to give priority to the work of the interagency task force on employer data (see Recom- mendations 5-9, Chapter 4) and to putting an employer data collection plan on as fast a track as possible.

OCR for page 132
42 ASSESSING POLICIES FOR RETIREMENT INCOME Recommendation 14. The U.S. Department of Labor should consider the development of employer models as an important long-term goal. Retirement-income- related policy analysis requires models of employer behavior that can project the likely future distribution of benefit plan offerings and the demand for older workers. However, there is little basis on which to design, let alone implement, such models until investments have been made in needed data and behavioral research. Microsimulation Models of Individuals and Families Policy makers often want to know the effects of proposed policy changes on population groups, categorized by age, sex, race or ethnicity, income level, and the like. Cell-based models can provide distributional detail, but the cells must be defined in advance and are difficult to change in response to user needs. In principle, large-scale microsimulation is a much more flexible tool with which to provide detailed information about distributional effects, limited only by sample size constraints on the reliability of small cell estimates. Broad-based micro- simulation can also address important interactions among policies, such as how a change in Social Security benefits will likely affect benefits from employer pen- sion plans that are linked to Social Security. Dynamic microsimulation can in principle forecast the long-term effects of policy changes, including behavioral responses: for example, how a change in Social Security benefits will likely change retirement and savings behavior and, ultimately, affect retirement income levels for different cohorts of workers. We believe that a general-purpose, dynamic microsimulation model is needed to project the distributional effects of policy alternatives on the retirement income security of individuals and families over the medium term and for modeling interactions among policies. Other, more limited modeling tools are often appro- priate and cost-effective for specific policy questions. However, as Burtless (1996) argues, it is important to have the capability to address all three legs of the retirement income stool Social Security, employer pensions, and personal sav- ings in an integrated modeling framework that can provide needed distribu- tional detail. Such a modeling framework is also useful because it can help structure related analytical work. The effort to develop and apply a large-scale micro- simulation model will invariably identify behavioral interactions and processes that need to be better understood. It will also help determine which parameters are crucial for analysis and which are less important, and it can suggest how concepts and variables should be consistently defined and measured to be useful for modeling purposes. Unlike the situation for employer models, there are existing individual-level

OCR for page 132
154 ASSESSING POLICIES FOR RETIREMENT INCOME to conduct "ex post forecasting" studies, in which one puts oneself in the place of an analyst who, say, 10 or 20 years ago, was asked to simulate a policy change to take effect in some future year. One chooses the "future" year to be in the recent past so that measures of what happened are available from administrative records or other sources. Correspondingly, one chooses the policy change to be the actual policy in effect during the comparison year. (An alternative approach is to use a current model to develop "backcasted" estimates of policies for earlier years.) Because differences between a model' s estimates and what actually occurred may be due to economic or social changes that the model could not have been expected to forecast (e.g., an unanticipated recession or an unexpected increase or decrease in fertility rates), it will be helpful to conduct ex post forecasting (or backcasting) studies that use actual values for the factors that could not have been anticipated and focus on the accuracy of the model with regard to other elements. Also, in many instances, the important criterion will be whether the model accu- rately projected the differences in the outcomes under two or more policy alterna- tives and not the more stringent criterion of whether it accurately projected the levels of outcomes for a specific alternative. In addition to studies that help establish confidence in a model's track record, it is important to develop estimates of the range of uncertainty of a model's estimates. On average, a model may produce unbiased estimates, but the results of any particular model run may have a wide band of uncertainty around them. The sources of error in model simulations can be summarized in four broad categories: (1) sampling variability in the input database, which is only one of a family of possible data sets that could have been obtained; (2) sampling variabil- ity in other inputs, such as imputations, regression coefficients, and control totals; (3) errors in the database and other inputs; and (4) errors due to model . . ^. . m~sspecl~lcatlon. Advances in statistical methodology that take advantage of modern comput- ing power have made it possible to estimate the uncertainty in model estimates due to such sources as sampling variations in the input data. One such technique is the bootstrap (see Cohen, l991c), which, simply put, measures variability by using the observed sample distribution in place of the unobserved population distribution. The strength of this approach is that variance estimation becomes a function of computing power rather than an exercise in solving multidimensional integrations for complex estimators and distributions. For complex models, like dynamic microsimulation models, sample reuse techniques can be used to measure the uncertainty in a model's estimates due to the two kinds of sampling variability. However, the uncertainty due to the two kinds of errors may be difficult to put into a probabilistic context. Sensitivity analysis, which is carried out by developing and running one or more alternative versions of one or more model components, can help assess the uncertainty in model estimates due to the choice of a particular model structure or specification.

OCR for page 132
DEVELOPMENT OF PROJECTION MODELS 155 For example, one could use sensitivity analysis to investigate the effects of a particular equation for simulating the labor supply response to a change in the retirement age for Social Security or employer pensions. In its simplest terms, sensitivity analysis is a diagnostic tool for ascertaining which parts of an overall model could have the largest impact on results and therefore are the most impor- tant to scrutinize for potential errors that could be reduced or eliminated (see Citro and Hanushek, 1991:89-96; Chap. 9~. Sensitivity analysis is particularly important for retirement income modeling in view of the long time horizon for most projections, which will compound any errors in model specification or assumptions. For various reasons, external validation studies, estimation of uncertainty, and sensitivity analysis have rarely been carried out for projection models.6 His- torically, one reason was that such studies were very costly and time-consuming. However, if models are implemented in computing environments that provide ready, cost-effective access to users for multiple runs and model respecifications, then validation becomes very feasible. Another reason there are few estimates of uncertainty is that decision makers usually want only point estimates, without measures of uncertainty. It is impor- tant, therefore, to develop innovative ways to convey estimates of uncertainty that decision makers will take into account in debates about policy alternatives (see Citro and Hanushek, 1991:86-88~. This task may be easier for retirement- income-related policy projections because decision makers have somewhat more experience with estimates of uncertainty for such projections than for estimates on other topics. Thus, because of the long time horizon, the SSA Office of the Actuary routinely presents high-cost, intermediate, and low-cost scenarios for the balance in the Social Security trust funds under current law. However, this approach to bounding uncertainty is problematic (see "SSA Model Validation," below). Moreover, there is no comparable record of regularly projecting the long-term funding status of employer pensions, even though this source of retire- ment income may be more uncertain than Social Security (see Burtless, 1993; also, Schieber and Shoven, 1993, develop such a projection for defined benefit and defined contribution plans). Finally, an essential feature of an acceptable projection model is that it incorporate best professional judgment about relevant factors, particularly for aspects of the model to which estimates are sensitive. "Best" professional judg- ment is not always the newest theory or latest academic model, which may be too new to have been properly evaluated. Also, because there is a range of profes- sional opinion in many areas, it will be important to conduct sensitivity analyses of alternative specifications before selecting a particular approach to use for a 6See Cohen (1991a) for a review of validation studies that have been done of social welfare policy analysis microsimulation models, including DYNASIM2 and PRISM.

OCR for page 132
156 ASSESSING POLICIES FOR RETIREMENT INCOME central or base case. Yet it is clear that when important changes have occurred- for example, in worker productivity or savings rates or mortality rates a model that uses data or estimates of behavioral parameters that do not reflect those changes will not be helpful in informing the policy debate. Also, given that relatively little work has been done to date to conduct external validation studies or sensitivity analyses of models, the extent to which a model incorporates best professional judgment is often the only indicator of the likely level of model performance. SSA Model Validation The Social Security actuarial cost model, which uses cell-based methods to carry forward the results of time-series analysis, is an exception to our statement that model validation and assessment of uncertainty are rarely performed. The Office of the Actuary's 75-year projections of trust fund balances under current law, which are revised every year, are often evaluated by comparing them to what actually occurred in the short run and by looking at the patterns of yearly revision (see, e.g., Schieber and Shoven, 1993~. Also, the Office of the Actuary routinely prepares trust fund balance projections for low-cost, intermediate, and high-cost scenarios, thereby providing a type of sensitivity analysis (similar to that pro- vided for Census Bureau population projections). SSA also provides sensitivity analyses of the separate effects of key assumptions. In these analyses, the high- cost, intermediate, and low-cost values are used for one assumption, such as fertility, while the intermediate values are used for all other assumptions. However, as Burtless (1993) points out, the assumptions underlying the low- cost and high-cost scenarios are not chosen so as to bound the likely range of potential experience in any manner that approximates a statistical confidence interval. Hence, they are less useful to policy makers, analysts, and the public because they do not indicate the probability of a best-case or worst-case outcome relative to the intermediate projection. Also, the assumptions may be internally inconsistent. For example, the growth rate of real wages (for which a higher value is a favorable outcome for the Social Security system) may be negatively correlated with real interest rates (for which a higher value is also a favorable outcome for the system). Finally, key assumptions of the central or intermediate scenario may not always reflect current best professional judgment. For example, the mortality forecasts developed by SSA are at one extreme of the range of available fore- casts: they represent a more pronounced slowing of the rates of mortality decline observed in this century than is forecast by others (see Lee and Skinner, 1996~. The implication of the SSA forecasts is that people will not live as long as forecast by others and, hence, that the growth in the elderly population, particu- larly of the oldest old (those aged 85 and older), will be less pronounced. Conse- quently, the use of the SSA forecasts, even from the high-cost scenario, could

OCR for page 132
DEVELOPMENT OF PROJECTION MODELS 157 result in overly optimistic assessments of the ability of employer pensions or personal savings to provide adequate retirement income support over retirees' life spans. The use of the SSA forecasts is also problematic for assessments of the Social Security system, even though the relevant population estimate is not the number of elderly per se but the old-age dependency ratio (people aged 65 and older as a ratio of people aged 20-64~. While a lower mortality rate than that forecast by SSA will increase income to the system (more younger people will be alive to contribute payroll taxes), it will also increase payouts from the system, and the relative increase in outgo will exceed the relative increase in taxable payrolls (see Board of Trustees [OASDIi, 1994: 132-133~. Lee and Carter (1992) provide a credible alternative to the SSA mortality forecasts. Their method essentially projects into the future the rate of mortality decline observed over the twentieth century, which has been fairly steady despite periods of faster and slower progress. SSA's method extrapolates cause-specific death rates over the preceding 20 years, which necessarily results in a marked slowing in the rate of mortality decline, as the more rapidly declining causes of death come to claim a smaller and smaller share of total mortality. Of course, it is impossible to know which of the two methods will ultimately prove more accurate. A major advantage of the Lee and Carter approach, noted by Lee and Skinner (1996), is that it provides probability intervals describing the uncertainty in their extrapolative method. However, the Lee and Carter estimates of uncertainty do not allow for model specification error or the possibility of "structural breaks" in time series, such as the sharp decline in age-specific mortality rates that occurred between the nineteenth and twentieth centuries. Hence, Lee and Carter probably underestimate the extent of uncertainty in their forecasts of mortality rates. Recently, Lee and Tuljapurkar (1994) extended the methods of Lee and Carter to develop stochastic population forecasts with consistent and meaningful probability intervals for all items forecast (e.g., age categories of the population, taking account of all demographic processes). Lee and Tuljapurkar formulated models of age-specific fertility (from Lee, 1993) and age-specific mortality (from Lee and Carter, 1992), in which they estimated a single time-varying parameter for each process by using standard time-series analysis techniques. (Net immi- gration could also be handled in the same manner, but it was treated as determin- istic at the level assumed in the Census Bureau projections.) They then used these fitted models of stochastically varying rates as inputs to the population renewal process and carefully tracked the propagation and sources of error. They used analytic methods to find both the expected value of the forecast and the probability intervals. Similar results could be obtained with Monte Carlo methods, that is, by repeated stochastic simulations using the stochastic models of vital rates. The SSASIM model, recently developed with support from SSA, uses Monte Carlo techniques to characterize uncertainty in the Social Security trust fund balance

OCR for page 132
158 ASSESSING POLICIES FOR RETIREMENT INCOME forecasts (Holmer, 1995a, 1995b; see also Sze, 1995, and Appendix C). SSASIM implements stochastic processes for all of the key demographic and economic variables that are used in the SSA actuarial cost model in a general framework that can accommodate different expectations (specified by the user) about the dynamics of each variable over time and in association with other variables. For example, a policy analyst can assume that such variables as rates of mortality decline follow a consistent trajectory over the span of the projection period (as does the SSA actuarial cost model) or that the time trend will exhibit cyclical fluctuations. The analyst's assumptions are used to specify the means, standard deviations, and correlation coefficients of a multivariate normal distribution: multiple runs with different random draws from the distribution for a particular set of assumptions generate the expected values of variables and the probability bounds. The use of scenarios by SSA and the Census Bureau to convey uncertainty about forecasts stands in contrast to the use of probabilistic methods. To build scenarios, an analyst develops one forecast on the basis of the "expected" trajec- tories for each demographic or economic process. The analyst then also develops alternative trajectories for each process above and below the expected trajectories and combines them in various ways to produce high and low forecasts with which to bound the intermediate or expected forecast. For example, because trust fund balances are influenced by the old-age dependency ratio, SSA combines low fertility with low mortality for its high-cost scenario. The Census Bureau, on the other hand, combines high fertility with low mortality into a high scenario for total population growth. As a consequence, the Census Bureau's high and low estimates of the elderly population cover a wider band than the SSA estimates, while the Census Bureau's high and low estimates of the old-age dependency ratio cover a narrower band. The SSA high- and low-cost scenarios for the population aged 65 and older and the old-age dependency ratio cover a somewhat narrower band than the 95 percent probability intervals generated by Lee and Tuljapurkar (1994~. One could use the Lee and Tuljapurkar upper and lower bounds to produce new population scenarios. However, as Lee and Skinner (1996) point out, the prob- lem with any method of constructing scenarios is that there is no means by which to attach probability intervals to the resulting forecasts. The scenario approach assumes unrealistically that the high, medium, or low trajectory for each de- mographic process is followed consistently over the entire time span. In contrast, the stochastic forecasts of Lee and Tuljapurkar treat fertility and mortality as random variables at every iteration, producing true probability bounds for the projections at each period (e.g., each 5-year interval). Although stochastic methods may underestimate the uncertainty in demo- graphic and economic forecasts, depending on how the parameters of the stochas- tic process are estimated, they represent a major step forward over the scenario approach for conveying estimates of uncertainty to policy makers. The use of

OCR for page 132
DEVELOPMENT OF PROJECTION MODELS 159 stochastic methods to estimate the uncertainty of projections of demographic and economic variables seems imperative for the kinds of long-range forecasts that are often used in retirement-income-related policy debates. In this regard, it would be very useful to further develop the capabilities of the SSASIM model for estimating the uncertainty of the projections in the SSA model, including work on alternative methods with which to estimate the parameters of the stochastic pro cess. Recommendation 16. Models that project policy outcomes over long periods should rou- tinely assess the accuracy and uncertainty of the projections and report prediction errors in previous forecasts. To be able to evaluate a model's projections in a cost-effective manner, the model design must explicitly in- clude a capability for sensitivity analysis (e.g., how much the outputs vary with alternative assumptions and versions of model components) and a capa- bility to estimate uncertainty from such sources as sampling variability in the database and other model inputs. NEAR-TERM MODELING STRATEGIES The kinds of investments we recommend in data and research will take time to return benefits for retirement-income-related policy analysis. In addition, the development of needed employer models and a new large-scale individual-level microsimulation model, which must await advances in data and research knowl- edge, cannot be completed in the near term. In the meantime, the Department of Labor and other agencies must be able to answer decision makers' questions about the likely effects of alternative policies. They cannot respond that they are unable to provide requested information because data and knowledge are lacking with which to develop improved projection models. In order to be responsive to decision makers' legitimate information de- mands without incurring significant model development costs that could well be misdirected, agencies should try to frame policy questions as specifically as possible and use limited special-purpose models together with the best available data and research knowledge to answer those questions. Depending on the issue, a special-purpose model may already exist, or, more likely, it will be necessary to adapt one from another model or to construct one on the spot. Admittedly, limited ad hoc models may not provide very reliable answers to policy questions, and, in particular, they are not likely to be able to handle important policy interactions. Also, special-purpose models may be able to model behavioral responses to policy changes only very crudely, although they should incorporate such responses when reasonable parameters are available and feasible to implement. However, the alternative of trying at this time to develop

OCR for page 132
160 ASSESSING POLICIES FOR RETIREMENT INCOME more complex models, when needed data and research knowledge are lacking, will not likely give a better result and will very likely waste scarce resources. We acknowledge that a phased approach to retirement income modeling, as we recommend, may not provide useful answers before there is actually a retire- ment income crisis that requires extensive evaluation of alternative policy pro- posals in a comprehensive framework. However, there will be continuing need for retirement-income-related modeling capabilities (as is true for health care policy), and without investments in data and behavioral research, government agencies will never be in a position to develop markedly improved projection models. (The health care agencies have recognized this point by redesigning and expanding their surveys.) Again, we are concerned about the use of scarce resources in the near term. We believe they are best used for data and research instead of extensive development (or redevelopment) of large-scale models. To the extent that limited investments can lead to better special-purpose models for specific questions, we support those investments. The CBO Approach The modus operandi that we have in mind is that of the Congressional Budget Office (CBO), whose analysts regularly prepare ad hoc, special-purpose models, often using spreadsheet tools, to develop estimates of the likely costs and other effects of proposed policy changes. CBO staff combine data and parameter estimates in their ad hoc models from a wide range of sources, sometimes includ- ing as inputs the outputs of larger, more formal projection models with the results adjusted as seems appropriate. Sometimes they make additional "out-of-model" adjustments, for example, to provide crude estimates of the likely effects of behavioral response to policy changes. (See Citro and Hanushek, 1991:41-51, for a case study of the modeling tools used by CBO and the U.S. Department of Health and Human Services in the late 1980s for estimating the effects of welfare reform.) In general, CBO staff develop their models only to the point to which they can answer the question at hand. For example, projections for a few years might be developed with distributional detail, while longer term projections are devel- oped on a very aggregate basis. Or a model might be developed to handle only the particular policy change that is under consideration. However, in instances in which the same kind of question arises frequently, CBO will develop more for- mal models that are intended for repeated use and that can handle a wider range of variations in a particular policy area. As just one example, CBO developed a hospital-based model that is designed to estimate the cost and distributional effects of changing various provisions of the Medicare prospective payment scheme used to reimburse hospital costs. The problem with an ad hoc approach to model building is that there is virtually no incentive to provide for such desirable model features as openness or

OCR for page 132
DEVELOPMENT OF PROJECTION MODELS 161 portability or to document a model for use by others. More important, there is no incentive to perform more than the crudest type of model validation. However, the strategy of special-purpose model development, when carried out by knowl- edgeable analysts, can provide answers to specific policy questions with mini- mum staff resources and time. The answers will very likely be limited and of highly variable quality, but they can assist the policy debate, and the process of developing them can identify issues to consider in the design of new, improved large-scale models for use in future policy projections. The problem for retirement-income-related projection modeling for the ex- ecutive branch in the near term (and in the long term, too) is that there is no equivalent of CBO in a cabinet department. CBO will itself prepare estimates for retirement-income related policy proposals, but the executive branch needs its own modeling capability, particularly when proposals are being drafted and have not yet been sent to the Congress. There is no staff group in the executive branch that is in a position to take a broad view of policy analysis in this area, much less to make effective use of available data, research, and modeling tools to develop the best possible estimates in the most direct manner in response to decision makers' questions. There are clusters of analysts in such agencies as DOL and SSA, but it would be helpful to have a larger critical mass working together on a regular basis. For example, SSA analysts are currently working on models of the distribu- tion of Social Security benefits by using matched SSA earnings records and SIPP data. The SSA analysts initially developed models with which to project earnings histories for 1984-1993 for members of the 1984 SIPP panel on the basis of prior earnings histories, and they obtained reasonable results. They then used these models to project future earnings histories for members of the 1990 SIPP panel (lams and Sandell, 1996; Sandell and lams, 1996~. The SSA analysts propose to extend their models to project pension income, as well as Social Security ben- efits, on the basis of earnings histories and other characteristics of SIPP sample members. If successful, and if a savings income projection capability is also added, this approach could offer some useful policy modeling capability without requiring a full-fledged dynamic microsimulation model (assuming that policy changes can be factored into the projections in some manner). SSA analysts are also working on cross-sectional microsimulation models of SSI and the disability insurance (DI) component of the Social Security system with matched SIPP and disability program data from SSA records. As part of this effort, they are developing ways to simulate the multistage disability application and determination process (see Lahiri, Vaughan, and Wixon, 1995~. Initially they expect to be able to make estimates of the number of people in the civilian noninstitutional population who are likely to meet SSA disability criteria. Esti- mates of those categorically eligible on the basis of disability will then be com- bined with estimates of financial eligibility for SSI and DI insured status. A subsequent model of the application decision is also planned and will be used to

OCR for page 132
162 ASSESSING POLICIES FOR RETIREMENT INCOME introduce selectivity adjustments to the estimates of the pool of those eligible for disability benefits. This SSA modeling work should be well known and available to other ana- lysts, such as those in the Pension and Welfare Benefits Administration in DOL. Similarly, work that PWBA analysts may do with employer pension issues should be well known and available to analysts in other relevant agencies. It is probably not feasible to establish a cross-agency retirement income modeling group with authority to prepare estimates on behalf of the executive branch, but perhaps it is possible to establish coordinating mechanisms. For example, it would be useful to establish an interagency users group that meets regularly to review retirement- income-related projection model developments at the member agencies. Such meetings could contribute to the design of new, improved models in the future. Improvements to Existing Models In the near term, while large investments in projection model development are in abeyance, it is important to seek opportunities for cost-effective improvements to existing retirement-income-related projection models and to make them known and available across agencies. For example, it may be possible to marginally improve the pension components of such models as DYNASIM2 and PRISM, even though major improvements in pension simulation capability will depend on the development of new models that use greatly improved data. An important model for which it appears feasible to effect useful improve- ments in the near term is the SSA actuarial cost model. The SSA model is the linchpin of retirement-income-related projection modeling. It produces rela- tively few outputs (principally, the trust fund balances over a 75-year time hori- zon), but its projections are critical in setting the tone for policy debates. Also, many other models calibrate their outputs to components of the SSA projections, such as the forecasted population size by age and sex and wage growth and other economic factors. We urge attention to extending the SSA model in several ways.7 First, it is critically important to develop probabilistic estimates of uncertainty for each of the variables projected in the model and then to develop estimates of uncertainty of the model's outputs as a whole, taking account of covariances among the various time series (mortality, labor force participation, wage growth, etc.~. Such estimates should replace the current scenario approach, which has no probabilis- tic structure. Work is under way at SSA along these lines (see the previous discussion in "SSA Model Validation"), and we urge that it be continued. Second, it would be highly useful to add a capability to the model for limited 7The 1994-95 Advisory Council on Social security Technical Panel on Assumptions and Methods (1995:60-66) has made similar recommendations.

OCR for page 132
DEVELOPMENT OF PROJECTION MODELS 163 distributional analysis of the effects of current and proposed Social Security provisions. For example, it would be useful to be able to project costs and benefits of policy alternatives for low, middle, and high earners among the popu- lation. Such projections would be helpful in evaluating outputs of more disaggre- gated models, and vice versa. A distributional component of the SSA model should include a capability to simulate the effects of cyclical or random varia- tions in key aggregate variables, such as interest rates and growth in wages. Third, because of the importance of mortality projections for estimates of the retirement-age population, we urge SSA to evaluate the sensitivity of the model's outputs to mortality projections that are disaggregated by such characteristics as marital status and income levels. If the analysis suggests a strong degree of sensitivity, it would be important to undertake research on the best form of such projections for use in the model. SSA administrative data could provide the basis for a comprehensive study of the relationship of mortality to earnings for people of retirement age. (Studying the relationship of mortality to marital status with these data would require SSA to track the marital status of all beneficiaries, which could also be useful for projections of beneficiaries.) SSA also has several SIPP panel files to which it is appending year-by-year mortality records. These files could permit a rich, multivariate analysis of mortality correlates for a limited sampled Finally, it would be highly useful to develop means by which to document and provide research access to the model. Outputs of the SSA actuarial cost model are so widely used for inputs to other models that it would be very helpful if outside analysts could more readily learn about and have access to the model. Such access would also facilitate outside reviews of the model's properties to identify other improvements. Recommendations 17. To respond to immediate policy needs, agencies should use limited, special-purpose models with the best available data and research findings to answer specific policy questions. Although such models may not provide very accurate estimates, the alternative of developing complex new indi- vidual-level microsimulation or employer models in advance of needed im- provements in data and research knowledge has little prospect of producing better results and will likely represent, in the immediate future, a misuse of scarce resources. 8Panis and Lillard (1996) recently developed mortality models with data on age, race, sex, marital status, and household income of surviving sample members and decedents in the PSID. They used these models to simulate the effects of proposed Social Security changes, such as increasing the contribution rate and decreasing benefits, on the net present value of Social Security benefits for prototypical types of beneficiaries.

OCR for page 132
164 ASSESSING POLICIES FOR RETIREMENT INCOME 18. The Social Security Administration should consider enhancements to the SSA actuarial cost model in the near term. This model provides useful outputs for other models and helps frame the overall retirement income security policy debate. Priority improvements include the following: replacing the scenario approach with probabilistic estimates of un- certainty for the model's projections that take into account covariances among the various time series (mortality, labor force participation, wage growth, etc.~; adding a capability for limited distributional analysis of the effects of current and proposed Social Security provisions (e.g., of costs and benefits for low, middle, and high earners); evaluating the sensitivity of the model's outputs to mortality projec- tions that are disaggregated by such characteristics as marital status and income levels and planning to develop such projections, if warranted; and developing means to document and provide research access to the model.