Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 132
Development of Projection Models
As the concern about retirement income security and the costs to provide it
increases over the next few years, decision makers will look to projection models
to estimate the likely costs and other effects of alternative policy proposals. Such
models will require good data and a solid base of research knowledge with which
to project demographic and economic trends and behavioral responses to policy
changes and other factors. Outputs from analytical models of the behavior of
individuals and employers that receive general consensus should be inputs to
projection models. Otherwise, when policy proposals are expected to elicit a
behavioral response, projection models will fall short in estimating the likely
effects. In the preceding two chapters, we identified critical deficiencies in both
data and research. As we discuss below, available projection models are also
deficient in many respects. The question is how to allocate limited resources for
developing better capabilities for retirement-income-related policy analysis.
As we stress throughout this report, the U.S. Department of Labor (DOL)
and other relevant agencies should give priority to effecting needed improve-
ments in data and analytical research before making significant investments in
broad-based projection models. Such investments will be misplaced until there is
better understanding of key behaviors such as employer decisions about hiring
and compensation policies and individual decisions about savings and consump-
tion and better data with which to project trends in those behaviors and other
factors. Until efforts to improve data and research bear fruit, we recommend that
agencies make limited investments in special-purpose models that use the best
available data to answer policy makers' specific questions.
At the same time, agencies should consider the kinds of broad-based projec
132
OCR for page 133
DEVELOPMENT OF PROJECTION MODELS
133
lion modeling capabilities that it will be important to develop in the future. In
this chapter, we address: types of projection models that could be useful; the
dimensions of models that should be considered in planning for improved projec-
tion capabilities; the capabilities and limitations of existing models; long-term
issues in developing models of employers and employer behavior; long-term
issues in developing improved individual-level microsimulation models; issues
in developing validation capabilities in new and current models; and strategies
for projection modeling in the near term with existing models or one-time spe-
cial-purpose models (including improvements to the model maintained by the
Social Security Office of the Actuary).
CURRENT MODELS AND THEIR USES
Types of Models
Projection models vary widely in scope, complexity, and modeling strategy.
Some of the major types are time-series models, cell-based models, microsimu-
lation models, macroeconomic models, and computable general equilibrium mod-
els. Some models incorporate multiple approaches: for example, in the Macro-
economic-Demographic Model (MDM), a macroeconomic growth model
interacts with cell-based models of population growth, family formation, the
labor market, and other model components. Other models reflect primarily one
approach but make use of the outputs of other kinds of techniques; see Appendix
C for descriptions of some models.
Times-Series Models Time-series models project aggregate historical data
into the future on the basis of a function, which can be simple or complex, of the
behavior of the data in previous periods. For example, future labor force partici-
pation rates for older men can be projected to continue to decline on the basis of
trend data from 1950 to 1995. However, the longer the projection period, the
more hazardous it is to assume that past trends will continue unchanged into the
future. Indeed, for the example just noted, there is currently debate about whether
the trend toward early retirement among men has peaked.
Cell-Based Models Cell-based models divide the population into groups,
such as 5-year age categories by sex, and develop separate projections for each
group for example, separate projections of life expectancies, retirement age
probabilities, and so forth. The separate projections may be based on time-series
analysis, modified by assumptions about the likely behavioral effects of policy
changes (e.g., the effects on retirement age of reductions in initial Social Security
benefits). Because a cell-based model reflects differences among groups, the
aggregate outcome it projects may differ greatly from the outcome of a simple
time-series projection for the population as a whole.
OCR for page 134
34
ASSESSING POLICIES FOR RETIREMENT INCOME
The Office of the Actuary in the Social Security Administration (SSA) uses
cell-based techniques to develop 75-year projections of the balance in the trust
funds. Such techniques are also used in actuarial models of employer pension
fund balances and in models used by the Census Bureau, SSA, and others to
develop population projections.
Microsimulation Models Microsimulation models use individual records
for large samples of the population, estimating outcomes for each individual on
the basis of that person's characteristics and aggregating the results. Such mod-
els usually employ stochastic or probabilistic methods to generate distributions
for such outcomes as retirement decisions.
DYNASIM2 and PRISM are two dynamic microsimulation models that have
been used to evaluate alternative Social Security and employer pension policies.
They use dynamic aging techniques to project their samples forward in time. In
dynamic aging, characteristics of persons in the sample are altered year by year
through the application of transition probabilities to simulate who in the sample
will live, die, marry, divorce, stay on the same job, take another job, retire, and so
on. Policy simulations are then performed on the samples for 1 or more years.
The transition probabilities may come from a variety of sources, including time-
series analysis (e.g., for life expectancy) and behavioral analysis (using simple or
complex analytical models). Thus, for example, a behavioral equation in
DYNASIM2 relates labor force participation to age, race, sex, education, disabil-
ity, marital status, presence of children, and spouse's earnings and includes a
complex error term by which labor force participation changes for an individual
are linked across time and with changes in hours and wages.
Another microsimulation projection technique is called static aging, in which
the characteristics of the sample are altered by adjusting the sample weights to
reflect the expected distributions in the projection year before conducting policy
simulations. This method is typically employed for short-range projections (e.g.,
of tax liabilities or welfare program participants in the next 5 years) and not for
the longer range projections that are often required for retirement-income-related
policy modeling.
Microsimulation models can estimate the effects of more detailed policy
changes for a larger number of groups than is possible with cell-based models. In
addition, Burtless (1996) argues that microsimulation provides an organizing
framework. It requires model developers and users to think through all of the
behavioral links and interactions. It also helps them identify gaps in research
knowledge and data and set priorities for filling in key gaps. However, micro-
simulation models have costs as well as benefits, which must be evaluated vis-a-
vis other modeling approaches in the context of users' information needs. In
particular, they tend to be complex and difficult to use.
Both microsimulation and cell-based models can be developed for organiza-
tions (e.g., employers) as well as for individuals. However, with the exception of
OCR for page 135
DEVELOPMENT OF PROJECTION MODELS
135
the Pension Insurance Management System (PIMS) model recently developed
for the Pension Benefit Guaranty Corporation (PBGC) (see Appendix C), we
know of no policy projection models of retirement-income-related behavior of
employers.
Macroeconomic Forecasting Models Macroeconomic forecasting models
are used to make projections for the U.S. economy. They use systems of simul-
taneous equations, estimated with historical time series, to forecast the effects of
aggregate factors, such as rising inflation or changes in the federal budget deficit,
on aggregate outcomes, such as gross national product or unemployment.
Computable General Equilibrium (CGE) Models CGE models estimate the
longer run effects of proposed policy changes on prices and quantities in markets,
taking into account behavioral responses and feedback effects between supply
and demand. For example, an increase in marginal tax rates may decrease work
effort, thereby leading to reduced labor supply, thereby leading to higher wage
rates and increased labor supply until a new equilibrium is reached. As generally
implemented, CGE models are macro rather than micro in nature, in that they are
not usually disaggregated on many dimensions. However, unlike macroeco-
nomic forecasting models, CGE models are based on explicit theories of the
behavior of households and employers rather than on relationships among aggre-
gates.
Dimensions of Models
There are three key dimensions on which to evaluate projection models:
· the degree to which the model provides accurate estimates of policy out-
comes, in that the estimates approximate what would occur if a proposed policy
change were implemented;
· the amount of information provided in the model output on the uncer-
tainty of the model estimates and their sensitivity to key assumptions; and
· the degree to which the model incorporates best current professional judg-
ment about the underlying behavior for example, the best judgment about the
appropriate functional form and parameter values with which to estimate changes
in personal savings in response to changes in employer pensions or Social Secu-
rity.
Obviously, it is highly desirable that models be accurate, provide estimates
of uncertainty and sensitivity to key assumptions, and incorporate best profes-
sional judgment (or be able to use alternative specifications to reflect important
disagreements). But determining the accuracy of model outputs is a difficult task
at best. Also, models have to date rarely included the capabilities with which to
OCR for page 136
136
ASSESSING POLICIES FOR RETIREMENT INCOME
estimate uncertainty, conduct sensitivity analyses, or incorporate alternative
model specifications; consequently, little model validation work has been done.
With recent advances in computing technology and statistical methods, it is now
possible to effect significant improvements in model capabilities in these areas
(see "Validation" below).
Other dimensions of projection models include:
· scope, in terms of the types of policies that a model can simulate and the
kinds of outcome measures for which it can provide information;
· detail, in terms of how well a model can represent all of the provisions of
current law and estimate the effects of changes to one or more provisions;
· extent of disaggregation of model outputs, that is, the extent to which the
model can estimate the distributions of outcomes for groups of interest, variously
defined, in addition to average tendencies;
· time period, whether the model can develop projections for 5 years or 75
years;
· extent of feedback loops, which permit estimating not only the first-order
effects of a policy change, but also the second-order effects, taking into account
behavioral responses to the new policy;
· ability to link to other models, either by supplying outputs for other mod-
els to use as inputs or by accepting other models' outputs as inputs;
· amount of elapsed time that is required to obtain outputs from the model
on one or more policy options, including options that are close variants of current
policies and options that differ markedly in one or more respects;
· extent of openness and transparency, that is, how usable and understand-
able the model is to people other than the developers;
· ease of use and modification as needed; and
· extent of portability, from one computing hardware and software environ-
ment to another.
The first five dimensions listed above scope, detail, disaggregation, time
period of projections, and feedback loops have to do with the breadth and depth
of model capabilities. Other things being equal, it is desirable to have a model
that can simulate a broad range of policy options and, for each option or combi-
nation of options, estimate a broad range of outcomes. Other things being equal,
it is also desirable to have a model that can evaluate proposals for fine tuning one
or more aspects of complicated sets of program provisions, that can develop
projections over long time horizons, and that can provide distributions of outputs
for particular groups of interest in addition to population means. As an example,
in considering a policy change that increases or decreases employer pension
benefits, decision makers are likely to be interested in the effects not only for all
workers, but also for workers in different industries, age groups, and earnings
categories and by size-of-employer categories. Finally, a projection model is
OCR for page 137
DEVELOPMENT OF PROJECTION MODELS
137
more useful to the extent that it can estimate the second-round effects of policy
changes, which, because of behavioral responses, are likely to differ from the
initially estimated effects. For example, changes designed to increase employer
pension coverage, if they also increase employer costs, could lead employers to
opt out of pension plans and hence reduce coverage over time.
Desirable though these dimensions are, there are costs to increasing the
breadth and depth of model capabilities. For example, models that simulate a
wide range of policy outcomes and options are likely to be time and resource
intensive to develop and apply. Also, modeling of feedback effects is quite
difficult, both operationally and because of the uncertainty of behavioral re-
sponses to policy changes. Hence, it may be cost-effective to have different
models for answering questions about particular types of policy changes, as well
as different models for estimating means versus distributions, effects over shorter
versus longer periods, and second-order as distinct from first-order effects. In-
deed, the various types of modeling strategies listed earlier, such as cell-based
and microsimulation approaches, reflect choices about how much and what kinds
of detail to include in models. With a range of models, it becomes critical to
consider ways to facilitate links between them (the sixth dimension listed).
The last four dimensions listed response time, openness and transparency,
ease of use, and portability have to do with operational features that determine
how readily a model can be applied during the course of a policy debate, modi-
fied as needed to incorporate additional capabilities, and validated and improved.
These dimensions, which cut across model types, also have to do with how large
a user community develops for a model, which, in turn, is a major factor in how
widely a model is used and how rigorously it is evaluated.
Features of models that facilitate a higher level of performance on these
dimensions include adequate documentation, public availability of the model to
analysts, a user-friendly interface that allows people who are not programmers to
operate the model, and other design elements that permit ready modification of
the model to simulate newly proposed policy options in a consistent manner and
to facilitate multiple runs and specifications for model validation purposes. Many
of these features (e.g., real-time response mode or a graphical, point-and-click
user-friendly interface) appear more obtainable when a model is designed for
microcomputer or workstation computing environments than when it is designed
for mainframe or supercomputing environments.
It clearly seems important for models to provide timely response and to be
open, easy to use, and portable, whatever their type or breadth and depth of
capabilities. However, the costs of achieving high levels of performance on these
dimensions must also be considered. As an example, it is typically time and labor
intensive to produce good documentation, although, with forethought, it is pos-
sible to build features into models to facilitate the documentation process. Also,
the nature of an agency's information requirements may dictate trade-offs be-
tween, say, the ease of use of a model and its capabilities. For example, a
OCR for page 138
138
ASSESSING POLICIES FOR RETIREMENT INCOME
particular set of applications may require computing power that is not available
from microcomputers or workstations (e.g., CGE models may require super-
computers to attain the level of desired detail of inputs and outputs). In this case,
the only modeling option is to invest in a model that has limited access. An
alternative is to implement a simpler version of the model on a microcomputer
and a more elaborate version or a version that will process larger numbers of
sample cases on a supercomputer.
Assessment
There are many projection models available today that relate in some way to
retirement income security. Some of these models were developed for very
narrowly targeted applications, such as actuarial cell-based models that are de-
signed to project the balance of assets and liabilities for a specific company's
pension plan. Although these kinds of models serve important purposes, we have
not reviewed them because of their lack of generality. Also, our review does not
extend to health care financing projection models, although such models are very
important to a full assessment of retirement income security, nor to tax revenue
models, such as that used by the U.S. Treasury Department.1 Rather, we have
looked primarily at models that can address somewhat broader retirement-in-
come-related policy questions, for example, the effects of tax policy changes on
accumulations in Individual Retirement Accounts (IRAs) and pension accounts
over people's working lives.
Our overall conclusions about existing models are mixed. Some models
appear to have useful features for some kinds of applications but, on balance, we
conclude that existing models are not adequate to fully inform the retirement
income policy debate.
For some important topics, there are currently no available relevant models.
Specifically, there are no models of employer behavior with regard to benefit
offerings or demand for older workers that could provide input for individual-
based models of labor supply and retirement asset accumulation. Rather, existing
employer-based models address such issues as pension funding effects on share
prices and employer pension policy within a corporate financial setting. The new
PIMS microsimulation employer-based model has been developed for the PBGC,
but it has a limited focus. PIMS estimates corporate bankruptcies among em-
ployers with defined benefit pension plans on the basis of simulating key eco-
nomic variables and employer characteristics. It also simulates plan participants
and funding status. Finally, it simulates the implications of projected bankrupt
1See Office of Technology Assessment (1994) for an evaluation of models that were used in the
recent health care reform policy debate.
OCR for page 139
DEVELOPMENT OF PROJECTION MODELS
139
cies of companies with underfunded plans for the PBGC fund that insures such
plans.
For individuals, there are microsimulation models that are broad in scope
and provide the capability for detailed, disaggregated analysis. They could be
very useful for retirement-income-related policy analysis, but they generally suf-
fer from several deficiencies: they have not been adequately validated and are
not designed to facilitate validation; they are based on old or limited data sets;
they do not reflect best current professional judgment about underlying behavior;
they have little facility for implementing alternative behavioral specifications
when there are important disagreements; their provision for feedback loops be-
tween policy changes and behavioral responses is weak; they generally have little
provision for ready exchange of inputs and outputs with other models; they are
not well documented; and they are not open and transparent or readily portable,
although they now are generally implemented with personal computing technol-
ogy.
CGE models that have been developed to answer such questions as the
intergenerational benefits and costs of alternative Social Security financing sys-
tems are impressive in their treatment of the interactions of the household, gov-
ernment, and private sectors of the economy and in their use of up-to-date infor-
mation and research knowledge. However, these models deal primarily with "big
picture" issues in a highly stylized manner and primarily for the Social Security
system, not the much more heterogeneous world of employer pensions. Also, to
date, these models have not addressed the limitations of the life-cycle model of
savings and consumption (see Chapter 3), and they are usually structured as
equilibrium models without unemployment, particularly long-term unemploy-
ment, among older workers. These models do not provide distributional detail,
nor can they address the effects of very specific policy changes or combinations
of changes. Finally, there are as yet no links between these models and cell-based
or microsimulation projection models that would provide additional specificity.
A positive trend is that newer models generally follow design principles that
encourage use and validation. For example, PIMS, the employer-based model,
operates on personal computers and has a design that facilitates model validation.
However, newer models, such as PIMS, SSASIM, the Solvency and Individual
Retirement model (SIR), and the model under development by Wolf et al. (1995),
tend to be limited in scope. SSASIM is a cell-based model similar in structure to
the SSA actuarial cost model that uses Monte Carlo methods to characterize
uncertainty about forecasts of Social Security trust fund balances. SIR is a cell-
based model similar to the SSA actuarial cost model that provides limited distri-
butional information for different types of workers. Wolf is developing a micro-
simulation model primarily for simulating kinship networks and disability status
of the elderly.
Because individual-level microsimulation models potentially offer very im-
portant capabilities for retirement-income-related policy analysis, we review as
OCR for page 140
140
ASSESSING POLICIES FOR RETIREMENT INCOME
pects of two of the existing dynamic models DYNASIM2 and PRISM in
Appendix D. The discussion focuses on some weaknesses of these models, not to
single them out for criticism, but to support our case for important areas of future
development for this type of model.2
LOOKING TO THE FUTURE
We recommend priority attention to data collection and research and analytical
modeling over the next few years rather than to significant investments in broad-
based projection models. However, we believe it is important to establish goals
for projection model development and, in some instances, to move forward on
design work, so that implementation can progress expeditiously once invest-
ments in data and research bear fruit. Specifically, the federal government should
establish goals for the development of employer models (using cell-based or
microsimulation techniques) and for a new general-purpose individual-level
microsimulation model. Other retirement income modeling tools are needed as
well, but it is clearly important to have the projection capabilities that employer
models and a new individual-level microsimulation model can provide.
Employer Models
Employer hiring and compensation policies are critical to retirement income
security, most obviously affecting the extent of pension coverage and benefits,
but also affecting Social Security benefit levels and the opportunities to contrib-
ute to such earnings-linked savings vehicles as IRAs. Employers also often
provide other important benefits, such as retiree health care and disability insur-
ance, as well as opportunities for part-time, post-retirement work. A full assess-
ment of the likely effects of many retirement-income-related policy proposals
will not be possible without considering what is happening to and with employ-
ers. Shifts in employment across sectors that have historically differed in wage
and benefit levels can affect retirement income security, as can employer re-
sponses to changes in policies and other factors that alter incentives (e.g., pension
simplification may increase the probability that small businesses will set up
pension plans).
Employer-based projection models are clearly needed for much retirement
income policy analysis. In some instances, decision makers will want informa
2For other assessments of microsimulation models, see Citro and Hanushek (1991); the papers
cited in Cohen (1991a); Employee Benefit Research Institute (1980a, 1980b, 1981); Reno (1993:29-
31); Ross (1991); and U.S. General Accounting Office (1986). Burtless (1989) and Grossman and
Watts (1982) focus on behavioral elements of microsimulation models. Anderson (1996) and
Hollenbeck (1995) review microsimulation and other projection models that are relevant for retire-
ment income policy modeling.
OCR for page 141
DEVELOPMENT OF PROJECTION MODELS
141
lion about how many employers will take a certain action within a specified time
period (e.g., set up a pension plan) and how many workers will be affected. Such
information could most likely come directly from an employer model that in-
cludes such characteristics as industry and size. In other instances, decision
makers will want to know the longer term implications for retirement income
flows (e.g., from pension accumulations) for different groups of workers; for this
information it will most likely be necessary to produce outputs from employer
models that can feed into individual-level models for workers and their families.
At present, there are no models that can project the distribution of employer
pension and related benefit offerings or the distribution of employer demand for
workers. Moreover, there is little basis on which to develop such models in the
near term: the needed data and research knowledge are simply lacking (see
Chapters 3 and 4~. Resources spent on model development cannot be cost-
effective until more complete data are available (e.g., about the distribution of
employer benefits) and until there is much better understanding of the factors that
influence hiring and compensation decisions for different kinds of employers.
Gaining such understanding will require considerable elaboration and testing of
alternative theories about employer behavior. The development of employer-
based projection models is an important goal for the Department of Labor and
other agencies, but one that can only be realized over the long term.
It is important to keep this long-term model development goal in mind as
decisions are made about investments in employer-related data and research. In
this regard, it could be useful to consider alternative projection model designs
and strategies and their requirements. One question is whether to start with a
cell-based design that classifies employers on a limited number of important
dimensions or to opt for a microsimulation model. A cell-based model would be
easier to implement and have fewer data requirements than a microsimulation
model, but it would have more limited capability for analysis and for simulation
of behavioral response.
It could also be useful to bring together researchers and people with experi-
ence in developing projection models to consider possible model designs on the
basis of anticipating what research results may show. For example, what would
be the design and data requirements for a projection model if employer costs are
taken to be the key factor in hiring and compensation policies? What would be
the requirements if employee preferences are also deemed to be an important
factor?
As new data about employers become available and are analyzed, DOL
could sponsor workshops to discuss research findings and their implications for
the development of employer-based projection models. Clearly, to make progress
toward the goal of model development, it will be important for DOL to give
priority to the work of the interagency task force on employer data (see Recom-
mendations 5-9, Chapter 4) and to putting an employer data collection plan on as
fast a track as possible.
OCR for page 142
42
ASSESSING POLICIES FOR RETIREMENT INCOME
Recommendation
14. The U.S. Department of Labor should consider the development of
employer models as an important long-term goal. Retirement-income-
related policy analysis requires models of employer behavior that can project
the likely future distribution of benefit plan offerings and the demand for
older workers. However, there is little basis on which to design, let alone
implement, such models until investments have been made in needed data
and behavioral research.
Microsimulation Models of Individuals and Families
Policy makers often want to know the effects of proposed policy changes on
population groups, categorized by age, sex, race or ethnicity, income level, and
the like. Cell-based models can provide distributional detail, but the cells must
be defined in advance and are difficult to change in response to user needs. In
principle, large-scale microsimulation is a much more flexible tool with which to
provide detailed information about distributional effects, limited only by sample
size constraints on the reliability of small cell estimates. Broad-based micro-
simulation can also address important interactions among policies, such as how a
change in Social Security benefits will likely affect benefits from employer pen-
sion plans that are linked to Social Security. Dynamic microsimulation can in
principle forecast the long-term effects of policy changes, including behavioral
responses: for example, how a change in Social Security benefits will likely
change retirement and savings behavior and, ultimately, affect retirement income
levels for different cohorts of workers.
We believe that a general-purpose, dynamic microsimulation model is needed
to project the distributional effects of policy alternatives on the retirement income
security of individuals and families over the medium term and for modeling
interactions among policies. Other, more limited modeling tools are often appro-
priate and cost-effective for specific policy questions. However, as Burtless
(1996) argues, it is important to have the capability to address all three legs of the
retirement income stool Social Security, employer pensions, and personal sav-
ings in an integrated modeling framework that can provide needed distribu-
tional detail.
Such a modeling framework is also useful because it can help structure
related analytical work. The effort to develop and apply a large-scale micro-
simulation model will invariably identify behavioral interactions and processes
that need to be better understood. It will also help determine which parameters
are crucial for analysis and which are less important, and it can suggest how
concepts and variables should be consistently defined and measured to be useful
for modeling purposes.
Unlike the situation for employer models, there are existing individual-level
OCR for page 154
154
ASSESSING POLICIES FOR RETIREMENT INCOME
to conduct "ex post forecasting" studies, in which one puts oneself in the place of
an analyst who, say, 10 or 20 years ago, was asked to simulate a policy change to
take effect in some future year. One chooses the "future" year to be in the recent
past so that measures of what happened are available from administrative records
or other sources. Correspondingly, one chooses the policy change to be the
actual policy in effect during the comparison year. (An alternative approach is to
use a current model to develop "backcasted" estimates of policies for earlier
years.)
Because differences between a model' s estimates and what actually occurred
may be due to economic or social changes that the model could not have been
expected to forecast (e.g., an unanticipated recession or an unexpected increase
or decrease in fertility rates), it will be helpful to conduct ex post forecasting (or
backcasting) studies that use actual values for the factors that could not have been
anticipated and focus on the accuracy of the model with regard to other elements.
Also, in many instances, the important criterion will be whether the model accu-
rately projected the differences in the outcomes under two or more policy alterna-
tives and not the more stringent criterion of whether it accurately projected the
levels of outcomes for a specific alternative.
In addition to studies that help establish confidence in a model's track record,
it is important to develop estimates of the range of uncertainty of a model's
estimates. On average, a model may produce unbiased estimates, but the results
of any particular model run may have a wide band of uncertainty around them.
The sources of error in model simulations can be summarized in four broad
categories: (1) sampling variability in the input database, which is only one of a
family of possible data sets that could have been obtained; (2) sampling variabil-
ity in other inputs, such as imputations, regression coefficients, and control totals;
(3) errors in the database and other inputs; and (4) errors due to model
. . ^. .
m~sspecl~lcatlon.
Advances in statistical methodology that take advantage of modern comput-
ing power have made it possible to estimate the uncertainty in model estimates
due to such sources as sampling variations in the input data. One such technique
is the bootstrap (see Cohen, l991c), which, simply put, measures variability by
using the observed sample distribution in place of the unobserved population
distribution. The strength of this approach is that variance estimation becomes a
function of computing power rather than an exercise in solving multidimensional
integrations for complex estimators and distributions.
For complex models, like dynamic microsimulation models, sample reuse
techniques can be used to measure the uncertainty in a model's estimates due to
the two kinds of sampling variability. However, the uncertainty due to the two
kinds of errors may be difficult to put into a probabilistic context. Sensitivity
analysis, which is carried out by developing and running one or more alternative
versions of one or more model components, can help assess the uncertainty in
model estimates due to the choice of a particular model structure or specification.
OCR for page 155
DEVELOPMENT OF PROJECTION MODELS
155
For example, one could use sensitivity analysis to investigate the effects of a
particular equation for simulating the labor supply response to a change in the
retirement age for Social Security or employer pensions. In its simplest terms,
sensitivity analysis is a diagnostic tool for ascertaining which parts of an overall
model could have the largest impact on results and therefore are the most impor-
tant to scrutinize for potential errors that could be reduced or eliminated (see
Citro and Hanushek, 1991:89-96; Chap. 9~. Sensitivity analysis is particularly
important for retirement income modeling in view of the long time horizon for
most projections, which will compound any errors in model specification or
assumptions.
For various reasons, external validation studies, estimation of uncertainty,
and sensitivity analysis have rarely been carried out for projection models.6 His-
torically, one reason was that such studies were very costly and time-consuming.
However, if models are implemented in computing environments that provide
ready, cost-effective access to users for multiple runs and model respecifications,
then validation becomes very feasible.
Another reason there are few estimates of uncertainty is that decision makers
usually want only point estimates, without measures of uncertainty. It is impor-
tant, therefore, to develop innovative ways to convey estimates of uncertainty
that decision makers will take into account in debates about policy alternatives
(see Citro and Hanushek, 1991:86-88~. This task may be easier for retirement-
income-related policy projections because decision makers have somewhat more
experience with estimates of uncertainty for such projections than for estimates
on other topics. Thus, because of the long time horizon, the SSA Office of the
Actuary routinely presents high-cost, intermediate, and low-cost scenarios for the
balance in the Social Security trust funds under current law. However, this
approach to bounding uncertainty is problematic (see "SSA Model Validation,"
below). Moreover, there is no comparable record of regularly projecting the
long-term funding status of employer pensions, even though this source of retire-
ment income may be more uncertain than Social Security (see Burtless, 1993;
also, Schieber and Shoven, 1993, develop such a projection for defined benefit
and defined contribution plans).
Finally, an essential feature of an acceptable projection model is that it
incorporate best professional judgment about relevant factors, particularly for
aspects of the model to which estimates are sensitive. "Best" professional judg-
ment is not always the newest theory or latest academic model, which may be too
new to have been properly evaluated. Also, because there is a range of profes-
sional opinion in many areas, it will be important to conduct sensitivity analyses
of alternative specifications before selecting a particular approach to use for a
6See Cohen (1991a) for a review of validation studies that have been done of social welfare policy
analysis microsimulation models, including DYNASIM2 and PRISM.
OCR for page 156
156
ASSESSING POLICIES FOR RETIREMENT INCOME
central or base case. Yet it is clear that when important changes have occurred-
for example, in worker productivity or savings rates or mortality rates a model
that uses data or estimates of behavioral parameters that do not reflect those
changes will not be helpful in informing the policy debate. Also, given that
relatively little work has been done to date to conduct external validation studies
or sensitivity analyses of models, the extent to which a model incorporates best
professional judgment is often the only indicator of the likely level of model
performance.
SSA Model Validation
The Social Security actuarial cost model, which uses cell-based methods to carry
forward the results of time-series analysis, is an exception to our statement that
model validation and assessment of uncertainty are rarely performed. The Office
of the Actuary's 75-year projections of trust fund balances under current law,
which are revised every year, are often evaluated by comparing them to what
actually occurred in the short run and by looking at the patterns of yearly revision
(see, e.g., Schieber and Shoven, 1993~. Also, the Office of the Actuary routinely
prepares trust fund balance projections for low-cost, intermediate, and high-cost
scenarios, thereby providing a type of sensitivity analysis (similar to that pro-
vided for Census Bureau population projections). SSA also provides sensitivity
analyses of the separate effects of key assumptions. In these analyses, the high-
cost, intermediate, and low-cost values are used for one assumption, such as
fertility, while the intermediate values are used for all other assumptions.
However, as Burtless (1993) points out, the assumptions underlying the low-
cost and high-cost scenarios are not chosen so as to bound the likely range of
potential experience in any manner that approximates a statistical confidence
interval. Hence, they are less useful to policy makers, analysts, and the public
because they do not indicate the probability of a best-case or worst-case outcome
relative to the intermediate projection. Also, the assumptions may be internally
inconsistent. For example, the growth rate of real wages (for which a higher
value is a favorable outcome for the Social Security system) may be negatively
correlated with real interest rates (for which a higher value is also a favorable
outcome for the system).
Finally, key assumptions of the central or intermediate scenario may not
always reflect current best professional judgment. For example, the mortality
forecasts developed by SSA are at one extreme of the range of available fore-
casts: they represent a more pronounced slowing of the rates of mortality decline
observed in this century than is forecast by others (see Lee and Skinner, 1996~.
The implication of the SSA forecasts is that people will not live as long as
forecast by others and, hence, that the growth in the elderly population, particu-
larly of the oldest old (those aged 85 and older), will be less pronounced. Conse-
quently, the use of the SSA forecasts, even from the high-cost scenario, could
OCR for page 157
DEVELOPMENT OF PROJECTION MODELS
157
result in overly optimistic assessments of the ability of employer pensions or
personal savings to provide adequate retirement income support over retirees'
life spans. The use of the SSA forecasts is also problematic for assessments of
the Social Security system, even though the relevant population estimate is not
the number of elderly per se but the old-age dependency ratio (people aged 65
and older as a ratio of people aged 20-64~. While a lower mortality rate than that
forecast by SSA will increase income to the system (more younger people will be
alive to contribute payroll taxes), it will also increase payouts from the system,
and the relative increase in outgo will exceed the relative increase in taxable
payrolls (see Board of Trustees [OASDIi, 1994: 132-133~.
Lee and Carter (1992) provide a credible alternative to the SSA mortality
forecasts. Their method essentially projects into the future the rate of mortality
decline observed over the twentieth century, which has been fairly steady despite
periods of faster and slower progress. SSA's method extrapolates cause-specific
death rates over the preceding 20 years, which necessarily results in a marked
slowing in the rate of mortality decline, as the more rapidly declining causes of
death come to claim a smaller and smaller share of total mortality. Of course, it
is impossible to know which of the two methods will ultimately prove more
accurate.
A major advantage of the Lee and Carter approach, noted by Lee and Skinner
(1996), is that it provides probability intervals describing the uncertainty in their
extrapolative method. However, the Lee and Carter estimates of uncertainty do
not allow for model specification error or the possibility of "structural breaks" in
time series, such as the sharp decline in age-specific mortality rates that occurred
between the nineteenth and twentieth centuries. Hence, Lee and Carter probably
underestimate the extent of uncertainty in their forecasts of mortality rates.
Recently, Lee and Tuljapurkar (1994) extended the methods of Lee and
Carter to develop stochastic population forecasts with consistent and meaningful
probability intervals for all items forecast (e.g., age categories of the population,
taking account of all demographic processes). Lee and Tuljapurkar formulated
models of age-specific fertility (from Lee, 1993) and age-specific mortality (from
Lee and Carter, 1992), in which they estimated a single time-varying parameter
for each process by using standard time-series analysis techniques. (Net immi-
gration could also be handled in the same manner, but it was treated as determin-
istic at the level assumed in the Census Bureau projections.) They then used
these fitted models of stochastically varying rates as inputs to the population
renewal process and carefully tracked the propagation and sources of error. They
used analytic methods to find both the expected value of the forecast and the
probability intervals.
Similar results could be obtained with Monte Carlo methods, that is, by
repeated stochastic simulations using the stochastic models of vital rates. The
SSASIM model, recently developed with support from SSA, uses Monte Carlo
techniques to characterize uncertainty in the Social Security trust fund balance
OCR for page 158
158
ASSESSING POLICIES FOR RETIREMENT INCOME
forecasts (Holmer, 1995a, 1995b; see also Sze, 1995, and Appendix C). SSASIM
implements stochastic processes for all of the key demographic and economic
variables that are used in the SSA actuarial cost model in a general framework
that can accommodate different expectations (specified by the user) about the
dynamics of each variable over time and in association with other variables. For
example, a policy analyst can assume that such variables as rates of mortality
decline follow a consistent trajectory over the span of the projection period (as
does the SSA actuarial cost model) or that the time trend will exhibit cyclical
fluctuations. The analyst's assumptions are used to specify the means, standard
deviations, and correlation coefficients of a multivariate normal distribution:
multiple runs with different random draws from the distribution for a particular
set of assumptions generate the expected values of variables and the probability
bounds.
The use of scenarios by SSA and the Census Bureau to convey uncertainty
about forecasts stands in contrast to the use of probabilistic methods. To build
scenarios, an analyst develops one forecast on the basis of the "expected" trajec-
tories for each demographic or economic process. The analyst then also develops
alternative trajectories for each process above and below the expected trajectories
and combines them in various ways to produce high and low forecasts with which
to bound the intermediate or expected forecast. For example, because trust fund
balances are influenced by the old-age dependency ratio, SSA combines low
fertility with low mortality for its high-cost scenario. The Census Bureau, on the
other hand, combines high fertility with low mortality into a high scenario for
total population growth. As a consequence, the Census Bureau's high and low
estimates of the elderly population cover a wider band than the SSA estimates,
while the Census Bureau's high and low estimates of the old-age dependency
ratio cover a narrower band.
The SSA high- and low-cost scenarios for the population aged 65 and older
and the old-age dependency ratio cover a somewhat narrower band than the 95
percent probability intervals generated by Lee and Tuljapurkar (1994~. One
could use the Lee and Tuljapurkar upper and lower bounds to produce new
population scenarios. However, as Lee and Skinner (1996) point out, the prob-
lem with any method of constructing scenarios is that there is no means by which
to attach probability intervals to the resulting forecasts. The scenario approach
assumes unrealistically that the high, medium, or low trajectory for each de-
mographic process is followed consistently over the entire time span. In contrast,
the stochastic forecasts of Lee and Tuljapurkar treat fertility and mortality as
random variables at every iteration, producing true probability bounds for the
projections at each period (e.g., each 5-year interval).
Although stochastic methods may underestimate the uncertainty in demo-
graphic and economic forecasts, depending on how the parameters of the stochas-
tic process are estimated, they represent a major step forward over the scenario
approach for conveying estimates of uncertainty to policy makers. The use of
OCR for page 159
DEVELOPMENT OF PROJECTION MODELS
159
stochastic methods to estimate the uncertainty of projections of demographic and
economic variables seems imperative for the kinds of long-range forecasts that
are often used in retirement-income-related policy debates. In this regard, it
would be very useful to further develop the capabilities of the SSASIM model for
estimating the uncertainty of the projections in the SSA model, including work on
alternative methods with which to estimate the parameters of the stochastic pro
cess.
Recommendation
16. Models that project policy outcomes over long periods should rou-
tinely assess the accuracy and uncertainty of the projections and report
prediction errors in previous forecasts. To be able to evaluate a model's
projections in a cost-effective manner, the model design must explicitly in-
clude a capability for sensitivity analysis (e.g., how much the outputs vary
with alternative assumptions and versions of model components) and a capa-
bility to estimate uncertainty from such sources as sampling variability in
the database and other model inputs.
NEAR-TERM MODELING STRATEGIES
The kinds of investments we recommend in data and research will take time to
return benefits for retirement-income-related policy analysis. In addition, the
development of needed employer models and a new large-scale individual-level
microsimulation model, which must await advances in data and research knowl-
edge, cannot be completed in the near term. In the meantime, the Department of
Labor and other agencies must be able to answer decision makers' questions
about the likely effects of alternative policies. They cannot respond that they are
unable to provide requested information because data and knowledge are lacking
with which to develop improved projection models.
In order to be responsive to decision makers' legitimate information de-
mands without incurring significant model development costs that could well be
misdirected, agencies should try to frame policy questions as specifically as
possible and use limited special-purpose models together with the best available
data and research knowledge to answer those questions. Depending on the issue,
a special-purpose model may already exist, or, more likely, it will be necessary to
adapt one from another model or to construct one on the spot.
Admittedly, limited ad hoc models may not provide very reliable answers to
policy questions, and, in particular, they are not likely to be able to handle
important policy interactions. Also, special-purpose models may be able to
model behavioral responses to policy changes only very crudely, although they
should incorporate such responses when reasonable parameters are available and
feasible to implement. However, the alternative of trying at this time to develop
OCR for page 160
160
ASSESSING POLICIES FOR RETIREMENT INCOME
more complex models, when needed data and research knowledge are lacking,
will not likely give a better result and will very likely waste scarce resources.
We acknowledge that a phased approach to retirement income modeling, as
we recommend, may not provide useful answers before there is actually a retire-
ment income crisis that requires extensive evaluation of alternative policy pro-
posals in a comprehensive framework. However, there will be continuing need
for retirement-income-related modeling capabilities (as is true for health care
policy), and without investments in data and behavioral research, government
agencies will never be in a position to develop markedly improved projection
models. (The health care agencies have recognized this point by redesigning and
expanding their surveys.) Again, we are concerned about the use of scarce
resources in the near term. We believe they are best used for data and research
instead of extensive development (or redevelopment) of large-scale models. To
the extent that limited investments can lead to better special-purpose models for
specific questions, we support those investments.
The CBO Approach
The modus operandi that we have in mind is that of the Congressional Budget
Office (CBO), whose analysts regularly prepare ad hoc, special-purpose models,
often using spreadsheet tools, to develop estimates of the likely costs and other
effects of proposed policy changes. CBO staff combine data and parameter
estimates in their ad hoc models from a wide range of sources, sometimes includ-
ing as inputs the outputs of larger, more formal projection models with the results
adjusted as seems appropriate. Sometimes they make additional "out-of-model"
adjustments, for example, to provide crude estimates of the likely effects of
behavioral response to policy changes. (See Citro and Hanushek, 1991:41-51,
for a case study of the modeling tools used by CBO and the U.S. Department of
Health and Human Services in the late 1980s for estimating the effects of welfare
reform.)
In general, CBO staff develop their models only to the point to which they
can answer the question at hand. For example, projections for a few years might
be developed with distributional detail, while longer term projections are devel-
oped on a very aggregate basis. Or a model might be developed to handle only
the particular policy change that is under consideration. However, in instances in
which the same kind of question arises frequently, CBO will develop more for-
mal models that are intended for repeated use and that can handle a wider range
of variations in a particular policy area. As just one example, CBO developed a
hospital-based model that is designed to estimate the cost and distributional
effects of changing various provisions of the Medicare prospective payment
scheme used to reimburse hospital costs.
The problem with an ad hoc approach to model building is that there is
virtually no incentive to provide for such desirable model features as openness or
OCR for page 161
DEVELOPMENT OF PROJECTION MODELS
161
portability or to document a model for use by others. More important, there is no
incentive to perform more than the crudest type of model validation. However,
the strategy of special-purpose model development, when carried out by knowl-
edgeable analysts, can provide answers to specific policy questions with mini-
mum staff resources and time. The answers will very likely be limited and of
highly variable quality, but they can assist the policy debate, and the process of
developing them can identify issues to consider in the design of new, improved
large-scale models for use in future policy projections.
The problem for retirement-income-related projection modeling for the ex-
ecutive branch in the near term (and in the long term, too) is that there is no
equivalent of CBO in a cabinet department. CBO will itself prepare estimates for
retirement-income related policy proposals, but the executive branch needs its
own modeling capability, particularly when proposals are being drafted and have
not yet been sent to the Congress. There is no staff group in the executive branch
that is in a position to take a broad view of policy analysis in this area, much less
to make effective use of available data, research, and modeling tools to develop
the best possible estimates in the most direct manner in response to decision
makers' questions. There are clusters of analysts in such agencies as DOL and
SSA, but it would be helpful to have a larger critical mass working together on a
regular basis.
For example, SSA analysts are currently working on models of the distribu-
tion of Social Security benefits by using matched SSA earnings records and SIPP
data. The SSA analysts initially developed models with which to project earnings
histories for 1984-1993 for members of the 1984 SIPP panel on the basis of prior
earnings histories, and they obtained reasonable results. They then used these
models to project future earnings histories for members of the 1990 SIPP panel
(lams and Sandell, 1996; Sandell and lams, 1996~. The SSA analysts propose to
extend their models to project pension income, as well as Social Security ben-
efits, on the basis of earnings histories and other characteristics of SIPP sample
members. If successful, and if a savings income projection capability is also
added, this approach could offer some useful policy modeling capability without
requiring a full-fledged dynamic microsimulation model (assuming that policy
changes can be factored into the projections in some manner).
SSA analysts are also working on cross-sectional microsimulation models of
SSI and the disability insurance (DI) component of the Social Security system
with matched SIPP and disability program data from SSA records. As part of this
effort, they are developing ways to simulate the multistage disability application
and determination process (see Lahiri, Vaughan, and Wixon, 1995~. Initially
they expect to be able to make estimates of the number of people in the civilian
noninstitutional population who are likely to meet SSA disability criteria. Esti-
mates of those categorically eligible on the basis of disability will then be com-
bined with estimates of financial eligibility for SSI and DI insured status. A
subsequent model of the application decision is also planned and will be used to
OCR for page 162
162
ASSESSING POLICIES FOR RETIREMENT INCOME
introduce selectivity adjustments to the estimates of the pool of those eligible for
disability benefits.
This SSA modeling work should be well known and available to other ana-
lysts, such as those in the Pension and Welfare Benefits Administration in DOL.
Similarly, work that PWBA analysts may do with employer pension issues should
be well known and available to analysts in other relevant agencies. It is probably
not feasible to establish a cross-agency retirement income modeling group with
authority to prepare estimates on behalf of the executive branch, but perhaps it is
possible to establish coordinating mechanisms. For example, it would be useful
to establish an interagency users group that meets regularly to review retirement-
income-related projection model developments at the member agencies. Such
meetings could contribute to the design of new, improved models in the future.
Improvements to Existing Models
In the near term, while large investments in projection model development are in
abeyance, it is important to seek opportunities for cost-effective improvements to
existing retirement-income-related projection models and to make them known
and available across agencies. For example, it may be possible to marginally
improve the pension components of such models as DYNASIM2 and PRISM,
even though major improvements in pension simulation capability will depend
on the development of new models that use greatly improved data.
An important model for which it appears feasible to effect useful improve-
ments in the near term is the SSA actuarial cost model. The SSA model is the
linchpin of retirement-income-related projection modeling. It produces rela-
tively few outputs (principally, the trust fund balances over a 75-year time hori-
zon), but its projections are critical in setting the tone for policy debates. Also,
many other models calibrate their outputs to components of the SSA projections,
such as the forecasted population size by age and sex and wage growth and other
economic factors.
We urge attention to extending the SSA model in several ways.7 First, it is
critically important to develop probabilistic estimates of uncertainty for each of
the variables projected in the model and then to develop estimates of uncertainty
of the model's outputs as a whole, taking account of covariances among the
various time series (mortality, labor force participation, wage growth, etc.~. Such
estimates should replace the current scenario approach, which has no probabilis-
tic structure. Work is under way at SSA along these lines (see the previous
discussion in "SSA Model Validation"), and we urge that it be continued.
Second, it would be highly useful to add a capability to the model for limited
7The 1994-95 Advisory Council on Social security Technical Panel on Assumptions and Methods
(1995:60-66) has made similar recommendations.
OCR for page 163
DEVELOPMENT OF PROJECTION MODELS
163
distributional analysis of the effects of current and proposed Social Security
provisions. For example, it would be useful to be able to project costs and
benefits of policy alternatives for low, middle, and high earners among the popu-
lation. Such projections would be helpful in evaluating outputs of more disaggre-
gated models, and vice versa. A distributional component of the SSA model
should include a capability to simulate the effects of cyclical or random varia-
tions in key aggregate variables, such as interest rates and growth in wages.
Third, because of the importance of mortality projections for estimates of the
retirement-age population, we urge SSA to evaluate the sensitivity of the model's
outputs to mortality projections that are disaggregated by such characteristics as
marital status and income levels. If the analysis suggests a strong degree of
sensitivity, it would be important to undertake research on the best form of such
projections for use in the model. SSA administrative data could provide the basis
for a comprehensive study of the relationship of mortality to earnings for people
of retirement age. (Studying the relationship of mortality to marital status with
these data would require SSA to track the marital status of all beneficiaries,
which could also be useful for projections of beneficiaries.) SSA also has several
SIPP panel files to which it is appending year-by-year mortality records. These
files could permit a rich, multivariate analysis of mortality correlates for a limited
sampled
Finally, it would be highly useful to develop means by which to document
and provide research access to the model. Outputs of the SSA actuarial cost
model are so widely used for inputs to other models that it would be very helpful
if outside analysts could more readily learn about and have access to the model.
Such access would also facilitate outside reviews of the model's properties to
identify other improvements.
Recommendations
17. To respond to immediate policy needs, agencies should use limited,
special-purpose models with the best available data and research findings to
answer specific policy questions. Although such models may not provide
very accurate estimates, the alternative of developing complex new indi-
vidual-level microsimulation or employer models in advance of needed im-
provements in data and research knowledge has little prospect of producing
better results and will likely represent, in the immediate future, a misuse of
scarce resources.
8Panis and Lillard (1996) recently developed mortality models with data on age, race, sex, marital
status, and household income of surviving sample members and decedents in the PSID. They used
these models to simulate the effects of proposed Social Security changes, such as increasing the
contribution rate and decreasing benefits, on the net present value of Social Security benefits for
prototypical types of beneficiaries.
OCR for page 164
164
ASSESSING POLICIES FOR RETIREMENT INCOME
18. The Social Security Administration should consider enhancements
to the SSA actuarial cost model in the near term. This model provides useful
outputs for other models and helps frame the overall retirement income
security policy debate. Priority improvements include the following:
· replacing the scenario approach with probabilistic estimates of un-
certainty for the model's projections that take into account covariances
among the various time series (mortality, labor force participation, wage
growth, etc.~;
· adding a capability for limited distributional analysis of the effects of
current and proposed Social Security provisions (e.g., of costs and benefits
for low, middle, and high earners);
· evaluating the sensitivity of the model's outputs to mortality projec-
tions that are disaggregated by such characteristics as marital status and
income levels and planning to develop such projections, if warranted; and
· developing means to document and provide research access to the
model.
Representative terms from entire chapter:
retirement income
