National Academies Press: OpenBook
« Previous: Sensitivity Analysis
Suggested Citation:"External Validation." National Research Council. 1991. Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers. Washington, DC: The National Academies Press. doi: 10.17226/1853.
×
Page 314
Suggested Citation:"External Validation." National Research Council. 1991. Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers. Washington, DC: The National Academies Press. doi: 10.17226/1853.
×
Page 315
Suggested Citation:"External Validation." National Research Council. 1991. Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers. Washington, DC: The National Academies Press. doi: 10.17226/1853.
×
Page 316
Suggested Citation:"External Validation." National Research Council. 1991. Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers. Washington, DC: The National Academies Press. doi: 10.17226/1853.
×
Page 317
Suggested Citation:"External Validation." National Research Council. 1991. Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers. Washington, DC: The National Academies Press. doi: 10.17226/1853.
×
Page 318

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

EVALUATING THE ACCURACY OF U.S. POPULATION PROJECTION MODELS 314 External Validation Population projections are by their nature conditional on the chosen level of fertility, mortality, and immigration assumptions. However, users often interpret the projections as unconditional forecasts, especially if the numbers are being used for planning purposes. Consequently, these projections should be validated against actual population outcomes. Comparing projections to the subsequent reality appears trivial, but a few difficulties must be handled first. To begin with, projections into the distant future are almost sure to vary from the truth more so than projections for next year. It is for this reason that projections fan out—the range of acceptable values tends to increase over time. This process can be seen in reverse by considering projections of the mid-year 1990 U.S. population made at various points in the past. Figure 3 plots Census Bureau projections since 1963, including high and low projections. Certainty about the 1990 total increases as 1990 approaches. There is some question regarding the appropriate measure of error in evaluating a population projection. A discrepancy in the absolute number of people is a misleading statistic because it depends heavily on population size, which varies across countries, regions, and time. Percent error does not take into account the period of time covered. Also, it considers only variations in the final population; in fact, the base population size may have been incorrect, an error that (for purposes of this discussion) is outside the realm of the projection itself. For this reason a preferred measure is the difference in average growth rates—the average rate implied by the projection minus the one actually observed. This statistic, ∆r, is defined as where and represent the projected size of the population for time T and the size of the base population used in the projection, respectively; PT and P0 represent actual population sizes at times T and 0; and represents the average annual growth rate between 0 and T. This statistic summarizes the error in the projection of total population but does not reflect errors in the age or geographical distribution (Stoto, 1983). Before considering external validations of Census Bureau and SSA national projections, it is worthwhile to review some of the broader literature comparing cohort component projections (which are used by both agencies) with alternative methodologies. If the age composition of the population is of interest, it is difficult to improve upon the performance of the cohort component calculations and extensions of them. However, if the only statistic of interest is total population size, evidence regarding which method is more accurate is mixed. Smith (1984) considers 5-, 10-, and 20-year projections of the populations of individual states in the United States. He evaluates simple extrapolations of

EVALUATING THE ACCURACY OF U.S. POPULATION PROJECTION MODELS 315 past growth, trends in the ratio of state population to U.S. population, cohort component models, and economic- demographic models that include economic inputs to predict separately the components of growth. In terms of total population the cohort component model fares poorly. Smith (1984:29) claims that FIGURE 3 Estimated U.S. population in 1990 (250 million) and projections to 1990 beginning in 1963. NOTE: For a discussion of scenarios, see text. SOURCE: Data from Spencer (1989). there is a common perception among both producers and users of population projections that forecast accuracy must improve as models become more sophisticated and complicated. There is as yet no evidence to indicate that this perception is true. The results presented…show the simple extrapolation and ratio projections to be every bit as accurate as the economic-demographic and cohort projections. Numerous other authors have reached similar conclusions. Smith goes on to cite Hajnal (1955), Greenberg (1972), Siegel (1972), and Ascher (1978) in support of his point. Stoto and Schrier (1982) draw the same conclusion using slightly different methods; they argue that the simpler techniques are preferable because they do not involve judgments that may become political. Beaumont and Isserman (1987), on the other hand, point out that any argument in favor of simpler methods must be weak, given that no rigorous comparative test exists. Keyfitz (1981) examined 1,100 projections made at different times for numerous countries by various agencies, primarily the

EVALUATING THE ACCURACY OF U.S. POPULATION PROJECTION MODELS 316 United Nations. Comparing a subset of the projections made by these agencies (essentially all using cohort component techniques) to simple extrapolations of the population growth rate, he concludes that the agency forecasts are far more accurate than those that use a simple geometric increase. It seems that the performance of simple techniques depends heavily on the set of projections examined. Long (1987) evaluated projections made by the Census Bureau since 1945. He presents the root mean square error9 of the ∆r values described above for projections 4, 8, 12, 16, and 20 years into the future. This statistic gives an absolute size of the error averaged over several years. For example, the root mean square error for a 4- year projection made in 1947 averages the squared errors for the projections to 1948, 1949, 1950, and 1951. In 31 of 51 comparisons between the Census Bureau's projection and a naive extrapolation of the growth rate, the naive extrapolations did better. They outperformed the Census Bureau prior to 1955 and between 1966 and 1970. Between 1955 and 1966 and after 1970, the naive projections were worse. It seems that the bureau did a poor job anticipating the baby boom and bust, partly because expert judgment lagged several years behind changes in fertility. With greater stability in fertility rates, the cohort component method does well. Mention of economic-demographic models should be made at this point. Social science researchers focus considerable attention on the determinants of fertility, mortality, and migration. Certain relationships are well documented (e.g., the inverse association between education and fertility), and one might expect that the demographic inputs to population projection models could be better specified if trends in other factors, such as education, income levels, marriage rates, push-and-pull forces, were accounted for. Nevertheless, Keyfitz (1982) deems the possibilities here to be dim, citing problems in changing relationships, imprecise predictions of other factors, uncertain timing of events, and compensating effects. He argues that the various determinants of fertility, mortality, and migration are best handled through the subjective assessments of experts. Attempts to include economic factors as inputs and outputs of a complicated economic-demographic model have been made, but there is little evidence that they outperform simpler models. Stoto and Schrier (1982) evaluated two predictions of the Bureau of Economic Analysis's OBERS economic model and found it to be biased and quite variable, having twice the variability of simple geometric extrapolations for state populations. On the other hand, Ahlburg (1990) found that forecasts of annual births made in 1979 using an economic-demographic model are more accurate than the Census Bureau's forecasts. The latter forecasts were made in 1977 using a base population from July 1, 1976, so the two forecasts are not strictly comparable. 9To calculate the root mean square error of the r values, one must square the r's, average them, and take the square root. This statistic is similar to the average of absolute values, but it gives greater weight to large errors. A simple average error is not helpful in determining the magnitude of errors because positive and negative values will cancel out.

EVALUATING THE ACCURACY OF U.S. POPULATION PROJECTION MODELS 317 TABLE 2 Values of ∆r for Census Bureau Projections of Total U.S. Population Size for Base Years 1945 to 1982 (percent) Duration (years) Base Year 1945 1950 1955 1960 1965 1970 1974 1982 5 −0.86 −0.07 −0.23* 0.11 0.09* 0.12* −0.21* −0.03* 10 −0.96 −0.18 −0.17* 0.31* 0.20* 0.21* −0.13* 15 −1.05 −0.20 −0.01* 0.44* 0.28* 0.28* 20 −1.03 −0.12* 0.12* 0.52* 0.35* 25 −0.97* −0.09* 30 −0.93* 35 −0.91* 40 −0.92* Average −0.95 −0.13 −0.07 0.35 0.23 0.20 −0.17 −0.03 NOTES: Numbers marked with an asterisk are based on our own calculations. All averages are based on our own calculations. All figures for 1955 are recalculated because those in Stoto (1983) pertain to base year 1958, even though Stoto labels them as being for 1955. ∆r is defined as where is the average population growth rate (see text). SOURCE: Stoto (1983); projections taken fro m Bureau of the Census (1953, 1955, 1962, 1966, 1971, 1977) and Spencer (1984); population estimates derived from Bureau of the Census (1990). Let us now turn to a consideration of the accuracy of Census Bureau and SSA projections of the entire U.S. population. It has become common practice in the publication of Census Bureau projection reports to present a brief evaluation of the errors made in past projections. This serves to warn users of the uncertainty surrounding future population characteristics but typically considers only short-term validations of recent projections. There is a need for a more comprehensive analysis of past errors. Stoto (1983) and Long (1987) are good examples of such comprehensive analyses. Stoto presents a table of ∆r values based on median projections10 made by the Census Bureau beginning in 1945. We have supplemented his numbers with calculations of our own and corrected others on the basis of revised estimates of the 1970 and 1975 populations. Our combined results are shown in Table 2. If each value in Table 2 is considered as a separate projection, errors range from −1.05 to 0.52 percentage points, with an average of −0.19. Thus, the median projections generally underestimated subsequent population growth during the 1945–1982 period. There is an obvious pattern in the errors, however, with projections prepared from 1960 to 1970 being too high and those made prior to 1960 and after 1970 being too low. Stoto notes that other expert projections of U.S. population size made between 1910 and 1947 are also too low. 10 Where an even number of projections is made, median refers to the average of the two middle series.

EVALUATING THE ACCURACY OF U.S. POPULATION PROJECTION MODELS 318 FIGURE 4 Root mean square errors for 4-year and 8-year projections of U.S. population by Bureau of the Census. SOURCE: Long (1987). Long (1987) presents root mean square errors for Census Bureau median projections. These numbers show the degree to which projections have varied from subsequent outcomes in the past. The average errors covering 4- and 8-year spans are shown in Figure 4. Clearly, errors have become smaller in more recent projections, especially since the peak of the baby boom in the late 1950s. Long argues that some of the improvement may be due to methodological improvements in fertility analysis but that, more likely, the lower variability of fertility since 1970 is responsible. (The issue of changes in the variability of population projections is discussed in the next section.) Considerably less work has focused on validating SSA's projections. SSA projections differ in various ways from those of the Census Bureau. Most importantly, the population covered by the SSA is larger and includes Puerto Rico, the Virgin Islands, Guam, and American Samoa as well as various citizens living overseas, whereas the universe used by the Census Bureau is defined as the population of the 50 states and members of the armed forces living overseas. Second, the SSA adjusts the base population for net census undercount, so its projected population is also adjusted for undercount. The Census Bureau, on the other hand, first inflates the base population to correct for undercount to do its projections but then deflates the result to be comparable to the population expected in the next census, given the same pattern of undercount. Finally, the

Next: CONFIDENCE INTERVALS »
Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers Get This Book
×
Buy Paperback | $100.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

This volume, second in the series, provides essential background material for policy analysts, researchers, statisticians, and others interested in the application of microsimulation techniques to develop estimates of the costs and population impacts of proposed changes in government policies ranging from welfare to retirement income to health care to taxes.

The material spans data inputs to models, design and computer implementation of models, validation of model outputs, and model documentation.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!