Airport Aviation Activity Forecasting (2007)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

26 ASSESSING UNCERTAINTY IN FORECASTS An often-neglected aspect of forecasting is the uncertainty associated with the forecasts themselves. Most often, forecasts are presented only as point estimates; for example, total annual enplanements are projected to be 500,000 in 2015. To deal with the likelihood that enplanements may well not be exactly 500,000, many forecasters also present alternative estimates that are based on differing assumptions about external factors thought to affect the forecast. With market share forecasting, a different share factor may be used to relate local enplanements to the regional, state, or national forecast being used to drive the analysis, or the projected growth rate of the external forecast itself may be adjusted. A similar technique is often used with econometric model forecasts shown in airport master plans—differing future values of the explanatory variables are assumed, which in turn results in a different set of estimates of future activity using the derived equation. These alternative forecasts are often presented as high/low or optimistic/pessimistic scenarios. A common variant of this approach is to use a number of dif- ferent forecasting methods and then present the range of results to show where the baseline forecast fits into the mix. Providing a range of alternative forecasts is the most fre- quently used method to assess uncertainty in future activity levels. It is important, however, to recognize that there may be other sources of uncertainty owing to the statistical methods employed. These include: 1. Specification error—this occurs if the wrong functional form has been used, if one or more relevant independent variables have been excluded from the model, or if the structure of the relationship between dependent and independent variables changes over the time period in question. 2. Conditioning error—if the forecasted values of the inde- pendent variables are inaccurate this can lead to inaccu- rate forecasts of the dependent variable. 3. Sampling error—because the coefficient estimates of the independent variables are just that—estimates—the reliability with which they are estimated can affect the reliability of the forecasts of the dependent variable. 4. Random error—the modeling equation includes a ran- dom error term whose mean is zero; however, fore- casts from the equation implicitly assume that all future values of the error term are exactly zero, which may not be true. These sources of uncertainty are most often analyzed in the context of econometric modeling, and there are standard sta- tistical methods available to assess these issues (Kennedy 2003). However, the same sources of uncertainty arise in shares forecasting, time series models, and simulation models as well. Chatfield (2001) provides an overview of computing prediction intervals for forecasts. Another approach to addressing uncertainty in economet- ric models is through a formal risk analysis, where all the causal factors and relationships are allowed to vary simultane- ously according to estimated probabilities, resulting in a range of likely forecast values. A paper by Lewis (1995) reported that such an approach was used successfully to forecast out- comes of a proposed capacity expansion at Vancouver Inter- national Airport in the early 1990s. ASSESSING FORECAST ACCURACY Although forecast accuracy would appear to be a primary cri- terion when evaluating forecasts and forecast methods, in practice this can only be done after the fact (ex post), when the values can actually be measured and compared with their forecast estimates. For forecasts with long time horizons, this means that their accuracy cannot be fully assessed for many years. In practice, very few airport activity forecasts are ever subjected to an ex post analysis of their prediction accuracy. There are many different ways of measuring forecast accu- racy once the future forecasted values are known. Some com- monly used metrics include: • Mean absolute deviation—the mean of the absolute val- ues of the forecast “errors” (the difference between the forecasts and the actual values). • Root mean squared error—the square root of the mean of the squared forecast errors. This measure implicitly weights large errors more heavily than small ones. • Mean absolute percentage error—the mean of the ab- solute values of the percentage forecast errors. In addition, there are other methods that involve regression estimates of actual changes in the variable being forecast against the predicted changes. CHAPTER FOUR EVALUATING FORECASTS

27 Aside from being used to assess the accuracy of a given forecast, the metrics mentioned previously can also be used to compare one forecast with another. However, there are some pitfalls associated with doing so. First, one should account for any differing use of historical data; if one model was con- structed using more (or better) historical data than another, it would be unfair to compare the models directly based on the above metrics. Or, if one model estimates only annual projec- tions whereas the other estimates on a quarterly basis, it would be difficult to directly compare their accuracy. In addition to simply comparing their forecast accuracy, it may be useful in some situations to compare two models using other statistical tests. In the common situation where one is comparing two models that are “non-nested” (i.e., one is not just a statistical “special case” of the other), there are a num- ber of test statistics that can be used to help one choose among competing models (Greene 1993). ISSUES OF OPTIMISM BIAS Medium- and long-term aviation forecasts are usually required for large aviation infrastructure projects, which are inherently risky owing to long planning horizons and the significant financial investments typically required. As noted earlier, the vast majority of airport aviation forecasts in the United States are conducted in support of airport master plans. In turn, these plans require FAA approvals to qualify for AIP funding grants, which can cover up to 95% of the costs of capital projects iden- tified in the plans. Both the funding process and the interests of the parties involved may contribute to a problem of optimism bias in air- port forecasts. From a national perspective, the provisioning of a well-functioning air transportation system is a clear responsibility of FAA. The very existence of the AIP program is evidence that the federal government has a strong commit- ment to help airports improve, upgrade, and expand their infrastructure in support of the NAS. However, although the primary funding source for airport capital projects is the fed- eral government, local airport sponsors hold the most detailed knowledge of the specific projects needed, and FAA relies on these sponsors to identify and oversee the capital projects that will best support development of the NAS. In this framework, it should not be surprising that the local authorities, who are in some respects competing with each other for limited AIP funding, may have an incentive to overstate future activity demand at their facilities to better justify their proposed cap- ital projects. This sort of situation is an example of a common “principal– agent” problem that arises in many business and government scenarios, where there is an information asymmetry among interested parties that leads to difficulties in decision making. The primary problem is how to get the agent (in this case the local airport sponsor) to act in the best interests of the princi- pal (the federal government) to carry out the principal’s ulti- mate goals, when the agent has an informational advantage. The primary way in which FAA seeks to counter its infor- mational disadvantage is through the issuance of guidance documents, requirements for master planning, and other rules that local sponsors must follow when applying for AIP grants. These efforts are intended to secure consistency across proj- ects and to help identify those that are best suited for funding from limited AIP resources. Perhaps the most direct preventa- tive measure to protect against optimism bias is the require- ment that sponsors’ five- and ten-year baseline forecasts must be within 10% and 15%, respectively, of FAA’s TAF. Although no other studies of optimism bias in forecasting airport activity in the United States were located, there have been many studies of potential biases in forecasting generally [see, e.g., Sanders and Ritzman (2001)]. As further evidence, a statistical study by Flyvbjerg et al. (2005) focused on 210 rail and road transportation infrastructure projects in 14 countries completed between 1969 and 1998. This study found that 90% of rail projects overestimated passenger traffic. The results for road projects were less one-sided, but still found an average forecast error of more than 20%. Although a formal study of optimism bias in airport avia- tion activity forecasts is well beyond the scope of the present study, it may be useful for interested stakeholders to consider additional ways to provide incentives for airport sponsors to produce realistic activity forecasts. COMPETING FORECASTS AND OPTIONS FOR RESOLUTION OF DIFFERENCES As seen previously, there are a variety of methods available for forecasting airport aviation activity. Different methods can yield different results; even if the same method is used by two different forecasters, results will vary because each forecaster may use different data or a different model speci- fication (e.g., a different set of explanatory variables in an econometric model or a different external forecast in a mar- ket share model). The issue of reconciling differing airport activity forecasts is particularly relevant when airport sponsors must compare their forecasts with FAA’s TAF as part of the master planning process. As noted earlier, when there are significant differ- ences beyond the limits prescribed by FAA guidance, these differences must be resolved with the agency. The specific procedures that FAA may use to reconcile differences in these cases are beyond the scope of discussion appropriate for this study. Nevertheless, the issue of reconciling different forecasts can be addressed in more general terms. There are a number of approaches available to try to resolve the differences. One is to critically analyze each forecast to assess which uses the

better data sources, inputs, and methods that are best suited to the particular situation; perhaps one will uncover data or sta- tistical error that would cast significant doubt on the reliabil- ity of the predictions. By doing so, it may be possible to deter- mine that indeed one forecast should be preferred to another based on data reliability or methodological grounds. Rather than look only at input and methodological features, another possibility is to assess the predicted values themselves from competing forecasts. In cases where there is significant domain knowledge, it may well be more important to focus on how reasonable the predictions appear to be according to expert opinions in the field. If a forecast does not pass a “com- mon sense” test among knowledgeable experts, it may not make sense to rely on it no matter how clean the data or how sophisticated the methodology appears to be. There is a sub- stantial amount of literature on using domain knowledge to make judgmental adjustments to statistical forecasts [Sanders and Ritzman (2001) provide a good overview.] Although it is likely that the use of appropriate domain knowledge in airport 28 activity forecasting could provide some benefits, those bene- fits would have to be weighed against the potential biases that might be introduced (e.g., the optimism bias discussed earlier). A third possible approach is to critically examine compet- ing forecasts and then attempt to combine them into a com- posite forecast; in principle, combining can reduce errors that may arise from bad data or faulty assumptions. Armstrong (2001c) argues strongly for combining forecasts when it is uncertain which forecasting method is most accurate, when there are high levels of forecast uncertainty, and when it is important to avoid large errors. In the context of airport activ- ity forecasting, all three of these conditions are likely to apply, especially when making long-term forecasts. Armstrong also argues that formal procedures should be used to combine fore- casts (such as using simple average weighting schemes) rather than making judgmental assessments of appropriate weights. He presents evidence from an analysis of 30 different com- bined forecast studies and found that forecast errors were reduced by an average of 12.5% using average weights.