Traffic Forecasting Accuracy Assessment Research (2020)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

II-40 Conclusions 4.1 Research Questions At the start of this report, the project team identified several research questions, each of which contributed to the project objective of developing a process to analyze and improve the accuracy, reliability, and utility of project-level traffic forecasts. Those research questions related both to the analysis of existing data and to the process to be followed to continue such analyses in the future. The first set of questions, addressed by the Large-N analysis, provided a means of describ- ing the historical range of forecast errors that have been observed for certain project types. A second set of questions, addressed by the deep dives, shed light on why specific forecasts might be in error. The third set of research questions focused on establishing an effective process for continuing analyses in the future. This chapter revisits each of these questions, summarizing the findings of the project team. 4.2 Large-N Findings A number of observations can be made from the Large-N Analysis: 1. Traffic forecasts show a modest bias, with actual ADT typically about 6% lower than forecast ADT. In the analyses performed on the forecast accuracy database, the precise percentage was dependent on which metric was used, but the results generally fell within a similar range. The MPDFF was +0.65% at a segment level and -5.6% at a project level. The median PDFF was -5.5% at a segment level and -7.5% at a project level. The difference between the mean and median values occurred because the distribution was asymmetric—actual values were more likely to be lower than forecast, the long right- hand tail of the distribution was indicative of a small number of projects that had actual traffic much higher than forecast. (For more detail, see Part II, Chapter 2, in this report.) When the bias was considered in an econometric framework, the project team’s median quantile regression model had an intercept close to 0 (zero), but a slope of 0.94, which was significantly lower than 1 (one). 2. Traffic forecasts show a significant spread, with a MAPDFF of 25% at the segment level and 17% at a project level. Some 90% of segment-level forecasts fell within the range –45% to +66%, and 90% of project-level forecasts fell within the range of -38% to +37%. (More information is presented in Part III, Appendix G, in the section titled “Overall Distribution.”) 3. Traffic forecasts are more accurate for higher-volume roads. This relationship can be observed in the figures and data presented in Part III, Appendix G, under “Forecast Volume.” For example, for segments with 60,000 ADT or more, the MAPDFF was 12.4% compared to 24.74% overall. The result was confirmed by the project team’s quantile regression models, C H A P T E R 4

Conclusions II-41 which had a slope closer to 1 (one) for volumes greater than 30,000 ADT. This result echoes the maximum desirable deviation guidance provided in NCHRP Report 255 (Pedersen and Samdahl 1982) and NCHRP Report 765 (CDM Smith et al. 2014), where there are tighter targets for calibrating a travel model for higher-volume links. 4. Traffic forecasts are more accurate for higher functional classes, over and above the volume effect. The project team’s quantile regression results show narrower forecast windows for freeways than for arterials, and for arterials than for collectors and local roads. The actual volumes on lower-class roads are more likely to be lower than the forecast volumes. The difference may relate to challenges related to limitations of zone size and network detail on smaller projects, as well as less opportunity for inaccuracies to average themselves out as they do on larger facilities. 5. The unemployment rate in the opening year is an important determinant of forecast accuracy. According to the models detailed in Part III, Appendix G, under “Inclusive Model for Inference,” for each point of increase in the unemployment rate in the opening year (e.g., an increase from 5% to 6% unemployment), the median estimate of ADT decreases by 3%. For example, consider two roads, each with the same forecast. One road is scheduled to open in 2005 with an unemployment rate of 4.5%, and the other road is scheduled to open in 2010 with an unemployment rate of 9.5%. The actual opening-year ADT would be expected to be 15% lower for the project that opens in 2010 ((9.5 − 4.5) p 0.03). Because the unemployment rate has such a large impact on actual ADT, this variable is very important to forecast accuracy. 6. Forecasts appear to implicitly assume that the economic conditions present in the year the forecast is made will perpetuate. This bias also was observed in the models detailed in Part III, Appendix G. In particular, this finding was based on the coefficient of the unemployment rate in the year produced, which was positive. The positive coefficient means that a high unemployment rate in the year the forecast is produced is more likely to result in an actual ADT that is higher than the forecast, whereas a low unemployment rate in the year the forecast is produced will likely have the opposite effect. 7. Traffic forecasts become less accurate as the forecast horizon increases, but the result is asymmetric, with the actual ADT more likely to be higher than the forecast ADT as the forecast horizon increases. The forecast horizon is the length of time into the future for which forecasts are prepared. For this research, the forecast horizon was measured as the number of years between when the forecast was made, and the project opening. The quantile regression results (available in Part III, Appendix G, in the sections titled “Inclusive Model for Inference” and “Forecasting Model”) show that the median, 80th percentile, and 95th percentile estimates increase with an increase in the DiffYear variable, but that the 5th and 20th percentile estimates either stay flat or increase by a smaller amount. 8. Regional travel models produce more accurate forecasts than traffic count trends. The MAPDFF for regional travel models is 16.9% compared to 22.2% for traffic count trends (see Part III, Appendix G, under “Forecast Method”). In addition, the quantile regression models show that using a travel model narrows the uncertainty window. 9. Some agencies produce more accurate forecasts than others. The agencies with the most accurate forecasts (involving more than a handful of projects) had a MAPDFF of 13.7%, compared to 32% for agencies with the least accurate forecasts. A portion of these differences showed up as significant in the quantile regression models. (For more information, see Part III, Appendix G, under “Inclusive Model for Inference.”). 10. Traffic forecasts have improved over time. This observation is based on the project team’s assessments of forecasts in both the years the forecasts were produced and the projects’ opening years. Forecasts for projects that opened during the 1990s were especially poor, exhibiting mean volumes 15% higher than forecast and with a MAPDFF of 28.1%. The quantile regression models for forecasting (discussed in Part III, Appendix G, under

II-42 Traffic Forecasting Accuracy Assessment Research “Forecasting Model”) show that, whereas older forecasts do not show a significant bias relative to newer forecasts, they do have a broader uncertainty window. 11. The project team found that 95% of forecasts reviewed were “accurate to within half of a lane.” For 1% of cases, the actual traffic volume was higher than the forecast volume, meaning that additional lanes would be needed to maintain the forecast LOS (for details, see “Data Exploration” in Part III, Appendix G). Conversely, for 4% of cases, actual traffic volume was lower than the forecast volume, and the same LOS could be maintained with fewer lanes. Revisiting the original research questions, the project team offers the following conclusions: • How accurate are traffic forecasts? The project team examined this question using a large sample of projects (the forecast accuracy database). In relation to the sample, the question can be recast as “What is the distribution of forecast errors across the sample as a whole?” and the forecast errors can best be summarized by the distribution shown in Figure II-1. • What are the sources of forecast error? To respond to this question, the project team addressed several more targeted questions: – Can statistically significant bias be detected in the forecasts? – If such bias is detected, is that bias a function of specific factors (e.g., the type of project, the time between the forecast and the opening year), or is it a function of the methods used? – After adjusting for any bias, how accurate are the forecasts? – Is the accuracy a function of specific factors (e.g., the type of project, the time between the forecast and the opening year, or the methods used)? The answer to the question of whether bias can be detected in the forecasts is “Yes.” Actual ADT is about 6% lower than forecast ADT, and this difference is statistically significant. Several factors were found to affect this bias, including economic conditions, forecast horizon, and facility type. The sign of the coefficients in the 50th percentile estimates of the quantile regression models can be used as a measure of this bias. Using the inclusive model, the project team found that economic conditions affect the bias. For each percent increase in the unemployment rate in the opening year (e.g., an increase from 5% to 6%), the median expected ADT is 3% lower than the forecast ADT. Similarly, for each percent increase in the unemployment rate in the forecast year, the median expected ADT is 1% higher. Facility type also affects the bias. Relative to freeways, the median expected ADT on arterials is 8% lower than forecast, and the median expected ADT on collectors and locals is 14% lower than forecast, all else being equal. For each additional year added to the forecast horizon, the median expected ADT increases by 1%. The MAPDFF of traffic forecasts is 17% at a project level, with 90% of project-level forecasts falling within the range of -39% to +37%. The level of accuracy is affected by several factors, including the functional class, the forecast horizon, economic conditions, and the forecasting method, agency, and year. One of the more important factors identified is that forecasts tend to be more accurate for higher-volume roads. It is important to note a limitation of this study: the data used in NCHRP 08-110 were not a random or representative sample of all traffic forecasts. The data were assembled based on availability and were obtained as shared by different agencies and from past researchers who have examined the topic. Therefore, the sample may contain some selection bias. For example, Agency A may have compiled data on the largest projects to have opened since 1990, whereas Agency B may have compiled data on all forecasts to be prepared since 2000. Agency B’s sample will naturally contain more routine projects, and more projects with a shorter planning horizon; moreover, these variables will be correlated with the agency itself and the methods that agency uses. The fields recorded by Agency A and Agency B also may differ, which means that varying subsets of the data fields in the forecast accuracy database will contain missing data.

Conclusions II-43 The issue of how to treat missing data becomes important when using the quantile regression models. The project team took differing approaches to the model for inference and the model for forecasting. In the inference model, the project team estimated a separate coefficient on any attribute that was missing, such that the project team did not bias the relative estimates of the other non-missing values. In the forecasting model, however, the project team took the view that the person developing a forecast will always know attributes such as the facility type, the forecasting method, and whether or not the facility is a new road. It is not logical to put forward a forecasting model that includes unknown facility type as an option; however, the project team wanted any uncertainty associated with the facility type and the forecasting method to be reflected in the base model. Consequently, when applying the forecasting model, an analyst would use a subset of the available variables. Data limitations also arose when interpreting the model estimation results. These limitations became most relevant in relation to the project team’s examination of data across agencies and over time. The project team did conclude that some agencies have more accurate forecasts than others and that traffic forecasts have improved over time. The data provided by different agencies came from different time periods, with different mixes of projects. Having examined the data, it appeared that data for routine projects such as repaving and minor improvements were more likely to be recorded in more recent years, as records of those routine or minor projects were less likely to have been maintained over a span of decades. Although one might think that forecasts get better over time because the forecasters now have access to better data, more computational power, and better models, it may also be that the forecasting task has become easier over time. Infrastructure budgets are constrained, and states today build fewer big projects. Between the 1970s and the 1990s, growing auto ownership and growing numbers of women in the workforce logically led to more VMT per capita and measured volumes that were higher than those forecast. By the 2000s, however, both these demographic trends had largely played out. 4.3 Deep Dive Findings Supplementing the information provided in this chapter, the detailed deep dive reports for each of the projects are given in Part III, Appendix H. The key findings of the deep dives can be summarized as follows: • On the Eastown Road Extension, actual volumes were 20% lower than forecast volumes for the existing portion of the road and 43% lower than forecast volumes for the extension. Correcting for errors in input values (employment, population/households, car ownership, fuel price, and travel time) improved these forecast values to 25% and 3%, respectively. The travel speeds appear to be of particular importance in this case, with the actual speeds on Eastown Road lower than the forecast speed. • On the Indian Street Bridge, actual volumes on the new bridge were 60% lower than forecast volumes even though the base-year validation was reasonable. Correcting errors in the inputs (employment, population, and fuel price) only improved the forecasts slightly. It is not clear why the discrepancy occurred. • For the Central Artery Tunnel, actual traffic on modified links was 4% lower than forecast traffic, and actual traffic on new links was 16% lower than forecast traffic. This represents a strong forecast for a massive project with a long time horizon. Correcting input errors (for employment, population, and fuel price) improved the forecast error to +3% for existing links and -10% for new links. • On the Cynthiana Bypass, actual traffic was about 30% lower than forecast traffic for three of four bypass segments, and 4% lower than forecast traffic for the fourth bypass segment.

II-44 Traffic Forecasting Accuracy Assessment Research The major sources of error on this project were the external traffic forecasts, wherein the actual traffic at external stations was 43% lower than the forecast traffic. Correcting this issue reduced the absolute error to less than 4% for three of four segments, although with this cor- rection actual traffic on the fourth segment remained higher than the adjusted forecast. • On the South Bay Expressway, the long-term forecasts appeared to be reasonably accurate, but a straight-line interpolation to the short-term created large deviations. There appeared to be three major contributors to this outcome: (1) the project opened as a privately financed toll road in November 2007, just before the recession caused a decrease in demand; (2) an important travel market for the road is border crossings from Mexico, particularly for truck traffic, and border crossings decreased from their long-term trend about the time the toll road opened; and (3) the operator responded to the decrease in actual traffic by increasing tolls, further reducing demand. The operator was unable to survive these challenges and went bankrupt in 2010. SANDAG bought the road and reduced the tolls, while border crossings and economic conditions recovered. • For the project section examined in the deep dive on US-41 in Brown County, the original traffic forecasts slightly overestimated traffic by 3% to 10% for three study sites, but the forecasts were generally close. The traffic forecasting accuracy improved after correcting the exogenous population forecast; however, the fuel price adjustment increased the forecast error. This increase could be accounted for in that the change in fuel price had little effect on the traffic volumes in the study area where public transportation is not a reasonable alternative mode. Similar to the findings from the Large-N analysis, the traffic for the five projects chosen for complete deep dive analysis was more likely to be overpredicted than underpredicted. The deep dives expanded the project team’s knowledge regarding this overprediction by iden- tifying the contributing sources to the inaccuracy. The key takeaways from the deep dive analyses are: • The reasons for forecast inaccuracy are diverse. While the above points list some of the factors that contribute to forecast inaccuracy, it is clear from our limited sample that the reasons for inaccuracies are diverse—external forecasts, travel speeds, population and employment forecasts, and short-term variations from a long-term trend have all be identified as contributing factors in one or more of the deep dives. • The forecasts for all six projects considered show “optimism bias.” For each project, the observed traffic is less than forecast, and, for all except US-41, correcting for the factors listed reduces the difference between the forecast and observed traffic. • Employment, population, and fuel price forecasts frequently contribute to forecast inaccuracy. Adjustments to the forecasts using elasticities and model re-runs confirmed that significant errors in opening year forecasts, employment, fuel price, and travel speed had a major role in the overestimation of traffic volumes. In addition, the project team observe that macroeconomic conditions in the opening year influence forecast accuracy, particularly for projects which opened during or after an economic downturn. • External traffic and travel speed assumptions also affect traffic forecasts. For the Bypass extension project in Cynthiana, the estimated growth rate for external trips was the largest source of forecast error. Travel speed was an important factor for the Eastown Road Extension because inaccurate speeds led to too much diversion from competing roads. Using these observations, the project team was able to answer the original research questions as follows: • What aspects of the forecasts (e.g., population forecasts, project scope) can be clearly identified as being accurate or inaccurate? The project team found that population, employment, and fuel price are common factors contributing to forecast inaccuracy, and that

Conclusions II-45 external traffic and travel speed can be important in some cases. However, the project team also found that, even within the small sample of deep dives examined, the reasons for forecast inaccuracy were diverse. • If the forecast had gotten those inaccurate aspects right, how much would it have changed the traffic forecast? For each deep dive, the project team calculated how much the forecast would have improved if corrections were made for identified and quantified input errors. In four of the five cases examined, correcting the inputs corrected for most of the forecast error. For the Indian Street Bridge project, correcting the inputs only improved the forecasts slightly. The summary of the deep dive findings is presented in Table II-16. 4.4 Process Findings Conducting these analyses also provided the opportunity to evaluate the process of learning from traffic forecast accuracy assessments. Similar to previous researchers, the project team found that the biggest challenge in completing the research was to acquire the data itself. Several observations can be made about the effectiveness of the process used to conduct this research: 1. Although it remains rare, several transportation agencies have started archiving their forecasts in recent years, and practitioners are beginning to see the benefits of that fore- sight. The data used in the Large-N analysis were provided by state DOTs and researchers who had studied traffic forecasts previously. Some of these previous efforts had involved creating datasets of all project-level traffic forecasts in recent years. Many of the projects in these datasets had not yet opened, but the data were retained for future analysis. Thus, whereas the analysis in NCHRP 08-110 was based on about 1,300 projects that had opened, data on thousands more projects in the forecast accuracy database (now available as the Project Original PDFF Remaining PDFF After Adjusting for Errors in: Remaining PDFF After all Adjustments Eastown Road Extension, Lima, Ohio -43% Employment -39% -28% Population/Household -38% Car Ownership -37% Fuel Price/Efficiency -34% Travel Time/Speed -28% Indian Street Bridge, Palm City, Florida -60% Employment -59% -56%Population -61% Fuel Price -56% Central Artery Tunnel, Boston, Massachusetts -16% Employment -10% -10%Population -14% Fuel Price -10% Cynthiana Bypass, -27% Employment -25% -8% Cynthiana, Kentucky Population -25% External Trips Only 7% US-41 (later renamed as I-41), Brown County, Wisconsin -5% Population -4% -6% Fuel Price -6% South Bay Expressway, San Diego, California Revenue less than projected, leading to bankruptcy of P3 Socioeconomic Growth The available documentation did not allow the effect of these factors on traffic volume to be quantified Border Crossing Toll Rates Table II-16. Known sources of forecast inaccuracy for deep dives.

II-46 Traffic Forecasting Accuracy Assessment Research Forecast Cards and Forecast Cards Data repositories) are waiting to be evaluated after the projects open. 2. Inconsistencies across multiple sources of data limited the project team’s ability to draw certain conclusions. The data from different sources was generally provided in a tabular format, but the fields that were included differed depending on the source. For example, the data provided for some projects included peak-hour forecasts, whereas the data for other projects included only ADT. Similarly, some projects included details on the forecast method, whereas others did not. Reports also used differing categories for the type of improvement. The project team was able to normalize much of the data to a common format but was left with missing values in a number of important fields. Because the missing values are not random, it is difficult to reliably draw conclusions related to specific fields. 3. The available data are not a random sample of projects. The analyses conducted for this project were based on available data, which are not a random sample of all transportation projects. The data were provided by different sources and included many different types of projects. For many projects, the forecasts were made in 2005 or later and the projects were fairly small and routine. Other projects involved forecasts that were made 20 or more years ago, and these tended to be for larger projects. Although the data show that more recent forecasts tend to be more accurate, it is difficult to determine whether the increased accuracy is because the methods have improved or because the projects are more routine and thus less challenging to forecast accurately. 4. Project documentation is often insufficient to evaluate the sources of forecast error. In the deep dives, the project team found that the forecasting accuracy improved after accounting for several exogenous variables like employment rate and population. For other potentially important variables, however, the effect of changes could not be ascertained for some of the projects. Improved documentation of the forecast methods would make such assessments more informative, particularly with respect to the definitions of the variables used in the model. 5. Forecast evaluation is most effective when archived model runs are available. The most successful deep dives were those for which the project team had archived model runs and associated inputs available. This provided deeper understanding of the parameters and methods used for forecasting traffic and allowed the researchers to test the effects of changes. While discussing this approach with colleagues early in the research, several colleagues expressed skepticism that it would be practical to learn from archived model runs when the software and technology has changed over the years since the forecasts were initially made. The project team was able to successfully rerun and learn from all three archived model runs that were provided. These included models as old as 15 or more years, and included models initially developed using software several versions prior to the current versions. 6. The best way to compare the accuracy of forecasting methods is by comparing compet- ing forecasts for the same project. The data do suggest that forecasts made with a travel demand model tend to be more accurate than those made by extrapolating traffic count trends, but the project team was unable to draw clear conclusions about the accuracy of different types of models. In part, the uncertainty reflects the fact that details about the models and their features often were not recorded with the original project data. When such details were available, the models varied by agency, but so did other factors, such as the characteristics of the state or metro areas involved in the forecast and the type of project. Given the amount of variation in the available records, it was difficult to distinguish the effect of each factor individually. The best way to compare types of models would be to produce competing forecasts for the same set of projects and compare their accuracy. This approach would be equivalent to conducting a controlled experiment accounting for all relevant factors.

Conclusions II-47 In relation to these six observations, the project team also considered the following process questions: • What information should be archived from a forecast? • What data should be collected about actual project outcomes? • Which measures should be reported in future Large-N studies? • Can an example structure be defined for future deep dives? The project team’s findings and recommendations in relation to these process questions are presented in Part I of this report, which presents guidance for archiving forecast data at various levels, the highest of which includes archiving information about the models used. It is hoped that as agencies begin to archive and share more of their forecast data—particularly forecasts archived at the Gold level—controlled experiments to compare the accuracy of forecasting methods will be more feasible in the near future.