National Academies Press: OpenBook

Traffic Forecasting Accuracy Assessment Research (2020)

Chapter: Chapter 5 - Improving Traffic Forecasting Methods

« Previous: Chapter 4 - Reporting Accuracy Results
Page 66
Suggested Citation:"Chapter 5 - Improving Traffic Forecasting Methods." National Academies of Sciences, Engineering, and Medicine. 2020. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 66
Page 67
Suggested Citation:"Chapter 5 - Improving Traffic Forecasting Methods." National Academies of Sciences, Engineering, and Medicine. 2020. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 67
Page 68
Suggested Citation:"Chapter 5 - Improving Traffic Forecasting Methods." National Academies of Sciences, Engineering, and Medicine. 2020. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 68
Page 69
Suggested Citation:"Chapter 5 - Improving Traffic Forecasting Methods." National Academies of Sciences, Engineering, and Medicine. 2020. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 69
Page 70
Suggested Citation:"Chapter 5 - Improving Traffic Forecasting Methods." National Academies of Sciences, Engineering, and Medicine. 2020. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 70

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

I-54 The project team recommends that planners and engineers consider the results of past accuracy assessments in improving traffic forecasting methods. Building from that broad rec- ommendation, this chapter describes three specific ways in which traffic forecast assessments can be used to guide method improvements. The project team used the term traffic forecasting methods to refer both to the breadth of tools used and to the full range of activities involved in generating a forecast. Travel demand models are a common method of traffic forecasting, and much of the focus in this chapter is on improving those models. However, it is important to recognize that creating a project-level traffic forecast may involve creating land use inputs for those models, assembling inputs to those models, making assumptions about various factors (e.g., expected gas prices), and pivoting the results from base-year traffic counts. The weak points of the forecasting process may lie in one of those activities rather than in the model itself. When that is the case, the specific input or activity should be the focus for improvement. 5.1 Using Deep Dives to Guide Model Improvements Chapter 4 in this guidance document described a process for conducting a deep dive to examine the forecast accuracy associated with a specific project. One goal of a deep dive is to better understand what may have contributed to error in the forecast. For example, the deep dives conducted for this research considered how the traffic forecast would change if the estimated inputs to the model were replaced by actual inputs based on information and counts available after the project’s opening. In other words, the deep dives considered how the forecast would have changed if the model were run using: • Actual population and employment numbers obtained for the time the facility opened; • Actual (observed) travel speeds on the facility; and/or • Accurate counts of external traffic. This examination allowed the relative importance of the inputs also to be examined. If a traffic forecast improved a great deal with any of these changes, the improvement would suggest that improving the accuracy of that specific input may be valuable. Conversely, if the traffic forecast changed very little with a change, then the lack of improvement would suggest that it may not be worth expending additional resources to further refine that specific input or step. For example, if several deep dives revealed that the largest source of error in traffic forecasts was associated with inputs on employment, this observation would suggest that improving the socioeconomic models used to create the employment inputs would be more productive than trying to improve the travel models. C H A P T E R 5 Improving Traffic Forecasting Methods

Improving Traffic Forecasting Methods I-55 It is logical that such a deep dive analysis would be performed before a model development project. When models are being developed, resources are already being allocated, and the results of this kind of analysis may inform and improve the allocation of those resources. Specifically, the project team recommends that the first task in a model development project should be to conduct three to five deep dives to examine facilities that are already open but are similar to projects that will be forecast in the future using the model. These deep dives would be conducted to evaluate the strengths and weaknesses of the existing forecasting process, with the analysis informing where that process most needs improvement. This is not to suggest that such accuracy assessments would be the only mechanism for guiding new model improvements. It remains important that new models be sensitive to the types of policies and projects expected in the future, even if those differ from the types of policies and projects considered in the past. It is also appropriate that new models should incorporate improved methods, both for theoretical reasons and in ways that take advantage of new data or computing resources. However, the project team suggests that past accuracy be considered on par with these other factors. Some researchers may question the value of evaluating past forecasts at precisely the time that a new forecasting tool is being developed. Would the findings not be irrelevant if a new tool is being developed anyway? The authors of this report also believe in the importance of testing a new model in a similar capacity and recommend doing so. Nonetheless, forecasting is unique precisely because the analyst does not know the outcome a priori. Evaluating prior forecasts on the basis of known outcomes offers an opportunity to identify and learn about unanticipated changes or risks in a way that is not otherwise possible. Important insights may be gained from that opportunity. 5.2 Project-Level Testing and Validation The project team’s second recommendation focuses on testing and improving newly devel- oped travel models by testing their ability to correctly predict the changes that occur when a project opens. This kind of testing would rely on projects that have already opened, using data archived as described in Part I, Chapter 3, in this report. The Travel Model Validation and Reasonability Checking Manual (the Manual) (Cambridge Systematics, Inc. 2010) recommends temporal validation and sensitivity testing. Project-level validation is one specific form of temporal validation. The Manual defines temporal validation tests as comparisons of model forecasts or backcasts against observed travel data, and sensitivity tests as evaluations of model forecasts for years or alternatives for which observed data do not exist. In our experience, the authors of this research report have commonly observed temporal validation tests to occur at a regional level. For example, a travel model developed with a 2015 base year may be applied to 2010 conditions and compared to 2010 traffic counts by area type and facility type. The project team also has observed that sensitivity testing often happens at a project level, meaning that a specific alternative is coded, and the model is then run with and without that alternative, comparing the build and no-build cases to make sure the result is reasonable. Both approaches are valuable, but each comes with its own limitations. A challenge of tem- poral validation as described is that many factors change at once—population, employment, networks, technology, and so forth—which makes it difficult to isolate the effects of specific projects. In addition, a focus at the regional level can make it difficult to detect the effects of indi- vidual projects. In contrast, sensitivity testing of specific alternatives allows the project effects to be isolated, and in some ways is more comparable to the way in which the model will be applied

I-56 Traffic Forecasting Accuracy Assessment Research for project-level forecasting. However, sensitivity testing is limited because it does not involve a comparison to observed data. What the project team proposes in this guidebook is a project- level temporal validation (sometimes referred to as longitudinal or dynamic validation) that involves testing the model for a specific change that has already occurred and then comparing the model’s predictions to observed outcomes. Precedent exists for such an approach in the before-and-after studies required by the FTA’s New Starts program. In order to receive federal funding for new transit infrastructure projects—generally rail or other fixed guideway systems—project sponsors must conduct a before-and-after study to measure the effects of the project. Meeting the requirement involves collecting detailed transit ridership data both before and after the project opens, and part of the study involves an evaluation of past transit ridership forecasts (similarly to the evaluation of past forecasts as described in the previous section). In some cases, the before-and-after data has been used to validate new travel models. For example, following the opening of the Central Light Rail Line in Phoenix, Arizona, new travel models were run with the project’s specific before-and-after data. The results allowed comparisons to be made based on how well each of the new models accommodated the changes between the estimated (before) and the observed (after) data. Those comparisons were used to diagnose and guide the resolution of issues in the new models. This type of analysis is valuable because forecasters develop models to predict change, and it is appropriate to evaluate new models on their ability to predict change. If one is merely interested in what is happening in the base year, one could simply collect data. By contrast, this kind of analysis is predicated on having the data available against which to compare the model results, and it is that data that distinguishes it from a sensitivity test. If forecasting agencies begin to systematically collect data on opened projects for which they have retained the forecasts, the agencies will also be compiling a library of relevant projects against which they can test new models. A few details bear mentioning in terms of what is collected: • For the actual outcomes, there is value in collecting traffic counts and associated data before and after the project opens. This approach allows the project effects to be better isolated, and it is the approach used by Highway England’s Post-Opening Project Evaluations (POPE) (Highways England 2015). • Archiving appropriate details about the scope of the project also is important. A map can be valuable so that the networks can be coded appropriately. This approach would be further facilitated by using the Gold-level archiving standard for select projects. • If the model runs for a project are available, it is likely that the existing inputs (in terms of socioeconomic data and networks) can be converted for use in a new model with modest effort. One of the challenges to backcasting efforts is the effort of coding networks and com- piling inputs, and if those inputs have been properly archived and are available, the cost to do such tests is lower. • A logical approach to conducting such tests is to look for symmetry with selected deep dives, as described in the previous section of this chapter. If a model development project starts with an evaluation of the performance of the forecasts created using the old model for three to five projects, it can end with an evaluation of the performance of the new model for those same three to five projects. If the new methods better replicate the observed changes from the forecasts for those three to five “benchmark” projects, such direct comparisons make a compelling case for the value of the new model.

Improving Traffic Forecasting Methods I-57 5.3 Large-N Analysis for Method Selection The third way that traffic forecast accuracy evaluations can be used to improve traffic fore- casting methods is to use Large-N analysis to determine whether some methods produce more accurate forecasts than others. Considering alternative methods and models is useful for many situations in traffic fore- casting. Examples include using traffic count trends versus a travel model, using a four-step travel model versus an activity-based travel model, a multinomial logit versus a nested logit mode choice model, a static versus a dynamic traffic assignment model, and so forth. In many cases, good reasons exist to prefer one method or model over another. Perhaps one approach has a stronger theoretical foundation, or a particular method might provide the sensitivity needed to test a broader range of policies. For example, a time-of-day choice model might provide the ability to test the effect of peak spreading, whereas static time-of-day factors would not. It may be that one method is preferred due to ease of use or better computational effi- ciency. These considerations in selecting forecasting methods are all legitimate, but the project team proposes that the ability to produce accurate forecasts should also be a key consideration in method selection. Unfortunately, limited empirical evidence exists to guide such decisions. Traffic forecasters report plenty of professional experience with the range of such methods, and they often report a general sense that the various methods work reasonably well; but little evidence has been com- piled systematically. This research project represents a start in that direction in that it compiles forecast and actual traffic data at the project level. The project team was able to gain some insight into the accuracy of different methods (e.g., finding that travel models tend to be more accurate than traffic count trends); however, the project team regrets that this analysis remains limited. Going forward, the challenge is two-fold: • A larger pool is needed of archived projects that include detailed information about the forecasting methods used. For this research, the project team was only able to record the methods used at a high level (e.g., travel model, traffic count trend, population growth rate, or professional judgment). Even at this high level, for a substantial number of projects in the database, the methods were not recorded. • Within the existing data, the use of certain methods is not randomly assigned. Instead, the use of certain methods correlates with the agency that produced the forecast, and often correlates with the type of project. In the database produced by this research, the effects of these correlations are seen in the difference between some of the older projects (which tend to be larger infrastructure projects) and some of the newer projects (which tend to be more routine). From a research design perspective, such correlations introduce a poten- tial for confounding across variables. If a certain method is used for routine projects and performs better, is that better performance due to the method itself or due to the routine nature of the project? This research design problem can be managed in one of two ways. First, the confounding factors can be introduced as control variables in the statistical analysis. This is the approach that was used when estimating quantile regression models for this research, and it provides an advantage over the univariate distributions as shown in the violin plots in Part II of this report. Such an approach works best if (1) the data are more complete, such that there are fewer fields with missing data, and (2) the variables used in the model are less correlated with each other. Both of these factors can be improved somewhat if data are added from more diverse sources and if those data are as complete as possible.

I-58 Traffic Forecasting Accuracy Assessment Research The second, and more robust mechanism for addressing the research design problem is to explicitly design the sample to control for potential confounding effects. One straight- forward approach is to compare forecasts for the same project that have been made using multiple (differing) methods. Such a research design controls everything but the method. In some cases this situation may occur naturally. For example, in the traffic forecast reports the project team examined from the Florida DOT, the forecasters used both a traffic count trend and a travel model, then selected the forecast they deemed most reasonable. Because both the traffic count trend and the travel model were documented, the researchers had an opportunity to go back and compare the two methods for the same project. Similarly, many FTA New Starts forecasts have involved transit ridership forecasts that were developed initially using a locally developed travel model, but then were checked against the FTA’s Simplified Trips-on-Project Software (STOPS) model. This mechanism can be applied more broadly when an agency is interested in understanding the uncertainties involved in a forecast by checking the methods that were used in the forecast against an independent method. Such an approach could be greatly facilitated by the development of models, such as STOPS, that can be readily deployed in multiple regions. It is reasonable to envision a semi-standard model that can be deployed in multiple regions with modest effort. Recent TRB reports have focused on a portion of this problem, including NCHRP Report 735: Long-Distance and Rural Travel Transferable Parameters for Statewide Travel Forecasting Models (Schiffer 2012) and SHRP 2 Report S2-C10A-RW-2: Transferability of Activity-Based Model Parameters (Gliebe et al. 2014). To be practical, such models would require a certain level of standardization in their input data. Currently, an effort is underway to define “Datasets of standardized input and observed data in order to facilitate the testing of and compare the performance of vari- ous algorithms, approaches, or parameters” (Zephyr Foundation https://zephyrtransport.org/ projects/2-network-standard-and-tools/). The eventual goal would be to move toward a set of “interchangeable parts” models, against which differing methods could be readily tested for the same project-level forecast. The ability to test different methods for the same set of forecasts, combined with the ability to evaluate which method yields a better forecast, would provide a strong founda- tion for guiding the continued improvement of traffic forecasting methods in a scientifically sound manner.

Next: Chapter 6 - Implementation and Future Research »
Traffic Forecasting Accuracy Assessment Research Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Accurate traffic forecasts for highway planning and design help ensure that public dollars are spent wisely. Forecasts inform discussions about whether, when, how, and where to invest public resources to manage traffic flow, widen and remodel existing facilities, and where to locate, align, and how to size new ones.

The TRB National Cooperative Highway Research Program's NCHRP Report 934: Traffic Forecasting Accuracy Assessment Research seeks to develop a process and methods by which to analyze and improve the accuracy, reliability, and utility of project-level traffic forecasts.

The report also includes tools for engineers and planners who are involved in generating traffic forecasts, including: Quantile Regression Models, a Traffic Accuracy Assessment, a Forecast Archive Annotated Outline, a Deep Dive Annotated Outline, and Deep Dive Assessment Tables,

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!