Traffic Forecasting Accuracy Assessment Research (2020)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

III-G-1 A P P E N D I X G Large-N Analysis Contents III-G-2 1 Introduction III-G-3 2 Available Data and Key Challenges III-G-3 2.1 Data III-G-5 2.2 Database Structure III-G-7 2.3 Decision Variables III-G-9 3 Method III-G-9 3.1 Methods Used in Existing Literature III-G-10 3.2 Evaluation Year III-G-11 3.3 Definition of Errors III-G-12 3.4 Distribution of Errors III-G-13 3.5 Bias Detection III-G-14 3.6 Level of Analysis: By Segment or by Project III-G-15 3.7 Data Cleaning and Filtering III-G-16 3.8 Outliers III-G-16 3.9 Calculating the Number of Lanes Required III-G-18 4 Data Exploration III-G-20 4.1 Overall Distribution III-G-22 4.2 Forecast Volume III-G-24 4.3 Functional Class III-G-25 4.4 Area Type III-G-26 4.5 Type of Project III-G-27 4.6 Tolls III-G-28 4.7 Year Forecast Produced III-G-30 4.8 Opening Year III-G-32 4.9 Forecast Horizon III-G-33 4.10 Unemployment Rate in Opening Year III-G-34 4.11 Change in Unemployment Rate III-G-35 4.12 Forecast Method III-G-36 4.13 Type of Forecaster III-G-37 4.14 Effect on Number of Lanes III-G-38 5 Econometric Analysis III-G-39 5.1 Base Model III-G-41 5.2 Inclusive Model for Inference III-G-44 5.3 Forecasting Model III-G-48 References

III-G-2 Traffic Forecasting Accuracy Assessment Research 1 Introduction To conduct the Large-N analysis, the NCHRP 08-110 project team collected data from numerous sources and created a database (hereafter called the “forecast accuracy database”). The forecast accuracy database was initially developed using Microsoft Access. Later, to facilitate systematic archiving and promote sharing of data by agencies going forward, the project team uploaded the datasets from the forecast accuracy database into the Forecast Cards and Forecast Cards Data repository. The project team also used this database to measure the accuracy of traffic forecasts and to develop quantile regression methods for estimating the uncertainty surrounding a forecast. A Microsoft Excel spreadsheet was developed (and is included as a downloadable file with this report) that demonstrates the application of these quantile regression methods. This appendix focuses on the creation and use of the forecast accuracy database and the methodologies that were used when running the Large-N analysis for this project. (For more information about the Forecast Cards and Forecast Cards Data repositories and the electronic tools that accompany this report, see Part I, Chapter 3, and Part III, Appendix A and Appendix D). The assessment of traffic forecasting accuracy in NCHRP 08-110 builds upon past efforts. There have been several studies done on assessing the accuracy of traffic forecasts, although most of them have been focused on toll roads. The inspiration seems to be from the fact that toll road forecasts have a bearing on investor expectations and that is why their accuracy is more important. As an evidence to this, The Australia Government (2012) cited ‘‘inaccurate and over-optimistic’’ traffic forecasts as a threat to investor confidence. Three lawsuits now underway challenge the forecasts for toll road traffic that subsequently came in significantly under projections (Bain 2013). Li and Hensher (2010) evaluated the accuracy of toll road traffic forecasts in the Australian toll roads and found a general overprediction of traffic. Actual traffic counts for the roads were about 45% lower than the predicted values on average in the first year of operation. The accuracy didn’t get better over time, as the percentage error reduces by only 2.44% each year after opening. Li and Hensher attributed this error in forecasting to several factors: less toll road capacity (when opened, as compared with the forecast); elapsed time of operation (roads opened longer had higher traffic levels); time of construction (longer construction time delayed traffic growth and increased the error); toll road length (shorter roads attracted less traffic); cash payment (modern no-cash payment increased traffic); and fixed/distance-based tolling (fixed tolls reduced traffic). Bain (2011), on the other hand, took the ratio of actual to forecast traffic for 100 toll road projects and found an underprediction of 23% on average. The factors he identified were mostly the toll culture (e.g., existence of toll roads previously, toll acceptance) and errors in data collection as well as unforeseen micro-economic growth in the locality. This observation is supported by research into the accuracy of toll road forecasts in the Spanish Toll Road Network (Gomez, Vassallo, and Herraiz 2016). According to the Spanish research, the economic factors (employment and GDP per capita) are consistent variables for travel demand estimation. Odeck and Welde (2017) looked into 68 Norwegian toll roads and found that while toll-road traffic is underestimated, the forecasts are close to accurate as the mean percentage error is a mere 4%. Flyvbjerg et al. (2006) conducted a before-and-after study of 183 real projects around the world that compared the actual and forecast traffic in the opening year. Flyvbjerg et al. (2006) found about half of the projects had a forecast error of ±20% and a quarter were off by about 40% in either direction. He attributed the errors to the uncertainties in trip generation and land-use pattern. Similar results were reported by Kriger, Shiu, and Naylor (2006), who reviewed the forecasts for 15 U.S. toll roads and found that, on average, the actual traffic was 35% below the predicted traffic. From 2002–2005, Standard & Poor's publicly released annual reports on the accuracy of toll road, bridge, and tunnel projects worldwide. The 2005 report (Bain and Polakovic 2005)—the most recent report available

Appendix G: Large-N Analysis III-G-3 publicly—analyzed 104 projects. The Standard & Poor’s report found that the demand forecasts for those projects were optimistically biased, and that this bias persisted into the first 5 years of operation. They also found that variability of truck forecasts was much higher than lighter vehicles. The authors noted that their sample “undoubtedly reflects an over-representation of toll facilities with higher credit quality” and that actual demand accuracy for these types of projects is probably lower than documented in their report. Despite the works assessing forecast errors for toll roads, comparable research has not taken place to assess the same errors for non-tolled roads. There have been a few recent studies examining the accuracy of non-tolled roadway forecasts. Buck and Sillence (2014) demonstrated the value of using travel demand models to improve traffic forecast accuracy in Wisconsin and provided a framework for future accuracy studies. Parthasarathi and Levinson (2010) examined the accuracy of traffic forecasts for one city in Minnesota. Giaimo and Byram (2013) examined the accuracy of over 2,000 traffic forecasts produced in Ohio between 2000–2012. Giaimo and Byram found the traffic forecasts slightly high, but within the standard error of the traffic count data. They did not find any systematic problems with erroneous forecasts. The presentation also described an automated forecasting tool for “low risk” projects that relies on trendlines of historical traffic counts and adjustments following procedures outlined in NCHRP Report 255 (Pedersen and Samdahl 1982) and updated in NCHRP Report 765 (CDM Smith et al. 2014). In the study of 39 road projects in Virginia, Miller et al. (2016) reported that the median absolute percent error of all studies was about 40%. This portion of NCHRP 08-110 aims to conduct a similar analysis using data on forecast and actual traffic for a combined data set of about 1,300 projects drawn from six states and four European countries. 2 Available Data and Key Challenges The Large-N analysis conducted for NCHRP 08-110 used a database that was compiled as part of this project. The database contained traffic forecast and actual traffic information for road projects in several states and some foreign countries. These records were compiled from existing databases maintained by the DOTs, from ESAL reports and project reports or traffic/environmental impact statements, as well as from databases from similar research efforts. For each project, the forecast accuracy database organizes length); the forecast (year forecast produced, forecast year, method, etc.); and the actual traffic count information. The primary metric used in comparing forecast accuracy was the average daily traffic (ADT). 2.1 Data Data were included from six states: Florida, Massachusetts (one project), Michigan, Minnesota, Ohio and Wisconsin. Data also are included from four European countries: Denmark, Norway, Sweden, and the United Kingdom. The project team also acquired data from Virginia and Kentucky, but the formats of the data available from these sources were significantly different, and they would have required additional effort to enter into the database. For example, in Kentucky the data included traffic forecast reports but provided limited context about the projects and when/if they opened, making it challenging to match the forecasts to opening year counts without adequate local knowledge. The project team therefore elected to table including data from Kentucky and Virginia, leaving their incorporation to be addressed as a future exercise. Since the forecast accuracy database was being compiled using datasets obtained from a variety of sources, the project team still needed to account for smaller inconsistencies across the datasets and missing information about the project itself (unique project ID, improvement type, facility type, location,

III-G-4 Traffic Forecasting Accuracy Assessment Research information. For example, data from Florida D4 and D5 were provided in different formats—D4 data were provided in Excel format while D5 data were extracted from scanned PDF reports. Actual count information for Florida D5 were obtained by matching the count station ID available in the report with the Florida Historical Traffic Count Database. The dataset provided by the Michigan DOT came in the form of both PDF reports and an Excel table. The Minnesota dataset was gathered from previous studies in the form of an Excel table. The raw data is available at Minnesota Historical Society Archives and is in the form of scanned PDF reports produced by the Minnesota DOT. Count maps were used to get the actual ADT information, which was also provided in scanned PDF format. When clear information was not available, assumptions had to be made by the project team. These assumptions (documented in Part III, Appendix D) were taken based on the data provided by the agencies. For example: in the Minnesota data, not much information was available in the reports for the forecast method. Since these were old forecasts, it was assumed that the forecasts were made using traffic count trends. In the case of several DOTs, while actual counts were given on the same roadway, there was no mention of when the project was completed. More commonly, missing details included key information like project type/type of improvement, roadway facility functional class, forecast method, and so forth. Because this project sought to compare the forecast traffic to the actual traffic after the project has been opened, it was imperative that both these datapoints be collected in the same year. In most cases, however, the databases maintained by the state DOTs did not clarify if the actual traffic counts were taken after project completion or if the project was completed in the year it was forecast to open. At time of publication of NCHRP Research Report 934, the forecast accuracy database includes project and forecast information from Ohio, Wisconsin, Florida (D4 and D5), Minnesota, and Michigan, and from Denmark, Sweden, Norway and the UK. Data for the European Projects come from the Nicolaisen (2012) database. The Wisconsin and Minnesota datasets came from two published research studies on assessing forecast accuracy: Buck and Sillence (2014) for Wisconsin and Parthasarathi and Levinson (2010) for Minnesota. The Florida D4 data also were obtained from a published study—the Traffic Forecasting Sensitivity Analysis (2015)—which compares the actual count in the forecast year with the forecast traffic. For purposes of the Large-N analysis conducted in NCHRP 08-110, we assumed that the actual traffic counts listed in these datasets were taken after the project had been completed. As for the Ohio dataset, for a few projects/segments the actual year of project completion was given. For others, there was no indication whether or not the counts were taken after the project had opened. Similarly, Florida D5 datasets were compiled from ESAL reports. Here again we do not have any indication of the actual opening year of the projects. Table III-G-1 presents a short summary of the information available in the database, with the state names replaced by an agency code to protect anonymity. In total, the forecast accuracy database contained reports for 2,611 unique projects, with 16,697 segments associated with those projects. A segment is a different portion of roadway for which a forecast is provided. For example, forecasts for an interchange improvement project may contain segment-level estimates for both directions of the freeway, for both directions of the crossing arterial, and for each of the ramps. Some of these projects have not yet opened, some of the segments do not have actual count data associated with them, and others do not pass our quality-control checks for inclusion in the statistical analysis (the filtering process is described later in this

Appendix G: Large-N Analysis III-G-5 appendix). While all records have been retained for future use, the Large-N analysis conducted for this project was based on a filtered subset of 1,291 projects and 3,911 segments. Table III-G-1. Summary of available data. Agency Code All Projects Opened Projects Number of Segments Number of Unique Projects Number of Segments Number of Unique Projects Agency A 1123 385 425 381 Agency B 12 1 12 1 Agency C 38 7 5 3 Agency D 2176 103 1292 99 Agency E 12413 1863 1242 562 Agency F 463 132 463 132 Agency G 472 120 472 113 Total Segments 16697 2611 3911 1291 In total, the forecast accuracy database contained reports for 2,611 unique projects, with 16,697 segments associated with those projects. A segment is a different portion of roadway for which a forecast is provided. For example, forecasts for an interchange improvement project may contain segment-level estimates for both directions of the freeway, for both directions of the crossing arterial, and for each of the ramps. Some of these projects have not yet opened, some of the segments do not have actual count data associated with them, and others do not pass our quality-control checks for inclusion in the statistical analysis (the filtering process is described later in this appendix). While all records have been retained for future use, the Large-N analysis conducted for this project was based on a filtered subset of 1,291 projects and 3,911 segments. A range of projects have been included in the database. The opening year varies from 1970 to 2017, with about 90% of the projects opening in year 2003 or later. While the exact nature and scale of the project isn’t always known, inspection reveals that the older projects are more likely to be major infrastructure projects, and the newer projects are more likely to be routine work completed for the DOT (e.g., resurfacing works on existing roadway). For example, almost half of the projects are design forecasts for repaving. Such differences are driven largely by data availability. Some state agencies have begun tracking all forecasts as a matter of course, and the records to do so rarely go back more than 10–15 years. The older projects are derived from someone going back to study and enter paper reports or scans of paper 2.2 Database Structure The forecast accuracy database provided a starting point for the Large-N analysis. The data initially were collected and organized in the form of a Microsoft Access database, whose structure and use was documented within an interim user's guide for NCHRP Project 08-110. (These initial versions have been fully replaced by the Forecast Cards and Forecast Cards Data repository, so the Access database and associated user's guide are not included as products of NCHRP Project 08-110. They are discussed here to document how the authors’ thinking evolved during the course of the research and provide context for reports, with the availability of documentation and the interest in spending the effort to examine bigger projects. Thus, it is not a random sample of projects, and there are notable differences not only in the methods used across agencies, but also in the mix of projects included in the database. This is an important limitation that readers should bear in mind as they understand and interpret our results.

III-G-6 Traffic Forecasting Accuracy Assessment Research how the final design was selected.) The primary fields in this initial forecast database were classified into three types: 1. Project Information, 2. Forecast Information, and 3. Actual Traffic Count Information. The Project Information table had all the information specific to the project characteristics. This included a Project/Report ID unique to each project, a project description, the year when the project/report was completed, type of project, city or location where the project took place, state, construction cost, etc. The Forecast Information table included the data related to the traffic forecast: the forecast itself, along with who made the forecast, the year the forecast was made, and the year for which the forecast was made. It also included the type of forecast year (i.e., opening year, mid-design year, or design year), the method used to create the forecast, whether any post-processing had been done, and similar information. Information regarding the actual traffic count included the actual traffic volume in a particular segment, year of observation, and project opening year. Table III-G-2 shows the key fields in the database. Table III-G-2. Key fields in forecast accuracy database. Name Description Brief Description Brief written description of the project Project Year Year of the project or Construction Year or the Year the Forecast Report was produced Length Project Length in miles Functional Class Type of facility (Interstate, Ramp, Major/Minor Arterial etc.) Improvement Type Type of project (Resurfacing, Adding lanes, New construction, etc.) Area Type Functional Class Area type where the facility lies (Rural, Urban, etc.) Construction Cost Project construction cost State State code Internal Project ID Project ID or Report ID or Request ID County County in which the facility lies Toll Type What kind of tolls are applied on the facility (No tolls, Static, Dynamic, etc.) Year of Observation Year the actual traffic count was collected Count Actual Traffic Count Count Units Units used to collect count information Station Identifier Count station ID or other identifiers for count station Traffic Forecast Forecast Traffic volume Forecast Units Units used to forecast traffic (AADT,AAWT) Forecast Year Year of forecast Forecast Year Type Period of forecast like opening, mid-design or design period Year Forecast Produced The year the forecast was produced/generated Forecasting Agency Organization which was responsible for this forecast Forecast Methodology Method used to forecast traffic (Traffic Count Trend, Regional Travel Demand Model, Project-Specific Model, etc.) Post Processing Methodology Any post processing or alternative methods used Post Processing Explanation Explanation, as warranted, in case post processing method is used Segment Description Description of the segment for which this forecast was done AADT = annual average daily traffic; AAWT = annual average weekly traffic.

Appendix G: Large-N Analysis III-G-7 2.3 Decision Variables Based on the nature of the forecast accuracy database, we could select some variables that might dictate future adjustments in the forecasts. These variables are: the type of project, the method used, roadway type, area type, and the forecast horizon (the difference between the year the forecast was produced and the year of opening). Project types were coded into the database as Improvement Type. Along with unknown improvement types, the improvement types were categorized into 12 types, which were further classified as projects on existing roadways, new construction projects or unknown project type (see Table III-G-3). Table III-G-3. Description of project types in the forecast accuracy database. ID in Database Improvement Type Unified Improvement Type 1 Resurfacing/replacement/no minor improvements Project on Existing Roadway 2 In existing facility, add intersection capacity 3 In existing facility, add mainline/mid-block capacity in general purpose lane(s) 4 In existing facility, add new dedicated lane(s) 5 In existing facility, add new managed lane(s) 6 In existing facility, add new reversible lane(s) 7 New general-purpose lane(s) facility New Construction Project 8 New dedicated lane(s) facility 9 New managed lane(s) facility 10 New reversible lane(s) facility 11 Other new facility 12 Unknown improvement Unknown Project Type Data recorded under the Functional Class column in the database was coded according to the FHWA-specified functional classification. For a few datasets, the functional classes of the roadway had been provided in an older format, and these were converted into the new format (see Table III-G-4). Table III-G-4. Description of functional class in the forecast accuracy database. ID in Database Functional Class 1 Interstate or Limited-Access Facility 2 Ramp 3 Principal Arterial 4 Minor Arterial 5 Major Collector 6 Minor Collector 7 Local 8 Unknown Functional Class Data on the area type where the facility lies was mainly coded in four categories: Rural, Mostly Rural, Urban, and Unknown area types (see Table III-G-5). The definitions of these categories were consistent with the U.S. Census Bureau’s definitions of Urban and Rural areas. The Bureau defines an

III-G-8 Traffic Forecasting Accuracy Assessment Research urban area as a territory that has at least 2,500 people. The percentage of people living in rural areas in a county determines whether the county is rural (100%), mostly rural (50-99%) or urban (<50%). Table III-G-5. Description of area type in the forecast accuracy database. ID in Database Area Type 1 Rural 2 Mostly Rural 3 Urban 4 Unknown Area Type The Forecast Methodology (forecast method) was identified from the project reports or the datasets given by the state DOTs. For example, for the Florida D4 dataset, the method was derived from the Method column and then reassigned into the NCHRP method (see Table III-G-6). Much of the forecast accuracy database contained data for which the method was not clearly described in the available documentation—which meant several assumptions had to be made to sort them by the NCHRP codes, as discussed in the previous section. Table III-G-6. Description of forecast method in the forecast accuracy database. ID in Database Forecast Methodology Explanation 1 Traffic Count Trend Compound and Linear Growth Rate, Linear Interpolation, Regression Models etc. using Historical ADT or traffic count on a specific count station. 2 Population Growth Rates Forecasts based on Socio-Economic data, population forecasts on TAZ or project catchment area. 3 Project-Specific Travel Model Travel Demand Model created specifically for a project. 4 Regional Travel Demand Model Travel Demand Model for a region, e.g. Central Florida Regional Travel Model (CFRPM), Florida Standard Urban Transportation Model Structure (FSUTMS) etc. 5 Professional Judgment Usually a combination of traffic count trend and Travel Demand Model volume. 6 Unknown Methodology No record of method used. Several assumptions also were made to code the descriptions of forecasting agencies in the forecast accuracy database (see Table III-G-7). For example, for the Florida D4, Minnesota, and Wisconsin projects, the forecasting agency was assumed to be the state DOT (i.e., the forecast was created by state DOT employees or members). Consultants generating forecasts under contract with state DOTs (like the Florida District 5 projects) were categorized separately.

Appendix G: Large-N Analysis III-G-9 Table III-G-7. Descriptions of forecasting agency in the forecast accuracy database. ID in Database Forecast Agency 1 State DOT 2 Metropolitan Planning Organization 3 City/County Agency 4 Other Public Agency 5 Consultant 3 Method This study used Large-N analysis to measure the amount and distribution of forecast errors, including those segmented by variables such as project type and various risk factors. Large-N studies consider a larger sample of projects in less depth. Flyvbjerg (2005) extols the virtues of Large-N studies as the necessary means of coming to general conclusions. Often, Large-N studies include a statistical analysis of the error and bias observed in forecasts compared to actual data. Flyvbjerg et al. (2006) considered a Large-N analysis of 183 road and 27 rail projects, and Standard and Poor’s conducted a Large-N analysis with a sample of 150 toll road forecasts (Bain and Plantagie 2004). Other examples of Large-N studies are the Minnesota, Wisconsin and Ohio analyses conducted by Parthasarathi and Levinson (2010), Buck and Sillence (2014), and Giaimo and Byram (2013). This section presents a brief overview of the methods used in the existing literature and explains the method used in NCHRP 08-110. 3.1 Methods Used in Existing Literature As observed by Miller et al. (2016), the goal of Large-N analysis is to answer the question, “How close were the forecasts to observed volumes?” In order to facilitate answering this question, researchers have generally looked at two sets of similar data: one during the opening year and the other one in the design year. Several authors have evaluated the accuracy of project level traffic forecasts by comparing them with the actual traffic counts. Odeck and Welde (2017) looked at 68 Norwegian Toll Road Projects, implemented between 1975 and 2013 and calculated the mean percentage error against the forecast value. A similar procedure is used by Li and Hensher (2010) in their study of 14 toll roads in Australia and by Flyvbjerg et al. (2006) for 183 toll projects from around the world. A summary of existing research and the methodologies used is given in Table III-G-8.

III-G-10 Traffic Forecasting Accuracy Assessment Research Table III-G-8. Summary of existing Large-N methodologies. Paper Research Data Analysis Procedure Odeck and Welde (2017) 68 Norwegian Toll Road Projects, implemented between 1975 and 2013. Mean percentage error compared with forecast value, Examine bias, and Efficiency of estimates using econometric framework Li and Hensher (2010) 14 Toll Roads in Australia Mean percentage error, Ordinary least squares (OLS) regression model, and Random effects regression models with percentage error as dependent variable Flyvbjerg et al. (2006) 183 projects around the world Percentage error Bain (2009) 104 international toll road, bridge, and tunnel case studies. Actual/forecast traffic Miller, Anam, Amanin, and Matteo (2016) 39 studies from Virginia Mean absolute percentage error for each segment, Median absolute percentage error for individual projects (both compared over the observed value) Parthasarathi and Levinson (2010) 108 project reports obtained from MnDoT Actual/forecast traffic 3.2 Evaluation Year From the forecast accuracy database and the project reports, it was observed that traffic forecasts are usually done for 3 years: 1. Opening Year, 2. Mid-Design or Interim Year (usually 10 years after Opening), and 3. Design Year (usually 20 years from Opening). The actual traffic counts were obtained from the DoT’s count stations. For example, the Florida D5 data had detailed traffic counts from their count stations from 1972 to 2016. Matching the count stations information with the traffic forecast report, it was possible to get the actual traffic count for a year. Three calculations of errors or percent difference from forecast could be performed: 1. Percent difference from the Opening Year forecast, 2. Percent difference from the Interim/Mid-Design Year forecast, and 3. Percent difference in a year in-between the Opening and Mid-Design years. (In the last case, the forecast traffic value could be interpolated.) The purpose of taking errors for different years is to evaluate whether forecast performance improves over time. Li and Hensher (2010) report that all other factors remaining unchanged, that the error in forecast reduces by 2.54 percentage points for every additional year since opening (i.e., we see annual improvements, on average, in the accuracy of forecasts as we move away from the start date). This finding is supported by Vassallo and Baeza (2007) with the evidence that traffic forecasting effectiveness for Spanish toll roads tends to improve over time. In particular the research by Vassallo and Baeza claimed that the average error was -35.18% for the first year, -31.14% for the second year, and -27.06% for the third year. NCHRP 08-110 focused on the evaluation of Opening-Year forecasts for the practical reason that the Interim and Design years had not yet been reached for the vast majority of projects.

Appendix G: Large-N Analysis III-G-11 3.3 Definition of Errors One of the differences in methodologies in previous Large-N studies has been how they have defined the Predicted Volume minus the Actual Volume such that a positive result is an overprediction. Odeck and Welde (2017), Welde and Odeck (2011), and Flyvbjerg, Holm, and Buhl (2005) defined error the other way, such that a positive value represents underprediction. There are also two schools of thought when presenting the error as a percentage: (1) over the actual traffic (Tsai, Mulley, and Clifton 2014 and Miller et al. 2016) vs. (2) over the forecast traffic (Flyvbjerg, Holm, and Buhl 2005; Nicolaisen and Næss 2015; and Odeck and Welde 2017). An advantage of the former approach is that the percentage is expressed in terms of a real quantity (observed traffic). An advantage of the latter approach is that when the forecast is made, uncertainty can be expressed in terms of the forecast Parthasarathi and Levinson (2010) evaluated the forecast performance by taking the ratio of actual and forecast traffic. PDFFi= Actual Count-Forecast Volume Forecast Volume *100% where PDFFi is the percent difference from forecast for project i. Negative values indicate that the actual outcome is lower than the forecast (overprediction), and positive values indicate the actual outcome is higher than the forecast (underprediction). The appeal of this expression is that it expresses the error as a function of the forecast, which is known first. The distribution of the PDFF over the dataset can answer the systematic performance of traffic forecasts. As for expressing the percent difference from forecast over the dataset, the use of mean percent difference from forecast and mean absolute percent difference from forecast have varied in different researches. Mean absolute percent difference from forecast has been acknowledged to “allow [researchers] to better understand the absolute size of inaccuracies across project” (Odeck and Welde 2017) since positive and negative values tend to offset each other in case of calculating the mean percent difference from forecast. We chose to continue in this tradition, but again translated it into the language of percent difference from forecast: (III-G-1) From the discussion above and the summary in Table III-G-8, we see basically two schemes for evaluating forecast performance: as a percentage error and as a ratio. Within those schemes, there is some disagreement as to whether the percentage error should be taken relative to the observed count or the forecast value, and as to the direction of the sign. In NCHRP 08-110, the project team continued in the convention as described in Odeck and Welde (2017) in which they expressed the percent error as the actual count minus the forecast volume divided by the forecast volume. We recognize that the Odeck and Welde approach differs from the standard convention of expressing percent error with the actual observation in the denominator. We found it more useful for understanding to express the error as a function of the forecast volume because the forecast volume is known at the time the project decision is made, whereas the actual volume is not. This means that if we know that we might expect a 10% error, then that 10% can be applied to the forecast volume. To make this distinction clear, we have expressed this as the percent difference from forecast (PDFF): Mean Absolute Percent Difference from Forecast (MAPDFF)= 1 n * |PDFFi| n i=1 where n is the total number of projects. (III-G-2) errors. Miller et al. (2016), CDM Smith et al. (2014), and Tsai, Mulley, and Clifton (2014) define error as value since the observed value is unknown (Miller et al. 2016). Beside these two methods, Bain (2009) and

III-G-12 Traffic Forecasting Accuracy Assessment Research Source: Flyvbjerg, Holm, and Buhl (2006) Source: Bain (2009) Figure III-G-1. Example histograms of forecast accuracy. This research reports distributions of the percent difference from forecast in terms of the percentage difference, . The mean as reported in the distribution gives the central tendency of the dataset, with median as the 50th percentile value and standard deviation as the spread. For categorical variables, this research employs violin plots (see Figure III-G-2). Violin plots are similar to histograms and box plots in that they show an abstract representation of the probability distribution of the sample. Rather than showing counts of data points that fall into bins or order statistics, violin plots use kernel density estimation (KDE) to compute an empirical distribution of the sample. In this research, we used the 5th and 95th percentile values as inter-quartile range as depicted in Figure III-G-2. The percentile values basically present the percentage of datapoints that fall below. In effect, this range depicts the 90% probability range of percent difference from forecast for any categorical variable. Figure III-G-2. Anatomy of a violin plot. 3.4 Distribution of Errors Researchers have presented the results of their Large-N studies mostly in histograms of percentage error, as shown in Figure III-G-1. Bain (2009) further fitted the distribution in a distribution fitting software, which suggested a normal distribution with mean 0.77 and standard deviation 0.26. Goodness of fit was measured by chi-squared statistics. To ascertain the significance of the statistics (biasedness), the t- test was also performed.

Appendix G: Large-N Analysis III-G-13 3.5 Bias Detection Odeck and Welde (2017) employed an econometric approach to determine the bias and the efficiency of the estimates by regressing the forecast value to actual value using the following equation: = + ̂ + , where = the actual traffic on project i, ̂ = the forecast traffic on project i, = a random error term, and α and β = estimated terms in the regression. Here α=0 and β=1 implies unbiasedness. Li and Hensher (2010) conducted ordinary least squares (OLS) analysis and used the random effect linear regression model to explain the variation in the error in forecast as a percentage over the explanatory variables (year open, elapsed time since opening, etc.). Miller et al. (2016) performed ANOVA (analysis of variation) tests on the median absolute percentage error on a limited number of explanatory variables (difference between forecast year and opening year, forecast method, duration of forecast and number of recessions between base year and forecast year). Both researchers found their models to be a good fit to explain the errors. The end-goal of such analysis is to present the range of errors of forecast based on several variables (e.g., when the project was opened, difference in the forecast year and existing year, and so forth). The NCHRP 08-110 research does so by following the Odeck and Welde (2017) structure, but introduces additional terms as descriptive variables: = + ̂ + + , where Xi = a vector of descriptive variables associated with project i, and γ = a vector of estimated model coefficients associated with those descriptive variables. To consider multiplicative effects as opposed to additive effects, we also consider a log- transformed model: = + ̂ + + , which is equivalent to: = ̂ . In such a formulation, α=0 and β=1 still implies unbiasedness, ignoring all other terms. In addition to the estimation of biases, we are also interested in how the distribution of PDFF relates to different descriptive variables. For example, it may be that forecasts with longer time horizons remain unbiased, but have a higher spread, as measured by the MAPDFF. To do this, we extend the above framework to use quantile regression instead of OLS regression. Whereas OLS predicts the mean value, quantile regression predicts the values for specific percentiles in the distribution (Cade and Noon 2003). Quantile regression has been used in transportation in the past for applications such as quantifying the effect of weather on travel time and travel time reliability (Zhang and Chen 2017), where an event may have a limited effect on the mean value but increase the likelihood (III-G-3) (III-G-4) (III-G-5) (III-G-6)

III-G-14 Traffic Forecasting Accuracy Assessment Research of a long delay. It has also been used to estimate error bounds for real-time traffic predictions (Pereira et al. 2014), an application more analogous to this project. In our case, we chose to estimate quantile regression models of the actual count as a function of the forecast and other descriptive variables. We have done so for the 5th percentile, the 20th percentile, the median, the 80th percentile and the 95th percentile. This establishes an uncertainty window in which the median value provides our expected value, or an “adjusted forecast,” the 5th or 20th percentiles provide a lower bound on the expected value, and the 80th and 95th percentiles provide upper bounds. 3.6 Level of Analysis: by Segment or by Project While assessing the project forecast accuracy, one question arises: What constitutes an observation? A typical road project is usually divided into several links or segments within the project boundary. The links are usually on different alignments or carrying traffic to different directions. To uniquely identify each project in the forecast accuracy database, a column was specified, titled “Internal Project ID.” This column typically contains the unique Financial ID of the project, Report Number, Control Number, or other unique identifier. Under the same Internal Project ID, forecast and traffic count information for the different segments could be recorded with unique Segment IDs. Analysis thus could be done on two levels: 1. Segment Level: assessing the accuracy of the forecast for each different segment or link. 2. Project Level: assessing the total accuracy of forecast for each individual project, identified by their Unique Internal Project ID. The limitation of presenting accuracy metrics at a segment level is that the observations are not independent. Consider, for example, a project with three segments that are connected end to end. It is reasonable to expect that the PDFF on these segments is correlated—perhaps uniformly high or low. Whether we treat these as one combined observation or three independent observations, we would expect the average PDFF to be roughly the same. There would be a difference, however, in the measured t- statistics, where the larger sample size from a segment level analysis could suggest significance where a project-level analysis would not. Project-level analysis seems to be free of the correlation across observations described, but still the question remains on how to assess the accuracy for a project. In the Virginia Study (Miller et al. 2016) where each project consisted of links ranging from 1 to 2493 in number, the researchers took the Median Absolute Percent Error over the segments or links for individual projects and then used the Mean to express the level of accuracy. Nicolaisen (2012) measured the accuracy by taking the sum of forecast and actual traffic volumes on the segments in a project. Another method that can be used is taking the weighted traffic volume as described in Miller et al. (2016): The issue with using the weighted traffic volume (forecast and actual) is the absence of length data in most of the records. In addition, taking the total traffic as Nicolaisen (2012) will not be able to show the relation between forecast accuracy and project type by vehicles serviced. Taking these into consideration, in this study we measure the inaccuracy at the project-level using average traffic volumes, where each segment within a project is given equal weight. (III-G-7)

Appendix G: Large-N Analysis III-G-15 We report the distribution of percent difference from forecast both at a project level and a segment level. The results, presented later in this chapter, show that averaging to the project level appears to average out some of the errors observed at a segment level. We report a number of one-dimensional metrics at a project level, but estimate the econometric models at a segment level. 3.7 Data Cleaning and Filtering As mentioned previously, our primary objective for analysis is to compare the forecast traffic with the actual post-opening traffic. The forecast accuracy database presents challenges in the analysis due to the differences in record-keeping practices of the contributing states (explained in Section 2 of this appendix). We arrived at a uniform scheme or algorithm to clean up the missing information and prepare the flat data for analysis. First of all, we filtered out the records in the database that don’t have any actual traffic count data and those which haven’t been completed yet. The second filter may seem redundant, but in the database we have records of actual traffic count even though the project was forecast to be completed at a later date. This discrepancy occurred mostly for projects on existing roadways that had traffic count stations on them which produced regular count data. The second step was to select the appropriate actual traffic count for the records filtered out in the first step. This step was necessary because in many cases traffic counts were collected on a regular basis on the same segments over several years. Selecting the earliest traffic count after project completion was often not obvious, because data from several states did not mention actual project completion dates. For such types of projects, we employed the following reasoning: a. Categorize the segments by schedule risk. Based on the improvement types, we created low-risk and high-risk categories. The “resurfacing, slips, slides, safety improvements etc.” projects that are usually completed on or within 1 year of the forecast opening year were categorized as low-risk projects. Complex projects like adding lanes, new construction, or projects to increase capacity and built within 2 to 3 years of the planned opening date were categorized as high-risk projects (Mark Byram, Personal Communication, April 3, 2018). b. Create a 1-year buffer for low-risk projects and a 2-year buffer for high-risk projects, and keep the first traffic count outside the buffer. For example, if a project to add lanes had a forecast opening year of 2010, we would keep the first count available in year 2012 or later. We did this because we did not know if construction had been delayed from what was originally planned, and we wanted to avoid a situation where we evaluated a project against a traffic count taken before the project opened. Next, we scaled the forecast to the year of the first post-opening count so that both data points would fall in the same year. We did this by linearly interpolating the forecast traffic between the forecast opening year and the design year, usually 20 years later. (The European projects were taken from Nicolaisen’s PhD thesis (Nicolaisen 2012) and had already been scaled to match the count year using a 1.5% annual growth rate. We maintained this logic for the European projects, but did the interpolation between opening and design year for the U.S. projects.) For project-level analysis, we took the average of the traffic volumes and measured the error statistics by comparing the average forecast and average actual traffic. Aggregating the counts and forecasts across the segments/links was done by the unique identifier in the column “Internal Project ID.” The variables for analysis were also aggregated by the same unique identifier, albeit with different measures for maintaining uniformity. The improvement type, area type, and functional class of a project were taken to

III-G-16 Traffic Forecasting Accuracy Assessment Research be the same as the most prevalent one among the segments. For example, if most of the segments in a project were of “Improvement Type 1,” the project was considered to be of Improvement Type 1. The forecast method was the same across the segments for a project, as were the unemployment rates and years of forecast and observation. The mean of each of these values was taken for the project-level analysis. 3.8 Outliers As part of the data cleaning process, we conducted an outlier analysis in which we examined specific records with a high deviation between forecast and counts. To identify the outliers in the analysis, the first order of business was getting the links or segments that showed a significantly large percentage difference. As a trial, we selected the rows, or links, with greater than 75% of absolute percentage difference from forecast. In all, there were 399 segments with an absolute PDFF value of over 75%, out of which 242 had a PDFF over 100% and 88 went over 200% in PDFF. We then manually inspected each of these segments. We looked at the original datasets that were provided by the DOTs and the forecast reports where they were available to identify potential sources of difference. Except for 18 segments, all the other outliers (absolute PDFF>75%) appeared to be due to input error in the database. This was in part due to a bug (which we have since corrected) in importing the data into the forecast accuracy database wherein the macro that attached the actual traffic count linked to a different segment under the same Project ID but without any actual count information. The rest of the outliers (except for the 18 remaining segments) also appeared to result from input errors in the data provided by DOTs. For example, in one segment the forecast traffic volume was only 10780 while the actual count was 129013. The specific project was a resurfacing one over a ramp on an Interstate, which made us infer that the error was in reporting the actual count information. Another case we encountered had an incorrect Station ID, which was assigned to a freeway whereas the project was on a ramp. Employing similar reasoning, we looked at the outliers and decided to either keep or discard them from our analysis. This step of the filtering reduced the number of available segments from 4,112 to 3,912. 3.9 Calculating the Number of Lanes Required One of the implications of an inaccurate forecast is how it could influence project decisions. The number of lanes required for the roadway to operate at a certain level of service (LOS) is a variable that is dependent on the anticipated traffic. In his Virginia Study, Miller et al. (2016) explored a variant of this in the decision concerning the LOS. One of the projects (called studies in Miller’s research) had seen an LOS of E instead of the target LOS of C because of forecast errors. The research identified two distinct factors that affect the impact of error on decision making: 1. The magnitude of the error and 2. The location of the error relative to the performance criterion. Replicating the method employed in the Virginia Study in our analysis was problematic because of the absence of critical information to calculate the LOS. The existing and forecast number of lanes and the K-factor used were not specified for most of the projects, and we would be dealing with a very small sample size. Besides, other factors influencing the LOS (e.g. lane width, traffic composition, grade, and speed) were not coded into the database. Another way to assess the impact of forecast error is to calculate the number of lanes required for a given traffic volume. Project traffic forecasts ultimately are used to determine how many lanes a corridor

Appendix G: Large-N Analysis III-G-17 or project may require. Using the best available current-year data, and projecting future values of directional design hourly volume (DDHV), service flow rate for LOS I (SFi) and peak hour factor (PHF), the number of lanes can be estimated. Using the method described in the 2010 edition of the Highway Capacity Manual (HCM 2010) to calculate the service flow rate per lane for a required LOS and PHF, the number of lanes can be determined. According to it, the simplified equation for estimating the capacity of a roadway section is: where N = number of lanes, PHF = peak hour factor, = adjustment factor for heavy vehicles, and = adjustment factor for driver population. Rearranging the equation to determine the number of lanes for given traffic flow on a given direction, we get to: The traffic volume on a given direction can be alternately named as directional design hourly volume, which can be determined using: The K-factors represent typical conditions found around the state for relatively free-flow conditions and are considered to represent typical traffic demand on similar roads. The magnitude of the K-factor is directly related to the variability of traffic over time. Rural and recreational travel routes, which are subject to occasional extreme traffic volumes, generally exhibit the highest K-factors. The millions of tourists traveling on Interstate highways during a holiday are typical examples of the effect of recreational travel periods. Urban highways, with their repeating pattern of home-to-work trips, generally show less variability and thus have lower K-factors. Similarly, the directional distribution factor, D30, is based on the 200th highest hour traffic count report. But the problem remains as to the availability of 30 and 30 information for projects. The Florida Department of Transportation (Florida DOT) recommends values for the K and D-factor in case information on that is unavailable during project forecast. Table III-G-9 was obtained from the Project Traffic Forecasting Handbook prepared by the Florida DOT. Table III-G-9. Recommended and factors for traffic forecasting. Road Type Low Average High Low Average High Rural Freeway 9.6 11.8 14.6 52.3 54.8 57.3 Rural Arterial 9.4 11 15.6 51.1 58.1 49.6 Urban Freeway 9.4 9.7 10 50.4 55.8 61.2 Urban Arterial 9.2 10.2 11.5 50.8 57.9 67.1 (III-G-9) (III-G-10) (III-G-8)

III-G-18 Traffic Forecasting Accuracy Assessment Research The HCM-recommended range of values for selecting appropriate 30and 30factors for project forecast is also given in the following figures. Figure III-G-3. HCM-recommended K-factor range. Figure III-G-4. HCM-recommended D-factor range. For a simple analysis, we chose the average values in each subsection as recommended by the Florida DOT. The equations for determining the base capacity for the roadway types are also recommended in HCM 2010, which are presented in Table III-G-10. In the absence of information on free flow speed, in our analysis we assumed the maximum lane capacities by default. Table III-G-10. Equations to determine service flow rate or maximum capacity. Roadway Type Equation Freeway (Interstate) 1700 + 10 × Free Flow Speed up to 2400 Multilane Highway 1000 + 20 × Free Flow Speed up to 2200 Rural 2-lane Highway Up to 1600 Signal Controlled Facility 1900 × green ratio The peak hour factors (PHFs) are taken as the default values given in the Highway Capacity Manual (HCM 2010): 0.92 for urban facilities and 0.88 for rural ones. Assuming similar LOS for forecast traffic and actual traffic and using Equations III-G-9 and III- G-10, we first calculated the number of lanes required for each case and then compared them with each other (details are provided in the next section). We used the upper bounds for the N values, as specified in the HCM. 4 Data Exploration This section presents the key findings from the Large-N analysis and building on the method prescribed in Section 3 of this appendix. Reiterating the key points, the Large N analysis hinges upon the following: 1. Typical road projects are divided into one or more segments; 2. Traffic volume is generally predicted for opening year, mid-design year (typically 10 years from opening) and design year (usually 20 years into the future);

Appendix G: Large-N Analysis III-G-19 3. Actual traffic volumes to compare against the forecast volumes are taken for the year after the project has been completed (and for records in the database that did not have a project completion date, a buffer of at least 1 year was created based on the type of project); 4. Error is calculated as the difference between the actual volume and forecast volume as percent prediction; 5. For aggregation, the mean of the absolute percent difference from forecast was used as the metric, since positive and negative values would neutralize each other in case the means of the percent difference from forecasts were taken. The distributions, however, were taken on the percent difference from forecast. Bearing these points in mind, the Large-N analysis was done in two ways: by segments for the general distribution of PDFF and by project level for the effect of PDFF on an aggregated level. As described in Section 3 of this appendix, the forecast accuracy database contains about 16360 unique records. The records contain forecast information by segments, forecast year type (opening, mid-design or design year) and actual count information, if applicable. For analysis purpose, the filters were applied and we got to 4278 unique records. All of these 4278 records have a traffic forecast, actual count in a year after the project has, or presumed to have, opened. Rerunning the analysis with the outliers and duplicates removed, we were left with 3912 unique records. The data-frame to be analyzed contained project information (unique project ID, type of project, segment ID, roadway functional classification, area type), forecast information (year forecast was produced, forecast year, forecast and adjusted traffic) and the actual count information (year of observation, count, station ID). Based on the nature of the forecast accuracy database, we could select some variables that might dictate future adjustments in the forecasts. These variables were: the type of project (Improvement Type), the method used (Forecast Methodology), the roadway type (Functional Class), the area type (Area Type Functional Class), and the Forecast Horizon (the difference between the year the forecast was produced and the year of opening). Table III-G-11 tabulates the descriptive variables chosen for use in our analysis. difference from forecast, so that negative value means overprediction and positive means under-

III-G-20 Traffic Forecasting Accuracy Assessment Research Table III-G-11. Descriptive variables for analysis. Variable Explanation Forecast Volume We expect the percent difference from forecast to be larger for lower volume roads because there are less opportunities for PDFFs to average out. Functional Class To test whether accuracy differs for different functional class of roads. The distribution is done on the FHWA-defined functional classes. Area Type To test whether urban or rural areas influence the forecast accuracy. Type of Project Distribution of PDFFs across different types of improvement (i.e., resurfacing project, adding lanes, new construction, etc.). Can be simplified as forecasts on existing roads and new constructions. Tolls Relation between toll forecasts and untolled road forecasts. Opening Year Projects affected by a recession may have uniformly low forecasts. The opening year is taken to be the year the actual traffic count was taken in our database. The years 2001 and 2008–9 were identified as recession years. Judging from the unemployment rate, the years affected by the recession were categorized. Year Forecast Produced To evaluate whether forecast accuracy has improved over the years. Forecast Horizon Derived variable from the difference between the forecast year and the year the forecast was produced. Test hypothesis that forecasts are better when the opening year is closer to the year the forecast was produced. Unemployment Rate in Opening Year To evaluate the effect of recessions on forecast accuracy. Change in Unemployment Rate This will be measured as the difference between the unemployment rate in the opening year and the unemployment rate in the year the forecast was produced. Forecast Type To evaluate the relative accuracy of trend-based forecasts or model-based forecasts, etc. Type of Forecaster To examine differences between forecasts made by DOTs, Metropolitan Planning Organizations, consultants, or others. Agency To test whether some agencies produced more accurate forecasts than others (identified as Agency A, Agency B, etc.). Review Indicates whether forecasts have gone through a review process. The remainder of this appendix examines the overall distribution of percent difference from forecast, as well as the percent difference from forecast segmented by each of these factors. 4.1 Overall Distribution Generally speaking, traffic forecasts have been found to be overpredicting: Actual traffic volumes after a project has been completed are lower than what has been forecast. This effect is shown in Figure III-G-5 and Figure III-G-6, which show a right-skewed distribution. As seen in Table III-G-12, the mean of the absolute percentage difference from forecast (MAPDFF) is 24.67% with a median of 16.69%, but these statistics are biased in the sense that multiple segments make up a single project, and a particular error or shortcoming of the method adopted accumulates over a project. In segment-level analyses, the traffic volumes are off by about 5150 vehicles per day on average.

Appendix G: Large-N Analysis III-G-21 Figure III-G-5. Distribution of percent difference from forecast (segment level). Figure III-G-6. Distribution of percent difference from forecast (project level). The 3,911 unique records/segments are part of 1,291 unique projects. Similar to our segment- level analysis, we noticed a general overestimation of traffic across the projects. The distribution of percent difference from forecast shown in Figure III-G-5 is heavier on the negative side (i.e., the actual volumes are generally lower than the traffic forecasts). As seen in Table III-G-12, the mean of the absolute percent difference from forecast is 17.29% with a standard deviation of 24.81. The Kernel Density Estimator displays an almost normal distribution, albeit with long tails. On an average, the traffic forecasts for a project are off by 3500 vpd.

III-G-22 Traffic Forecasting Accuracy Assessment Research We should expect overpredictions because, in many cases, these forecasts are used in design engineering. A design based on overpredicted traffic will typically be overbuilt and will not see that extra capacity utilized. On the other hand, if the underpredicted traffic is used as a basis for design, it would mean adding capacity at a later time at a greater cost to meet the actual demand. Table III-G-12. Overall percent difference from forecast. Traffic Forecast Range (ADT) Observations Mean Absolute Percent Difference from Forecast Mean Median Standard Deviation 5th Percentile 95th Percentile Segment Level 3911 24.67 0.65 -5.49 41.92 -44.89 66.34 Project Level 1291 17.29 -5.62 -7.49 24.81 -37.56 36.96 4.2 Forecast Volume Figure III-G-7 reports the difference from forecast as a function of forecast volume at the segment level. Figure III-G-8 shows it reported at the project level. They are reported separately here because the traffic volume can be quite different for different segments within a project, such as may be the case of a freeway interchange where the mainline freeway volume is much higher than the ramp volumes. An interesting observation from Figure III-G-7 is the low percentages as the traffic volumes increase. This is understandable, since the percentages were taken as a ratio over the forecast volume. Unless the actual traffic differs by a large margin, the percentage differences will not have risen to a big amount. Figure III-G-7. Percent difference from forecast as a function of forecast volume (segment level).

Appendix G: Large-N Analysis III-G-23 Figure III-G-8. Percent difference from forecast as a function of forecast volume (project level). Tables III-G-13 and III-G-14 show descriptive measures of percent difference from forecast of the forecasts by volume group for segments and projects, respectively. The measures represent the spread of the percent difference from forecast with the mean, standard deviation and 5th and 95th percentile values. The MAPDFF value for each category presents how much the actual traffic deviates from the forecast value. Mean is the central tendency of the data. Standard deviation and the 5th and 95th percentile data represent the spread of the distribution. In the tables, 90% of the data points fall between the 5th and 95th percentile values. Table III-G-13. Forecast inaccuracy by forecast volume group (segment level). Traffic Forecast Range (ADT) Observations Mean Absolute Percent Difference from Forecast Mean Median Standard Deviation 5th Percentile 95th Percentile 0–3000 359 36.17 14.04 -2.22 91.63 -44.78 106.91 3001–6000 419 26.64 3.90 -3.33 38.91 -40.03 83.78 6001–9000 394 24.83 -2.78 -8.93 33.06 -47.90 57.47 9001–13000 465 23.17 -2.54 -6.03 30.11 -44.49 54.98 13001–17000 353 25.31 -0.20 -3.34 34.49 -49.56 76.88 17001–22000 360 25.02 -5.21 -10.40 34.67 -51.54 65.85 22001–30000 415 28.01 3.87 -3.57 37.20 -47.40 77.78 30001–40000 386 25.71 -0.17 -7.92 35.23 -44.64 72.84 40001–60000 410 19.37 2.56 -0.89 26.34 -32.56 53.47 60000+ 350 12.38 -7.14 -6.40 14.98 -28.42 17.50

III-G-24 Traffic Forecasting Accuracy Assessment Research Traffic Forecast Range (ADT) Observations Mean Absolute Percent Difference from Forecast Mean Median Standard Deviation 5th Percentile 95th Percentile 0–3000 133 24.59 -1.85 -5.75 42.15 -45.01 75.17 3001–6000 142 20.53 -0.37 -4.64 29.74 -36.50 50.33 6001–9000 125 16.75 -5.68 -8.80 21.94 -35.29 36.67 9001–13000 145 15.59 -4.66 -7.29 19.99 -31.34 34.45 13001–17000 143 17.41 -6.20 -6.53 21.61 -37.76 30.65 17001–22000 113 17.98 -5.65 -8.31 25.47 -41.62 37.85 22001–30000 133 19.54 -5.65 -8.47 25.36 -40.31 41.75 30001–40000 115 15.56 -9.78 -10.26 18.23 -39.54 12.26 40001–60000 137 13.18 -8.95 -7.68 16.01 -34.44 7.49 60000+ 105 10.20 -8.96 -7.90 9.90 -24.50 3.68 One observation from Table III-G-14 is that as the forecast volume increases, the distribution of the percent difference from forecast has smaller spreads in addition to the MAPDFF value getting smaller. For example, for forecast volume between 30000 and 40000 ADT, percent difference from forecast for 90% of the projects lie between -39.54% and 12.26% with absolute deviation of 15.56% on average. 4.3 Functional Class The distributions of percent difference from forecast by functional class (Figure III-G-9 and Table III-G-15) are taken at the segment level, since a project may span over roadways of different functional class. Violin plots, as depicted in Figure III-G-9, show quantitative data with a kernel density estimation of the underlying distribution. The thick black bars represent the 25th and 75th percentile values, in effect depicting the range of values where 50% of the data-points fall in. These reiterate the point made about overprediction in forecasts: About 75% of the links have negative percent differences from the forecast values for Interstates, major arterials, and collectors. About 70% of the minor arterial links have been Table III-G-14. Forecast inaccuracy by forecast volume group (project level). overpredicted.

Appendix G: Large-N Analysis III-G-25 Figure III-G-9. Distributions of percent difference from forecast by functional class (segment-level analysis). Compared among themselves, it appears that forecasts for Interstates or limited access facilities fare better than do other classes of roadway, in terms of both the absolute deviation and the spread (Table III- G-15). Some 90% of the records of this functional class fall between -27.81% and 10.44%. The spread is greater for other functional classes (represented by the 5th and 95th percentile values). Table III-G-15. Forecast inaccuracy by functional class (segment-level analysis). Functional Class Observations Mean Absolute Percent Difference from Forecast Mean Median Standard Deviation 5th Percentile 95th Percentile Interstate or Limited Access Facility 434 12.32 -9.21 -8.48 13.58 -27.81 10.44 Principal Arterial 837 16.95 -9.63 -10.89 19.38 -37.51 23.95 Minor Arterial 404 18.92 -8.26 -10.24 24.54 -41.50 29.26 Major Collector 258 20.67 -10.81 -11.10 26.92 -51.11 23.85 Minor Collector 19 22.53 -12.74 -8.66 24.30 -41.43 28.58 Local 1 46.67 46.67 46.67 46.67 46.67 Unknown Functional Class 1958 32.42 10.69 2.68 53.67 -48.75 86.21 4.4 Area Type The distribution and spread of forecast differences as a function of the area type is presented in Figure III-G-10 . The spread for urban areas (-39.37% to 27.14%) is greater than that for rural areas (-27.93% to 24.72%). The MAPDFF values for rural and urban areas (14.09% and 17.66% respectively) point to traffic in rural or mostly rural areas having a smaller deviation from that predicted.

III-G-26 Traffic Forecasting Accuracy Assessment Research Table III-G-16 addresses forecast inaccuracy by area type at the segment level. As seen in the table, forecasts for both rural and urban areas are mostly overpredicting (i.e., actual traffic is less than forecast, as shown by 65% of the links in rural areas and 72% of links in urban areas). Figure III-G-10. Distribution of percent difference from forecast by project area type (segment-level analysis). The spread for urban areas (-39.37% to 27.14%) is greater than that for rural areas (-27.93% to 24.72%). The MAPDFF values for rural and urban areas (14.09% and 17.66% respectively) point to traffic in rural or mostly rural areas having a smaller deviation from predicted. Table III-G-16. Forecast inaccuracy by area type (segment-level analysis). Area Observations Mean Absolute Percent Difference from Forecast Mean Median Standard Deviation 5th Percentile 95th Percentile Rural or Mostly Rural 210 14.09 -4.02 -5.56 18.22 -27.93 24.72 Urban 543 17.66 -8.05 -9.58 22.32 -39.37 27.14 Unknown Area Type 3047 23.86 -0.12 -5.00 33.89 -47.31 68.05 4.5 Type of Project As described in Section 2 of this appendix, the forecast accuracy database includes the improvement type of the project as a required field. A lot of the segments/projects did not have any improvement type assigned but we were still able to unify the types coded in the database in three ways, designating them as: 1. Improvement on existing facility (i.e., resurfacing projects, replacement projects, and projects for adding capacity to existing roadway), 2. New construction (i.e., for new general-purpose, dedicated, managed or reversible lane(s) facilities), and 3. Unknown project type.

Appendix G: Large-N Analysis III-G-27 Among the 1,291 projects, the forecast accuracy database contained forecast and actual count information on only 28 new construction projects, whereas projects on existing roadways numbered 788. About 75% of the projects on existing roadways in the database had percentage differences below 0% (i.e., overpredicted the traffic). Similar proportions were obtained for new constructions (see Figure III-G-11 and Table III-G-17). Compared to aggregated differences over all types of projects (MAPDFF of 17.29%), forecasts for existing roadways have on an average slightly less percentage differences (MAPDFF of 16.26%). Forecasts for new constructions are even more accurate (MAPDFF of 10.57%). Figure III-G-11. Distribution of percent difference from forecast by project type (project- level analysis). The difference in sample sizes makes commenting on the relative accuracy of forecasts by project type difficult. But as the percentile values indicate, forecasts for new construction projects have a lower spread than that for existing roadways. Table III-G-17. Forecast inaccuracy by project type (project level). Project Type Observations Mean Absolute Percent Difference from Forecast Mean Median Standard Deviation 5th Percentile 95th Percentile Existing Road 899 16.26 -5.90 -7.43 23.55 -36.20 29.93 New Facility 28 10.57 -9.22 -8.76 9.54 -19.34 3.83 Unknown Type 364 20.36 -4.64 -7.64 28.38 -43.96 45.95 4.6 Tolls In our database we didn’t have much information about the toll roads. In all, there is forecast information on only 7 roads/links with static tolls on 1+ lanes. The MAPDFF for the tolled roads is 20.41% with a maximum of 93.38%. The distribution in Figure III-G-12 is not scaled by the number of observations. Table III-G-18 presents the breakdown of the distribution by toll type on links.

III-G-28 Traffic Forecasting Accuracy Assessment Research Figure III-G-12. Distribution of percent difference from forecast by toll types (segment-level analysis). Table III-G-18. Forecast inaccuracy by toll type (segment level). Toll Type Observations Mean Absolute Percent Difference from Forecast Mean Median Standard Deviation 5th Percentile 95th Percentile No Tolls on 1+ lane 3432 23.66 -1.53 -6.55 32.87 -45.9 64.66 Static Tolls on 1+ lane 7 20.41 16.16 8.60 34.96 -7.97 68.85 4.7 Year Forecast Produced The forecast accuracy database includes forecasts produced between 1962 and 2017. The analysis is limited to projects for which post-opening traffic counts are available by 2017, and within that set of projects, the most recently produced forecasts were made in 2014. In Figure III-G-13 and Table III-G-19, we compare the percent difference from forecast for forecasts produced in each year. The MAPDFF has steadily gone down, in addition to the spread of the distribution getting smaller. Also noticeable is the overall underprediction of traffic for projects that were forecast between 1981–1990 (i.e., actual traffic was more than the forecast volume). During the next decade (1991–2000), about 55% of the projects for which traffic was forecast had more traffic than forecast. After 2000 however, almost 75% of the projects forecast have seen less traffic than forecast, with an average absolute deviation of 15.7%. The improvement over time may suggest that the availability of better data and refinement as well as sophistication of forecasting method result in better forecast performance over the years. However, it also could be affected by the mix of projects and broader socioeconomic trends. Many of the earlier projects were larger in scale, and the 1970s through 1990s were a time of growing automobile ownership, increasing numbers of women in the workforce, and higher VMT per capita. Projects from the 2000s, by contrast, include more routine projects at a time of slower economic growth and slower growth in VMT per capita.

Appendix G: Large-N Analysis III-G-29 Figure III-G-13. Distribution of percent difference from forecast by the year forecast produced. Table III-G-19. Forecast inaccuracy by year forecast produced. Year Forecast Produced Observations Mean Absolute Percent Difference from Forecast Mean Median Standard Deviation 5th Percentile 95th Percentile Before 1980 94 30.76 11.25 8.98 39.89 -47.12 83.27 1981–1990 45 34.83 28.21 28.53 34.18 -19.96 86.28 1991–2000 51 23.17 11.13 -1.87 48.07 -24.79 53.56 2001–2010 924 15.79 -9.96 -10.32 18.23 -38.36 15.95 After 2010 177 11.83 -5.36 -2.65 18.81 -38.65 15.62 Analyzing the forecast accuracy for projects on existing roadways, we see similar trends, although after 2010 the MAPDFF declined, from 15.79% in the previous decade to 11.83%. Figure III-G-14 and Table III-G-20 present the distribution of inaccuracy in projects on existing roads. Figure III-G-14. Distribution of percent difference from forecast for projects on existing roadways, by year forecast produced.

III-G-30 Traffic Forecasting Accuracy Assessment Research Table III-G-20. Forecast inaccuracy for projects on existing roadways, by year forecast produced. Year Forecast Produced Observations Mean Absolute Percent Difference from Forecast Mean Median Standard Deviation 5th Percentile 95th Percentile Before 1980 26 25.59 21.13 21.22 25.87 -14.21 60.72 1981–1990 14 44.76 44.76 42.17 31.76 4.70 96.30 1991–2000 49 23.58 12.12 -1.87 48.74 -23.82 54.21 2001–2010 680 15.78 -9.54 -9.78 18.37 -38.59 18.50 After 2010 130 11.08 -4.51 -1.98 18.68 -32.53 16.39 4.8 Opening Year The distribution of percent difference from forecast by the project opening year presented in Figure III-G-15 and Table III-G-21 is a useful indicator of forecast performance over the years. As can be seen, the forecast performance has generally gotten better after 2000, with significantly lower MAPDFF values than in the previous decade, as well as smaller spreads. Most of the projects (about 78%) that have opened to traffic between 1991 to 2002 have had more traffic than forecast. Percent difference from forecast from 2003 to 2008 are more evenly spread (90% data points between -36.82% and 33.46%), while after 2012 the actual counts have generally been less than the forecast values (78% of the projects). Figure III-G-15. Distribution of percent difference from forecast by opening year of project. The opening years have been categorized to assess the effect of recession (a recession in 2001 and the great recession of 2008–09) on forecast performance. It is assumed that the 2001 recession would affect unemployment rates until 2002, and the great recession would do so until 2012, based on the unemployment rate for those years. One thing to notice is that during and after the recession years, the actual traffic was lower than usual. The median values (corresponding to 50th percentile value) give a good approximation, as 50% of the projects opened since 2012 have experienced traffic at least 5.78% less than the forecast values.

Appendix G: Large-N Analysis III-G-31 Table III-G-21. Forecast inaccuracy by project opening year. Opening Year Observations Mean Absolute Percent Difference from Forecast Mean Median Standard Deviation 5th Percentile 95th Percentile Before 1990 92 30.14 12.98 9.64 38.24 -43.71 89.49 1991–2000 72 28.09 15.83 3.74 45.17 -28.66 62.88 2001–2002 15 15.65 6.69 3.74 22.50 -22.86 51.82 2003–2008 351 18.92 -7.98 -11.52 23.76 -36.82 33.46 2009–2012 512 14.22 -9.21 -8.46 17.08 -35.07 12.25 After 2012 249 13.56 -8.73 -5.78 18.41 -42.71 13.45 Again, it is not clear the degree to which the differences observed are a function of different forecasting methods, events in the real world, or a mix of the two. Looking strictly at the projects done on existing roadways, a similar distribution was observed. The ranges have become tighter, with a lower MAPDFF value (except for the projects opening between 1991 and 2000). The distribution and statistical results are given in Figure III-G-16 and Table III-G-22. Figure III-G-16. Distribution of percent difference from forecast for projects on existing roadways by opening year of project. 2009-20122003-2008

III-G-32 Traffic Forecasting Accuracy Assessment Research Table III-G-22. Forecast inaccuracy in projects on existing roadways, by opening year. Opening Year Observations Mean Absolute Percent Difference from Forecast Mean Median Standard Deviation 5th Percentile 95th Percentile Before 1990 40 32.30 29.40 25.32 29.93 -11.69 90.59 1991–2000 49 23.58 12.12 -1.87 48.74 -23.82 54.21 2001–2002 11 13.88 3.47 -0.75 20.88 -24.81 34.60 2003–2008 247 17.69 -9.21 -11.94 20.24 -35.99 20.27 2009–2012 373 13.95 -8.82 -8.44 16.82 -35.23 13.72 After 2012 179 13.68 -8.65 -5.78 19.08 -42.45 14.12 4.9 Forecast Horizon Another question that comes to mind while evaluating the accuracy of forecasts is whether the number of years that elapsed between the time the forecast was produced and the time the project was opened has a bearing on the accuracy. As is evident from Figure III-G-17 and Table III-G-23, the average of the absolute percent difference from forecast increases as the number of years elapsed increases, except for the same-year projections. The difference in years introduces other variables (e.g., the micro economy and macro economy, changes in land use and fuel prices, and so forth) that can directly affect the traffic. These are all variables that are difficult to predict, and their effects are evident. This finding is consistent with findings by Bain (2009), who identified the critical dependence of longer-term forecasts on macro- economic projections. According to the Standard and Poor’s studies (2002–2005), “A number of comments were recorded about the relationship between economic growth and traffic growth; concerns being raised about traffic forecasts—particularly over longer horizons—relying on strong and sustained economic growth assumptions that resembled policy targets rather than unbiased assessments of future economic performance.” Figure III-G-17. Distribution of percentage differences by forecast horizon.

Appendix G: Large-N Analysis III-G-33 Forecasts that go beyond 5 years tend to be less accurate (although still relatively unbiased) and when the project opens the actual traffic count tends to have a higher percent difference from forecast (90% of the data points fall within -45% to 72%, with a MAPDFF of 29%.) Table III-G-23. Forecast inaccuracy by forecast horizon. Forecast Horizon (Years) Observations Mean Absolute Percent Difference from Forecast Mean Median Standard Deviation 5th Percentile 95th Percentile 0 165 20.10 8.08 0.00 34.77 -25.18 57.71 1 206 12.88 -9.20 -8.12 14.64 -36.32 11.38 2 340 15.23 -7.79 -7.64 19.93 -40.26 20.38 3 251 16.25 -10.36 -10.74 18.49 -37.02 17.29 4 131 16.05 -10.36 -12.16 16.87 -35.43 20.19 5 67 16.82 -10.44 -13.82 22.23 -43.99 13.40 5+ 131 29.55 4.71 -3.13 39.47 -44.73 72.07 A point of concern in this analysis has to be why the MAPDFF value, as well as the range of forecast differences, is higher for a forecast horizon of 0 year. Some 50% of the observations fall on either side of 0% difference. 4.10 Unemployment Rate in Opening Year State-level unemployment rate data were pulled from the U.S. Bureau of Labor Statistics and then matched with the year each actual traffic count was taken. For the European projects, the unemployment rate was measured at the national level. The rates were categorized into 7 classes or ranges, and Figure III- G-18 presents the distribution of percent difference from forecast. Except for unemployment rates below 3, the percent difference from forecast hovers in the negative side (i.e., there is an overprediction for all other ranges). For unemployment rates between 1 to 3, the actual traffic exceeds the forecast volume for most of the cases, but this statistic should be taken with a grain of salt given the small sample size. Figure III-G-18. Distribution of percent difference from forecast by unemployment rate in opening year.

III-G-34 Traffic Forecasting Accuracy Assessment Research Of the projects that opened in years with an unemployment rate in the range of 7–8%, 72% of the forecasts overpredicted the traffic, with average absolute deviation of 17.3%. Comparing between the ranges, unemployment rates between 3 and 5 seem to produce the maximum absolute deviation from forecast volume. Other ranges hover close to the overall average. A breakdown of the statistics is given in Table III-G-24. Table III-G-24. Forecast inaccuracy by unemployment rate in the opening year. Unemployment Rate Observations Mean Absolute Percent Difference from Forecast Mean Median Standard Deviation 5th Percentile 95th Percentile Up to 3 4 19.44 16.73 13.08 21.70 -3.21 41.78 3–5 229 22.95 2.13 -2.84 36.05 -40.20 55.83 5–7 371 16.10 -7.35 -7.68 21.30 -39.70 26.86 7–8 128 17.30 -7.05 -6.45 24.00 -43.19 26.12 8–9 168 17.07 -5.41 -7.51 24.68 -33.34 35.09 9–10 35 18.17 -5.15 -11.22 22.33 -28.14 39.05 10+ 356 14.90 -8.68 -9.64 18.08 -34.43 19.60 4.11 Change in Unemployment Rate To assess the impact of changes in the unemployment rate on forecast inaccuracy, we took the difference of the unemployment rate between the project opening year and the year the forecast was produced. At least 70% of the projects for which the unemployment rate changed by at least ±4% exhibited actual traffic less than the forecast value. The distributions of the percent difference from forecast are presented in Figure III-G-19 and Table III-G-25. Figure III-G-19. Distribution of percent difference from forecast, by change in unemployment rate from forecast year and opening year. An interesting but not quite unexpected observation is the spread of the distribution for cases in which the unemployment rate increased in the opening year from the year forecast was produced by at

Appendix G: Large-N Analysis III-G-35 least 2 points. Some 90% of the projects fall either between -36.1% to 26.67% for change of 2–4% and between -35.26% to 18.78% for change of 4–6%. With an increase in the unemployment rate, it stands to reason that the actual traffic would be less. The possibility of underprediction would thus get even lower. Table III-G-25. Forecast inaccuracy by change in unemployment rate. Change in Unemployment Rate Observations Mean Absolute Percent Difference from Forecast Mean Median Standard Deviation 5th Percentile 95th Percentile (-8, -6] 8 15.01 -8.69 -2.02 19.29 -32.69 15.29 (-6, -4] 93 14.91 -5.63 -7.18 20.30 -31.30 31.45 (-4, -2] 136 19.21 4.45 -0.67 31.39 -30.61 54.60 (-2, 0] 367 17.64 -4.27 -6.16 27.88 -38.82 36.58 (0, 2] 263 16.8 -6.00 -6.32 23.27 -40.58 30.62 (2, 4] 217 17.05 -8.01 -8.63 22.12 -36.09 26.67 (4, 6] 166 17.54 -11.75 -13.94 17.80 -35.26 18.78 4.12 Forecast Method One derivative of the Large-N analysis is assessing the performance of the tools at disposal for the state DOTs and metropolitan planning organizations. For project-level traffic forecasting, NCHRP Report 765 (CDM Smith et al. 2014) examines various methods that are in use and presents a guideline for employing them. Still, one question should arise: Does the forecast performance depend on the method used? A follow-up question might be, “Is a certain type of forecast method better for a certain type of project or even a certain type of roadway?” A field in the forecast accuracy database was specified to record the method used to forecast the traffic for each project. The coded methodologies were: Traffic Count Trend, Population Growth Rate, Regional Travel Demand Model, Project-Specific Travel Demand Model, Professional Judgment, and Unknown Methodology. For the 1,291 projects selected for the Large-N analysis, 252 forecasts were created using traffic count trends, 179 were created using the Regional 4-Step Travel Demand Model, and 177 were created using professional judgment. “Professional judgment” referred to the use of both count trends and volumes from demand models, combined as the forecaster saw fit. Here again, the problem of missing data arose, as 676 of the projects in the database had no data regarding the method used to forecast the traffic. Figure III-G-20 and Table III-G-26 present distributions of inaccuracy.

III-G-36 Traffic Forecasting Accuracy Assessment Research Figure III-G-20. Distributions of percent difference from forecast by forecast method. Table III-G-26. Forecast inaccuracy by forecast method. Forecast Method Observations Mean Absolute Percent Difference from Forecast Mean Median Standard Deviation 5th Percentile 95th Percentile Traffic Count Trend 252 22.21 -0.10 -5.22 31.24 -39.34 55.06 Population Growth Rate 7 11.32 -2.18 -0.35 13.56 -16.43 13.89 Regional Travel Demand Model 179 16.88 -8.42 -9.75 21.76 -44.91 27.16 Professional Judgment 177 17.84 -11.77 -11.94 19.87 -43.11 18.52 Unknown Methodology 676 15.49 -5.36 -6.45 23.67 -34.39 29.49 The distributions of percentage differences by forecast methodology (Table III-G-26) suggest that forecasts created using a regional travel demand model are more accurate when comparing the MAPDFF values (the MAPDFF listed for the Regional Travel Demand Model is 16.88, compared to 22.21 for the Traffic Count Trend model). In addition, trend analysis cannot be used on all types of projects, whereas models can be used on virtually any type of project, suggesting that models may be more accurate for a more challenging set of projects to forecast. 4.13 Type of Forecaster Figure III-G-21 and Table III-G-27 present the distribution of forecast inaccuracy by the forecaster. As can be seen, 90% of the projects forecast by state DOTs fall in the range of -44.94% and 54.32%. Some 50% of these projects are overpredicted. The spread for forecasts done by consultants is lower (90% of

Appendix G: Large-N Analysis III-G-37 the projects lie between -35.83% and 31.42%), as is the mean absolute deviation (MAPDFF of 17.36% compared to 21.47% for state DOT-produced forecasts). Figure III-G-21. Distribution of percent difference from forecast by type of forecaster. Table III-G-27. Forecast differences by type of forecaster. Forecasting Agency Observations Mean Absolute Percent Difference from Forecast Mean Median Standard Deviation 5th Percentile 95th Percentile State DOT 489 21.47 -0.89 -5.58 32.34 -44.94 54.32 Metropolitan Planning Organization 2 6.86 -6.86 -6.86 0.90 -7.43 -6.29 Consultant 237 17.36 -6.36 -8.20 22.13 -35.85 31.42 4.14 Effect on Number of Lanes There is an old axiom that traffic forecast only need to be accurate to within half a lane. To test the extent to which we meet this standard, we calculated the number of lanes required for forecast traffic and the actual traffic, assuming the same LOS. The method for this calculation is described in Section 3 of this appendix. Comparing the two numbers, we found 37 links out of the 3912 (1.0%) required an additional lane to allow the traffic to flow at the forecast LOS. This small number reinforces our interpretation of overprediction in traffic forecasts. As for these 37 links, if the assumptions regarding the number of lanes hold true, the LOS would get worse. Of the 37 links, five were minor arterials and the rest were Interstate (16) and major arterials (16). Conversely, analyzing for the links that overestimate the traffic by an amount such that they could do with a lesser number of lanes per direction, we got to 158 links (4.2%). Of these links, 92 were Interstate, 64 were principal arterials, and the rest were minor arterials.

III-G-38 Traffic Forecasting Accuracy Assessment Research 5 Econometric Analysis The uncertainties involved in forecasting traffic call for assessing the risks and subsequently developing a range of traffic forecasts that can be expected on a project. Considering the current dataset to be representative (i.e., “national average”), we developed several quantile regression models to assess the biases in the forecasts on the variables described in the previous section. The models were developed on the 5th, 20th, 50th (median), 80th, and 95th percentile values. Apart from detecting bias of the traffic forecast, another goal of such econometric analyses is to obtain the range of actual traffic as a function of the forecast traffic and other project-specific criteria. The variables in the analysis are explained in Table III-G-28. Table III-G-28. Descriptive variables for regression models. Variable Name Explanation AdjustedForecast Forecast ADT value for a segment/link or project AdjustedForecast_over30k Variable to account for links with ADT value greater than 30,000. Defined as: If Forecast > 30,000 then value=Forecast – 30,000 Scale_UnemploymentRate_OpeningYear Unemployment rate in the Project Opening Year Scale_UnemploymentRate_YearProduced Unemployment rate in the year forecast was produced Scale_YP_Missing Binary Variable to account for missing information in the Year Forecast Produced Column in the NCHRP database Scale_DiffYear Difference in the year forecast produced and forecast year i.e. Forecast Horizon Scale_IT_AddCapacity Binary Variable for projects that add capacity to existing roadway. Reference class is the Resurfacing/Repaving/Minor Improvement projects Scale_IT_NewRoad Binary Variable for new construction projects Scale_IT_Unknown Binary Variable for projects of unknown improvement type Scale_FM_TravelModel Binary Variable for forecasts done using Travel Model. Reference class is the forecasts done using Traffic Count Trend Scale_FM_Unknown Binary Variable for forecasts done using unknown method Scale_FA_Consultant Binary Variable for forecaster, reference class being state DOTs Scale_Agency_BCF Binary Variable for projects under the jurisdiction of Agency B, C, or F, reference class being Agency A Scale_Europe_AD Binary Variable for European projects Scale_OY_1960_1990 Binary Variable for projects opened to traffic before 1990; the reference value for Opening Year is 2013 and later Scale_OY_1991_2002 Binary Variable for projects opened to traffic from 1991 to 2002 Scale_OY_2003_2008 Binary Variable for projects opened to traffic from 2003 to 2008 Scale_OY_2009_2012 Binary Variable for projects opened to traffic from 2009 to 2012 Scale_FC_Arterial Binary Variable for forecasts on Major or Minor Arterials. Interstate or Limited Access facility are kept as reference class Scale_FC_CollectorLocal Binary Variable for forecasts on Collectors and Local Roads Scale_FC_Unknown Binary Variable for forecasts on roadways of unknown functional class

Appendix G: Large-N Analysis III-G-39 5.1 Base Model In the first model, we regressed the actual count on the forecast traffic volume. The structure follows Equation III-G-3 (reported previously): where = the actual traffic on project i, ̂ = the forecast traffic on project i, = a random error term, and and = estimated terms in the regression. Here α=0 and β=1 implies unbiasedness. The quantile regression parameter estimates the change in a specified quantile of the response variable produced by a 1-unit change in the predictor variable. This allows comparing how some percentiles of the actual traffic may be more affected by forecast volume than other percentiles. This is reflected in the change in the size of the regression coefficient. Table III-G-29 presents the regression statistics (coefficients or and β values and the t value to assess the significance). The highlighted cells are where -1.96 < t-value < 1.96 (i.e., variables that are statistically insignificant at 95% confidence interval). For the median, we observe that the intercept is not significantly different from 0 (zero), but the slope (the forecast volume coefficient) is significantly different from 1 (one), which we can interpret as a detectable bias. Table III-G-29. Quantile regression results (actual count modeled as a function of the forecast volume). 5th Percentile 20th Percentile 50th Percentile 80th Percentile 95th Percentile Pseudo R- Squared 0.433 0.619 0.723 0.750 0.748 Coef. t value Coef. t value Coef. t value Coef. t value Coef. t value Intercept -826.73 -10.55 -434.03 -5.06 37.15 0.54 1395.74 6.59 2940.45 6.50 Forecast Volume 0.62 30.68 0.81 89.56 0.94 148.10 1.05 76.12 1.42 42.26 In addition to detecting bias, these quantile regression models can be applied to obtain an uncertainty window around a forecast, as follows: • 5th Percentile Estimate = -827 + 0.62 * Forecast; • 20th Percentile Estimate = -434 + 0.81 * Forecast; • Median Estimate = 37 + 0.94 * Forecast; • 80th Percentile Estimate = 1396 + 1.05 * Forecast; and • 95th Percentile Estimate = 2940 + 1.42 * Forecast. As an example, if we produce a forecast for 10,000 ADT on a road, we would expect that the median number of vehicles to actually show up on the facility is 9,437 ADT (37 + 0.94 * 10,000), which we can refer to as our median estimate, or alternatively an expected value or adjusted forecast. We would expect (III-G-3)

III-G-40 Traffic Forecasting Accuracy Assessment Research that for 5% of the forecasts we do, the actual traffic will be less than 5,415, and that for 5% of forecasts we do, the actual traffic will be more than 17,153 ADT. The 20th and 80th percentile values can be calculated similarly. Table III-G-30 and Table III-G-31 give the ranges of actual traffic and percent difference from forecast over the forecast traffic volume, respectively. Table III-G-30. Range of actual traffic volume over forecast volume (actual count modeled as a function of the forecast volume). Forecast Forecast Window * 5th Percentile 20th Percentile 50th Percentile 80th Percentile 95th Percentile 0 -827 -434 37 1,396 2,940 5000 2,294 3,612 4,742 6,670 10,047 10000 5,415 7,658 9,448 11,944 17,153 15000 8,536 11,705 14,153 17,218 24,259 20000 11,656 15,751 18,859 22,492 31,365 25000 14,777 19,797 23,564 27,766 38,471 30000 17,898 23,843 28,269 33,040 45,578 35000 21,019 27,890 32,975 38,314 52,684 40000 24,139 31,936 37,680 43,588 59,790 45000 27,260 35,982 42,385 48,862 66,896 50000 30,381 40,028 47,091 54,136 74,002 55000 33,502 44,075 51,796 59,410 81,109 60000 36,622 48,121 56,501 64,684 88,215 * Estimate Table III-G-31. Range of percent difference from forecast as a function of forecast volume (actual count modeled as a function of forecast volume). Forecast Forecast Window: Percent Difference from Forecast 5th Percentile 20th Percentile 50th Percentile 80th Percentile 95th Percentile 0 5000 -54% -28% -5% 33% 101% 10000 -46% -23% -6% 19% 72% 15000 -43% -22% -6% 15% 62% 20000 -42% -21% -6% 12% 57% 25000 -41% -21% -6% 11% 54% 30000 -40% -21% -6% 10% 52% 35000 -40% -20% -6% 9% 51% 40000 -40% -20% -6% 9% 49% 45000 -39% -20% -6% 9% 49% 50000 -39% -20% -6% 8% 48% 55000 -39% -20% -6% 8% 47% 60000 -39% -20% -6% 8% 47% Applying the coefficients as an equation, we constructed ranges of actual traffic and percent difference from forecast for different forecast volumes (see Figure III-G-22).

Appendix G: Large-N Analysis III-G-41 Figure III-G-22. Expected ranges of actual traffic (base model). In Figure III-G-22, the lines depicting various percentile values can be interpreted as the range of actual traffic over a forecast volume. For example, it can be expected that 95% of all projects with the forecast ADT of 30,000 will have an actual traffic count below 45,578. Only 5% of the projects will experience actual traffic less than 17,898. Not considering other variables, this range (45,578 to 17,898 for a forecast volume of 30,000) holds true for 90% of the projects (i.e., there is 90% probability of actual traffic being in this range). 5.2 Inclusive Model for Inference volume on forecast volume and several other descriptive variables: where 1, through , are descriptive variables associated with Project i, and 1 through are estimated model coefficients associated with those descriptive variables. Each is multiplied by ̂ , which makes the effect of that variable scale with the forecast volume (i.e., changing the slope of the line) rather than be additive (i.e., shifting the line up or down). For example, consider a median model where = 0 (zero), = 1, and a single descriptive variable ( 1, ) exists that is a binary flag (i.e., 1, will equal 1 if the forecast is for a new road, and 0 otherwise). If Perfect Forecast 5th Percentile Median 95th Percentile 20th Percentile 80th Percentile 0 10000 20000 30000 40000 50000 60000 0 10000 20000 30000 40000 50000 60000 Ex pe ct ed A DT Forecast ADT (III-G-11) For our second model, we adapted the structure of Equation III-G-4 by regressing the actual

III-G-42 Traffic Forecasting Accuracy Assessment Research 1 has a value of -0.1 then it means that the median actual value would be 10% lower than the forecast. If 1 has a value of +0.1 then it means that the median actual value would be 10% higher than the forecast. The variables chosen in this analysis are given in Table III-G-28. Distributions of forecast inaccuracy as a function of each of these variables are presented in Section 4 of this appendix. For the analysis, the reference class was “forecasts done for a resurfacing project on Interstate or limited access facility using traffic count trend.” The reference project was opened on or later than 2013. Looking at the results of the quantile regression (Table III-G-32), we see that the variables that have the highest bearing on the actual count, apart from the forecast volume, are the opening year and the functional class of the roadway. Positive coefficients signify an increase in the actual count compared to the reference class and negative coefficients signify a decrease. For example, according to our analysis, actual traffic count decreases as the unemployment rate in the opening year increases in value. This is reasonable, since unemployment rate negatively affects the traffic. Again, with an increase in the unemployment rate in the year the forecast was produced, the actual traffic increases. This direct proportionality can be attributed to the assumption of unchanged socioeconomic state between the base year and future forecast year. Statistically significant coefficients on the binary variables allow us to compare the actual count with the reference class. For example, the coefficient for the travel model (Scale_FM_TravelModel in Table II-G-32) is 0.02 for the 80th percentile against the reference class of traffic count trend. This means that using the travel demand model shifts the 80th percentile estimate up by 1.7% times the forecast traffic, compared to the traffic count trend. Accurate forecasts would have intercepts of 0 (zero) and slopes of 1 (one). Therefore, coefficients that shift the slopes closer to 1 are associated with better forecasts. For the median forecasts, this is a measure of the degree of bias, but for the outlying percentiles it is a measure of the spread in the forecasts. In general, variables with positive coefficients in the 5th percentile model and negative coefficients in the 95th percentile model will be associated with more precise forecasts (a narrower uncertainty window), although it must be considered how they interact with the other terms in the models. An interesting observation from Table III-G-32 is how much the actual traffic compares based on the opening year. For example, 95% of the projects that opened between 1991 and 2002 have seen at least 31.2% more traffic compared to those opened after 2012. Similarly, arterials, collectors and local roads have less traffic compared to Interstates if other variables remain the same. Figure III-G-23 plots the actual traffic versus the forecast traffic for the 80th percentile using the coefficients in Table III-G-32. Interpretation of the graph is that 80% of the projects on arterials or interstate have actual traffic that falls below their respective lines.

Appendix G: Large-N Analysis III-G-43 5th Percentile 20th Percentile 50th Percentile 80th Percentile 95th Percentile Pseudo R-Squared 0.513 0.662 0.762 0.827 0.853 Coef. t value Coef. t value Coef. t value Coef. t value Coef. t value (Intercept) -75.44 -0.59 145.11 2.70 331.44 11.70 535.37 6.89 1616.95 6.90 Adjusted Forecast 0.80 8.49 0.71 14.84 1.05 24.26 1.08 19.47 0.92 12.45 Forecast_over30k 0.04 1.45 0.06 3.42 -0.01 -0.34 -0.17 -7.37 -0.26 -6.25 Unemployment Rate Opening Year -0.03 -2.89 -0.01 -2.42 -0.03 -5.88 -0.01 -2.48 0.00 -0.03 Unemployment Rate Year Produced 0.00 0.84 0.01 4.70 0.01 3.89 0.01 2.51 0.02 18.04 Scale_YP_Missing 0.03 0.64 0.10 3.31 0.06 2.55 0.02 0.14 0.34 7.35 Scale_DiffYear 0.00 -0.96 0.00 1.41 0.01 5.65 0.01 10.05 0.02 10.71 Scale_IT_AddCapacity 0.01 0.37 0.02 0.99 0.04 2.54 0.04 1.40 0.13 5.13 Scale_IT_NewRoad 0.05 2.43 0.04 3.51 0.03 1.80 -0.01 -0.74 -0.02 -3.56 Scale_IT_Unknown -0.07 -1.64 0.01 0.51 0.06 4.28 0.13 6.23 0.14 6.35 Scale_FM_TravelModel 0.04 2.19 0.07 2.60 0.00 0.06 0.02 1.13 0.03 1.01 Scale_FM_Unknown -0.05 -1.39 -0.04 -1.35 -0.03 -1.40 -0.02 -0.50 0.02 0.35 Scale_FA_Consultant -0.02 -0.55 0.02 0.46 0.02 0.75 0.02 0.64 0.04 0.85 Scale_Agency_BCF 0.01 0.20 -0.05 -1.75 -0.13 -5.85 -0.16 -4.03 -0.11 -1.84 Scale_Europe_AD 0.11 2.55 0.05 1.76 0.04 1.91 0.01 0.28 -0.01 -0.28 Scale_OY_1960_1990 0.01 0.18 -0.09 -2.62 -0.05 -1.36 -0.02 -0.55 0.02 0.83 Scale_OY_1991_2002 0.31 9.88 0.25 8.63 0.27 10.86 0.39 9.93 0.48 10.73 Scale_OY_2003_2008 0.12 3.88 0.12 4.73 0.05 2.72 0.07 4.79 0.12 5.42 Scale_OY_2009_2012 0.21 8.23 0.09 3.44 0.10 4.45 0.07 3.42 0.06 2.60 Scale_FC_Arterial -0.16 -10.38 -0.08 -4.79 -0.08 -5.13 -0.09 -5.00 -0.06 -1.73 Scale_FC_CollectorLocal -0.34 -3.65 -0.13 -3.63 -0.14 -5.03 -0.24 -12.46 -0.33 -2.19 Scale_FC_Unknown -0.15 -2.91 -0.08 -3.08 -0.14 -5.34 -0.16 -4.41 -0.13 -2.60 Table III-G-32. Quantile regression results [inclusive model].

III-G-44 Traffic Forecasting Accuracy Assessment Research Figure III-G-23. Comparison of actual traffic for arterials and Interstate for 80th percentile using inclusive model. While these models are useful for understanding which factors may bias forecasts and which factors may be associated with broader or narrower uncertainty windows, they are not useful at the time the forecast is made because not all variables are known. For example, while it is interesting to know how the unemployment rate in the opening year affects forecast accuracy, that information will obviously not be known until the project actually opens. Therefore, we estimated another, more limited set of models that can be applied at the time of forecasting. These models are presented in the next section. 5.3 Forecasting Model The uncertainty inherent in forecasting traffic is hard to get rid of, even with advances in modeling procedures. Hartgen (2013) suggests “Convert forecasts from single point-based estimates to range-based with probability of outcome.” For risk assessment of a project, this range of forecasts also comes in handy. The goal of this analysis is to ascertain that limit (i.e., How much should we expect the actual traffic to vary against the forecast volume for a specific type of project, roadway, etc.?). Again considering the forecast accuracy database to be representative of the national average, we can create a confidence interval for traffic as a function of several variables, similar to the analysis shown in Section 3. We employ the same quantile regression as above, but for the descriptive variables, we chose the ones that would be known at the time of producing the forecasts: 1. Forecast traffic, 2. Unemployment rate in the current year, 3. Years before 2010, a control variable to account for forecasts getting better over the years, or after 2010, - 10,000 20,000 30,000 40,000 50,000 60,000 70,000 0 10000 20000 30000 40000 50000 60000 70000 A ct ua l T ra ffi c Forecast Traffic Arterial Interstate

Appendix G: Large-N Analysis III-G-45 4. Forecast horizon, or how many years into the future traffic is being forecast, 5. Improvement type, or if the project is on an existing road or constructing a new one with the earlier one as reference class, 6. Forecast method with traffic count trend as reference class, and 7. Functional Class of the roadway, interstate being the reference. The results of the quantile regression for 5th, 20th, 50th, 80th and 95th percentiles are given in Table III-G-33. The highlighted cells are where -1.96<t-value<1.96 i.e. variables which are statistically insignificant at 95% confidence interval. Interpretation of the coefficients is simple: positive coefficients mean positive slope on actual traffic and the opposite for negative coefficients. For the binary variables, the positive coefficients also mean increase in actual traffic compared to reference class. Table III-G-33. Quantile regression results (forecasting model). 5th Percentile 20th Percentile 50th Percentile 80th Percentile 95th Percentile Pseudo R-Squared 0.475 0.631 0.739 0.804 0.830 Coef. t value Coef. t value Coef. t value Coef. t value Coef. t value (Intercept) -182.267 -1.769 154.578 3.082 255.551 4.667 287.909 3.943 976.786 4.787 Adjusted Forecast 0.705 15.972 0.732 36.186 0.891 45.198 1.027 44.195 1.254 23.880 Adjusted Forecast _over30k 0.024 0.568 0.057 3.053 -0.004 -0.219 -0.190 -8.296 -0.413 -9.887 Scale_Unemployment Rate_YearProduced -0.006 -1.411 0.005 2.770 0.002 0.871 0.007 2.762 0.010 1.865 Scale_YearForecast Produced_before2010 -0.007 -5.639 -0.005 -5.225 0.0002 0.270 0.004 3.913 0.003 2.359 Scale_DiffYear 0.006 2.809 0.009 6.682 0.008 5.620 0.014 8.234 0.020 10.501 Scale_IT_NewRoad 0.093 4.336 0.009 1.096 -0.008 -0.901 -0.036 -1.932 -0.090 -4.288 Scale_FM_TravelModel 0.068 3.307 0.014 1.631 -0.008 -0.516 -0.018 -1.252 -0.101 -7.356 Scale_FC_Arterial -0.150 -5.237 -0.061 -4.855 -0.062 -5.171 -0.084 -5.964 -0.116 -5.881 Scale_FC_CollectorLocal -0.212 -4.027 -0.111 -4.794 -0.126 -5.212 -0.201 -5.780 -0.321 -2.362 We can apply these coefficients in the form of an equation to create a forecast window for actual traffic as a function of the descriptive variables. A simple example is demonstrated here for a project with the following specification: • The forecast was produced in the year 2018, • The unemployment rate (at state level) in 2017 was 4%, • The forecast estimates the traffic for 2020 (i.e., a forecast horizon of 2 years), • The project is a new construction project on a minor arterial, and • The forecast is done using a travel demand model. The contributions of the specific values of the variables to the equation are shown in Table III-G-34 and the forecast window for actual traffic for different forecast volume is shown in Table III-G-35.

III-G-46 Traffic Forecasting Accuracy Assessment Research Table III-G-34. Contributions of specific values to the equation. Variables Values 5th Percentile 20th Percentile 50th Percentile 80th Percentile 95th Percentile (Intercept) -182.27 154.58 255.55 287.91 976.79 Adjusted Forecast 0.70 0.73 0.89 1.03 1.25 Adjusted Forecast _over30k - - - - - - Scale_Unemployment Rate_YearProduced 4 -0.02 0.02 0.01 0.03 0.04 Scale_YearForecast Produced_before2010 - - - - - - Scale_DiffYear 2 0.01 0.02 0.02 0.03 0.04 Scale_IT_NewRoad 1 0.09 0.01 -0.01 -0.04 -0.09 Scale_FM_TravelModel 1 0.07 0.01 -0.01 -0.02 -0.10 Scale_FC_Arterial - - - - - - Scale_FC_CollectorLocal - - - - - - Table III-G-35. Forecast window for forecast model on specified values. Forecast Volume 5th Percentile 20th Percentile 50th Percentile 80th Percentile 95th Percentile 0 -182 155 256 288 977 5000 4,087 4,116 4,740 5,429 6,687 10000 8,357 8,078 9,225 10,571 12,398 15000 12,626 12,040 13,709 15,712 18,109 20000 16,896 16,001 18,194 20,854 23,819 25000 21,165 19,963 22,678 25,995 29,530 30000 25,435 23,924 27,163 31,137 35,240 35000 29,823 28,173 31,626 35,327 38,885 40000 34,211 32,421 36,090 39,518 42,530 45000 38,599 36,670 40,553 43,709 46,175 50000 42,988 40,918 45,017 47,899 49,819 55000 47,376 45,166 49,480 52,090 53,464 60000 51,764 49,415 53,944 56,281 57,109

Appendix G: Large-N Analysis III-G-47 Table III-G-36. Percent difference from forecast window for forecast model on specified values. Forecast Volume 5th Percentile 20th Percentile 50th Percentile 80th Percentile 95th Percentile 0 5000 -18% -18% -5% 9% 34% 10000 -16% -19% -8% 6% 24% 15000 -16% -20% -9% 5% 21% 20000 -16% -20% -9% 4% 19% 25000 -15% -20% -9% 4% 18% 30000 -15% -20% -9% 4% 17% 35000 -15% -20% -10% 1% 11% 40000 -14% -19% -10% -1% 6% 45000 -14% -19% -10% -3% 3% 50000 -14% -18% -10% -4% 0% 55000 -14% -18% -10% -5% -3% 60000 -14% -18% -10% -6% -5% Let’s consider for the specifications described above, the traffic volume forecast for 2020 is 45,000. From Table III-G-35 we get the 20th and 80th percentile values to be 36,670 and 43,709. These values mean that the probability of actual traffic being within these two values is 60%. The forecast window is graphically presented in Figure III-G-24. Figure III-G-24. Range of actual traffic as a function of forecast traffic. Perfect Forecast 20th Percen le Median or 50th Percen le 95th Percen le 5th Percen le 80th Percen le 0 10000 20000 30000 40000 50000 60000 0 10000 20000 30000 40000 50000 60000 Ex pe ct ed A DT Forecast ADT

III-G-48 Traffic Forecasting Accuracy Assessment Research References Australian Government (2012). “Addressing Issues in Patronage Forecasting for PPP/Toll Roads.” Department of Infrastructure, Regional Development and Cities, Canberra, Australia. https://infrastructure.gov.au/ infrastructure/public_consultations/files/patronage_consultation_paper.pdf. Bain, R (2009). “Error and Optimism Bias in Toll Road Traffic Forecasts.” Transportation 36 (5): 469–82. https://doi.org/10.1007/s11116-009-9199-7. Bain, R (2011). “The Reasonableness of Traffic Forecasts Findings from a Small Survey.” Traffic Engineering and Control (TEC) Magazine, May 2011. Bain, R. (2013). “Toll Roads: Big Trouble Down Under.” Infrastructure Journal, January 17, 2013. Bain, R., and J. W. Plantagie (2004). “Traffic Forecasting Risk: Study Update 2004.” http://www.people.hbs.edu/besty/projfinportal/s&p_traffic_risk_2004.pdf. Bain, R., and L. Polakovic (2005). “Traffic Forecasting Risk Study Update 2005: Through Ramp-up and Beyond.” Standard & Poor’s, London. http://toolkit.pppinindia.com/ pdf/standard-poors.pdf. Buck, K., and M. Sillence (2014). “A Review of the Accuracy of Wisconsin’s Traffic Forecasting Tools.” In . https://trid.trb.org/view/2014/C/1287942. Cade, B. S., and B. R. Noon (2003). “A Gentle Introduction to Quantile Regression for Ecologists.” Frontiers in Ecology and the Environment 1 (8): 412–20. https://doi.org/10.1890/1540- 9295(2003)001[0412:AGITQR]2.0.CO;2. CDM Smith, A. Horowitz, T. Creasey, R. Pendyala, and M. Chen. (2014) NCHRP Report 765: Analytical Travel Forecasting Approaches for Project-Level Planning and Design. Transportation Research Board of the National Academies, Washington, D.C. Flyvbjerg, B., M. K. S. Holm, and S. L. Buhl (2005). “How (In)Accurate Are Demand Forecasts in Public Works Projects?: The Case of Transportation.” Journal of the American Planning Association 71 (2). https://trid.trb.org/view.aspx?id=755586. Flyvbjerg, B., M. K. S. Holm, and S. L. Buhl (2006). “Inaccuracy in Traffic Forecasts.” Transport Reviews 26 (1). https://trid.trb.org/view/2006/C/781962. Flyvbjerg, B. (2005). “Measuring Inaccuracy in Travel Demand Forecasting: Methodological Considerations Regarding Ramp up and Sampling.” Transportation Research Part A: Policy and Practice 39 (6): 522–30. https://doi.org/10.1016/j.tra.2005.02.003. Giaimo, G., and M. Byram (2013). “Improving Project Level Traffic Forecasts by Attacking the Problem from All Sides,” presented at The 14th Transportation Planning Applications Conference, Columbus, OH. Gomez, J., J. M. Vassallo, and I. Herraiz (2016). “Explaining Light Vehicle Demand Evolution in Interurban Toll Roads: A Dynamic Panel Data Analysis in Spain.” Transportation, 43(4): 677–703. https://doi.org/10.1007/s11116-015-9612-3. Hartgen, D. T. (2013). “Hubris or Humility? Accuracy Issues for the next 50 Years of Travel Demand Modeling.” Transportation, 40 (6): 1133–57. https://doi.org/10.1007/s11116-013-9497-y. Kriger, D., S. Shiu, and S. Naylor (2006). NCHRP Synthesis 364: Estimating Toll Road Demand and Revenue. Transportation Research Board of the National Academies, Washington, D.C. https://trid.trb.org/view/2006/M/805554. Li, Z., and D. A. Hensher (2010). “Toll Roads in Australia: An Overview of Characteristics and Accuracy of Demand Forecasts.” Transport Reviews, 30(5): 541–69. https://doi.org/10.1080/01441640903211173. Marlin Engineering (2015). “Traffic Forecasting Sensitivity Analysis.” TWO#13. Miller, J. S., S. Anam, J. W. Amanin, and R. A. Matteo (2016). “A Retrospective Evaluation of Traffic Forecasting Techniques.” FHWA/VTRC 17-R1. Virginia Transportation Research Council. http://ntl.bts.gov/lib/37000/37800/37804/10-r24.pdf.

Appendix G: Large-N Analysis III-G-49 Nicolaisen, M. S., and P. Næss (2015). “Roads to Nowhere: The Accuracy of Travel Demand Forecasts for Do- Nothing Alternatives.” Transport Policy, 37(0). https://trid.trb.org/view/2015/C/1334458. Odeck, J., and M. Welde (2017). “The Accuracy of Toll Road Traffic Forecasts: An Econometric Evaluation.” Transportation Research Part A: Policy and Practice, 101 (July): 73–85. https://doi.org/10.1016/j.tra.2017.05.001. Parthasarathi, P., and D. Levinson (2010). “Post-Construction Evaluation of Traffic Forecast Accuracy.” Transport Policy, 17(6): 428–43. https://doi.org/10.1016/j.tranpol.2010.04.010. Pedersen, N. J., and D. R. Samdahl (1982). NCHRP Report 255: Highway Traffic Data for Urbanized Area Project Planning and Design. TRB, National Research Council, Washington, D.C. Pereira, F. C., C. Antoniou, J. A. Fargas, and M. Ben-Akiva (2014). “A Metamodel for Estimating Error Bounds in Real-Time Traffic Prediction Systems.” IEEE Transactions on Intelligent Transportation Systems, 15(3): 1310– 22. https://doi.org/10.1109/TITS.2014.2300103. Transportation Research Board (2000). Highway Capacity Manual [Book and CD-Rom edition] TRB, National Research Council, Washington, D.C. Tsai, C.-H. P., C. Mulley, and G. Clifton (2014). “Forecasting Public Transport Demand for the Sydney Greater Metropolitan Area: A Comparison of Univariate and Multivariate Methods.” Road & Transport Research: A Journal of Australian and New Zealand Research and Practice, 23(1): 51. Vassallo, J. M, and M. Baeza (2007). “Why Traffic Forecasts in PPP Contracts Are Often Overestimated?” EIB University Research Sponsorship Programme. Welde, M., and J. Odeck (2011). “Do Planners Get It Right? The Accuracy of Travel Demand Forecasting in Norway.” EJTIR, 1(11): 80–95. Zhang, Xu, and Mei Chen. 2019. “Quantifying the Impact of Weather Events on Travel Time and Reliability.” Journal of Advanced Transportation. https://doi.org/10.1155/2019/8203081.