5
Shortcomings of Current Forecasting Processes

The four-step (or, in some cases, three-step) trip-based modeling process used by the vast majority of metropolitan planning organizations (MPOs) has evolved over a period of about 50 years. Originally conceived as an aid to developing transportation networks for large cites, the process was widely adopted to support planning for the urban segments of the Interstate highway system and to support the metropolitan planning requirements of the Federal-Aid Highway Act of 1962. Over the years, the procedures employed have been modified to address other planning questions and issues (e.g., air quality, transportation operations, Transit New Starts). While many projects have been planned and justified on the basis of data produced from models of this type, it has long been recognized that the process has many shortcomings.

Models used to forecast travel are critical in estimating likely impacts of investment and policy decisions, with the understanding that socioeconomic conditions over the forecast period may change in ways that cannot be predicted. Estimates of differences among alternatives may reasonably be regarded as more precise and reliable than overall forecasts since alternatives are likely to be equally affected by global changes. Travel forecasting introduces a reason-based rigor into the planning process that would otherwise be lacking. Given the inherent uncertainty in knowing the future, it is imperative that forecasting models themselves not introduce undue additional uncertainty.

Travel forecasting as practiced by MPOs is a type of systems analysis. It requires a set of environmental system inputs (small-area socioeconomic projections), specified alternative strategies to be evaluated (capital investments in new facilities or operational policies), models that describe relationships between the data inputs and strategies (the four-step travel forecasting



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 65
5 Shortcomings of Current Forecasting Processes T he four-step (or, in some cases, three-step) trip-based modeling process used by the vast majority of metropolitan planning organizations (MPOs) has evolved over a period of about 50 years. Originally conceived as an aid to developing transportation networks for large cites, the process was widely adopted to support planning for the urban segments of the Interstate highway system and to support the metropolitan planning requirements of the Federal-Aid Highway Act of 1962. Over the years, the procedures employed have been modified to address other planning questions and issues (e.g., air quality, transportation operations, Transit New Starts). While many projects have been planned and justified on the basis of data produced from models of this type, it has long been recognized that the process has many shortcomings. Models used to forecast travel are critical in estimating likely impacts of investment and policy decisions, with the understanding that socioeconomic conditions over the forecast period may change in ways that cannot be pre- dicted. Estimates of differences among alternatives may reasonably be regarded as more precise and reliable than overall forecasts since alternatives are likely to be equally affected by global changes. Travel forecasting introduces a reason- based rigor into the planning process that would otherwise be lacking. Given the inherent uncertainty in knowing the future, it is imperative that forecast- ing models themselves not introduce undue additional uncertainty. Travel forecasting as practiced by MPOs is a type of systems analysis. It requires a set of environmental system inputs (small-area socioeconomic projections), specified alternative strategies to be evaluated (capital invest- ments in new facilities or operational policies), models that describe relation- ships between the data inputs and strategies (the four-step travel forecasting 65

OCR for page 65
METROPOLITAN TRAVEL FORECASTING Current Practice and Future Direction 66 models), and estimated consequences of each alternative strategy (such as forecasts of traffic, ridership, and travel times). Modeled outputs from an iteration of the process aid in redesigning alternatives to be examined in suc- ceeding iterations. While this analytic forecasting process is logically and intuitively appealing, it has limitations and shortcomings. Critiques of the four-step process, of its ability to address the issues with which MPOs must deal, and of the forecasts obtained using the process are numerous: • According to a report of the Transportation Research Board (TRB), “the state of knowledge and modeling practice are not adequate for predict- ing with certainty the impacts of highway capacity additions. In particular, the models are not well suited to the types of analyses and levels of precision called for by the conformity regulations. They were developed to address dif- ferent questions and cannot be readily adapted to the task at hand” (TRB 1995, 224). • A report of the National Cooperative Highway Research Program reviews the current state of the art for analyzing transportation control measures and concludes that “serious reservations exist concerning the accuracy of these results, the robustness of the underlying data, and whether the correct set of variables are captured in the model systems.” The report recommends a new modeling framework consisting of the following modules: disaggregate and activity-based demand, household sample enumeration, incremental analysis, traffic microsimulation, and household travel survey data with stated prefer- ence data to support policy analysis (Cambridge Systematics 2001). • Another TRB report suggests that “the available models are not suited to estimating the emissions effects of small projects or linking these effects with air quality” (TRB 2002). • Meyer and Miller (2001) state: “While UTMS [the Urban Transpor- tation Modeling System] has been employed . . . for almost 40 years, it has also been seriously criticized from many points of view for almost the same length of time. Most fundamentally, UTMS is not behavioral in nature; that is, it is not based . . . on a coherent theory of travel behavior.” They suggest further that “the trip based approach to travel demand modeling is not well suited to representing . . . traveler responses to the complex range of policies typically of interest to today’s planners (pricing, HOV and carpooling options, telecommuting, other [transportation control] measures, etc.).” The following discussion of the shortcomings of current modeling prac- tice is presented with the understanding that MPOs must use the best tools

OCR for page 65
Shortcomings of Current Forecasting Processes 67 available to them in doing their work. Newer, advanced modeling tools may be available but beyond the resources of the agency or not yet proven in prac- tice. This having been said, the weaknesses of current practice can be catego- rized as follows: (a) inherent weaknesses of the models, (b) errors introduced by modeling practice, (c) lack or questionable reliability of data, and (d) biases arising from the institutional climate in which the models are used. INHERENT WEAKNESSES OF CURRENT MODELS In general, as the detail required to address transportation issues increases, the complexity of appropriate analysis techniques must also increase (see Figure 4-3 in Chapter 4). The current four-step travel demand forecasting models are not well suited to applications that require the portrayal or analysis of detailed travel markets, decisions of individuals, effects of value of time and value of reliability, continuous time-of-day variations in travel, and goods movement. In particular, the current widely used four-step metropolitan travel demand forecasting process cannot adequately characterize the following (without the use of off-model adjustments): • Road pricing; • Time-specific policies, such as parking, work schedules, and scheduling of truck deliveries; • Hourly speeds or traffic volumes; • Improvements in traffic operations; • Improvements or policies addressing freight movement; • Nonmotorized travel; • Peak spreading and highly congested networks; or • Goods movement. The inherent weaknesses of current models are discussed in more detail below. Inability to Represent Individual Decisions The aggregate models in general use today are limited by an inability to repre- sent the detailed decision patterns of individuals or households easily. The con- ventional four-step trip-based models rarely attempt to associate traveler characteristics with trips being made. In some cases, market segmentation is used to incorporate information about the household characteristics of travelers—

OCR for page 65
METROPOLITAN TRAVEL FORECASTING Current Practice and Future Direction 68 typically income—throughout the steps of the modeling process. In theory, market segmentation could be used to account for other household or trav- eler attributes, but doing so is difficult in practice. A larger problem is the failure of the conventional models to consider the full range of choices avail- able to individuals. In conventional models, the available choices are typically to make or not make a trip (trip generation), the destination visited (trip dis- tribution), the mode used (mode choice), and the path taken (assignment). In reality, travelers have other choices, including making a trip at a different time or on a different day, incorporating a trip to fill one need into a trip to fill other needs, having a trip made by another member of the household or trip-making unit, or substituting communication for travel. Lack of Sensitivity to Current Issues Models can address only questions to which they are sensitive. If a quantity is not an independent variable included in the model, the model cannot be used to answer questions about the impact of a change in that variable on travel demand. Two examples illustrate this point—road pricing and goods movement. Road Pricing The summary of a 2005 Expert Forum on Road Pricing and Travel Demand Modeling notes that “the four-step modeling system does not capture behav- ioral responses to pricing options because pricing has dynamic, interactive effects that cannot be accommodated in a linear, static modeling system” (Schofer 2006, 10). A paper prepared for the forum identifies important modeling challenges: • Accounting for reliability, • Accounting for heterogeneity among users and their values of time, and • Dealing with time-of-day variations and peak spreading. These challenges can be addressed to some extent for a fixed-toll facility using a well-calibrated four-step modeling process, supplemented by local surveys and off-model adjustments. But “representing the full spectrum of pricing out- comes will require a shift to more advanced tools,” such as activity- or tour- based models, microsimulation, and dynamic traffic assignment (Vovsha et al. 2005).

OCR for page 65
Shortcomings of Current Forecasting Processes 69 Considerable evidence reveals the shortcomings of travel forecasting models for predicting the performance of new toll facilities. The process for bond financing of new toll roads includes review and evaluation of proposed financial plans by a bond-rating agency. At the heart of financial planning is an “investment grade forecast” of the traffic and revenues the toll road will attract upon opening. In recent years, underperformance of new toll roads and consequent risk to investors have caused bond-rating agencies to take a hard look at these forecasts. Standard and Poor’s (S&P) has assembled a database of 104 international toll road, bridge, and tunnel case forecasts and actual experience of traffic and revenues. Analysis of this database shows what S&P calls “systematic optimism bias.” For all case studies, toll road forecasts overestimated actual first-year traf- fic by an average of 20 to 30 percent. This situation does not improve for the second through fifth years after opening; the overestimates for these years are similar to those for the first year. If the database is arrayed as a ratio of actual to forecast traffic, the population is normally distributed in a bell-shaped curve, but the mean rests well below 1.0 at 0.77, underscoring the tendency toward optimism bias. S&P also found that truck forecasts were considerably more variable than those for total traffic; for the ratio of actual to forecast traffic, the standard deviation for trucks was 0.33, compared with 0.26 for total traffic. The dif- ferential probably reflects the more primitive state of freight forecasting mod- els. The variability in truck forecasts has consequences for toll roads, where trucks account for a larger share of revenues than other traffic (S&P 2005). Another rating agency, FitchRatings, has also studied the toll road fore- casting issue. While noting examples of start-up toll roads that have exceeded forecasts (e.g., 407, Toronto, Canada; Chesapeake Expressway, Virginia; Mid- Bay Bridge, Florida), FitchRatings cites many more projects for which traffic and revenues have been significantly below forecasts (e.g., Dulles Greenway, Virginia; E-470, Colorado; Foothill Eastern, California; Osceola Parkway, Florida; Pocahontas Parkway, Virginia; San Joaquin Hills, California; Garcon Point Bridge, Florida; Sawgrass Expressway, Florida; Southern Connector, South Carolina). The skew toward overestimated forecasts suggests optimism bias (see the discussion below of biases arising from the institutional climate), but Fitch- Ratings also points to “the use of regional travel demand models intended for other planning purposes and not necessarily appropriate for use to support the issuance of toll road debt” (FitchRatings 2003, 2).

OCR for page 65
METROPOLITAN TRAVEL FORECASTING Current Practice and Future Direction 70 Modeling challenges become considerably more complex for projects for which tolls charged vary by time of day. Several metropolitan areas are con- sidering managed-lane projects in which the price for traveling on a facility could vary dynamically on the basis of usage of the facility. Implementing this approach could require forecasting demand and revenue for an existing free- way segment that is to be reconstructed, expanded, and subsequently operated as a toll road. As noted by Spear, however, “Virtually all of the road pricing models implemented to date have been used to analyze the travel demand and revenue impacts of static tolls (i.e., toll charges that remain constant over a fixed time period). Current four-step travel demand models cannot easily ana- lyze the impacts of variable tolls (i.e., toll charges that are adjusted within a peak period to discourage overuse of the facility to maintain acceptable levels of service), because they do not specifically consider the temporal build-up and dispersal of traffic during peak period” (Spear 2006, 19). Goods Movement Freight has emerged as a major issue in the transportation community. High- ways, railroads, and ports are running out of capacity to accommodate projected increases in the volume of goods to be moved. In an economy organized around fast and reliable delivery of goods, congestion becomes an important variable in the cost of business and in economic development (FHWA 2006). Regional transportation plans and project analyses must address goods movement. Doing so is important not only from the perspective of mobility but also from an environmental and roadway design point of view. Given the nature of their fuel, size, and cargo, trucks are a source of significant nitrogen oxide and particulate emissions. Trucks also have a disproportionate impact on the road infrastructure. As goods movement becomes an increasingly important concern for many regions, the lack of validated models of goods movement and truck activity is receiving greater notice. The recently instituted Freight Model Improvement Program is a partial response to this need (FHWA 2006). The number of truck trips (including commercial vehicles and trucks of all sizes) and resultant vehicle miles traveled (VMT) are growing at a rate more than twice that of trips made by personal vehicles in some areas. As conges- tion increases, the delivery of goods and services by truck throughout a metropolitan area is becoming more difficult and less reliable. This situa- tion leads in turn to concerns regarding the economic vitality of businesses within an area.

OCR for page 65
Shortcomings of Current Forecasting Processes 71 The information and tools available to address goods movement, how- ever, are severely limited. Truck count data, information on distribution pat- terns, and trip chain profiles are but a few areas in which the analyst faces data shortages. Characteristics of goods movement can vary by commodity, pay- load, time of day, and truck type. Furthermore, little is known about how businesses make decisions on freight logistics. Without a better understand- ing of freight activity and models based on data that reflect real-world logis- tics and distribution systems, planners cannot begin to assess, for example, how the performance of the transportation system would change if truck deliv- eries were limited to off-peak delivery times. Failure to Deal with Uncertainty in Model Estimates Most travel forecasting models produce a single answer, although the model is estimated, calibrated, and validated on the basis of data sets that are sub- ject to many sources of error and uncertainty. The data used are based on sampling and include sampling errors, as well as other types of errors due to survey methodology. Errors also are made, for example, when data are aggre- gated and entered into databases. The models themselves may suffer from misspecification. When models are used for prediction, additional errors are necessarily introduced because the values of parameters in the future are always estimates and thus subject to error. Some degree of error is unavoidable. Within reason, moreover, the pres- ence of errors does not prevent effective applications. It is necessary and appropriate, however, to develop sampling and modeling strategies that are informed by the patterns in which errors occur and especially by understand- ing of the ways in which errors are propagated through sequences of models. Errors should be discussed in the course of normal practice; their influence understood and disclosed; and proper account taken of the variation that nec- essarily occurs in the use of models for forecasting purposes, particularly when forecasts are used to evaluate alternatives that differ only modestly or to produce point estimates of travel to meet regulatory requirements. As noted, even though it is highly unlikely that all of the factors input to forecast travel demand will occur as projected, travel demand forecasts are typically presented as a single value (e.g., transit boardings, traffic volumes). Methodologies have been developed and in a few cases applied to associate a probable variance with each input factor and produce an expected error range

OCR for page 65
METROPOLITAN TRAVEL FORECASTING Current Practice and Future Direction 72 for the final forecasts. Presenting model results with an estimate of error allows users either to derive a point estimate (midpoint of the confidence interval) or to use a range estimate (defined by the confidence limits). In either event, users will be more knowledgeable about the output of the model. Inability to Represent Dynamic Conditions The conventional travel demand models make use of networks, both highway and transit, in which impedances are averages over an extended period, do not reflect any uncertainty or unreliability, and are not representative of the con- ditions that would be expected or found by an individual traveler at the time a trip choice is made. Agencies are being asked to evaluate road pricing schemes in which tolls can vary rapidly over the course of a few minutes on the basis of levels of congestion. The regional travel demand models in use today can treat such variation only in an aggregate estimate, although some studies have used detailed simulation procedures to augment the forecasts derived from these models. One barrier to including reliability as a variable in road pricing models is that traditional four-step travel demand models are designed structurally to work with average or mean values (e.g., average daily or average peak period travel volume) and not the variation about those mean values. Recent progress in the development and deployment of simulation techniques in traffic modeling suggests considerable promise for addressing variability in traffic congestion, but a much better understanding of the factors that influ- ence traffic variability is needed as well. Moreover, as Spear notes: “Despite the potential importance of (travel time) reliability in road pricing (especially as a congestion mitigation strategy), there are few, if any, examples of oper- ational travel demand models that explicitly include reliability as a variable” (Spear 2006, 19). ERRORS INTRODUCED BY MODELING PRACTICE Inadequate Validation Practices A primary concern is the lack of sufficient data for proper validation of mod- els after estimation of model parameters. The cost and difficulty of collect- ing data on both household characteristics and trip patterns limit the ability

OCR for page 65
Shortcomings of Current Forecasting Processes 73 of model developers, MPOs, and others to validate an estimated or calibrated model. The size of household survey data sets is a particular issue. A data set may be of sufficient size and stratification to be used to identify proper func- tional forms and to estimate key parameters of most travel models, but the same data set often may not provide sufficient information for validation of geographic patterns beyond a rather gross level. This is particularly true for the trip distribution element of a four-step model. The U.S. census provides some independent information about the distribution pattern of work travel, but other than results of household travel surveys, there are no data against which nonwork trip distributions can be validated. [In the future, these travel data will be obtained in the annual American Community Survey (ACS) rather than the decennial census.] Trip distribution modeling would benefit from new, more advanced pro- cedures and more extensive data for model development and validation. Current gravity-type trip distribution models used by MPOs can often be flawed because of poor model calibration and application. Even if the data collected in a household survey are considered adequate for validating the base-year application of a model, similar data are not avail- able for validation as the model ages. As a result, validation may be based almost solely on the ability of the assigned volumes—the final step of the modeling— to accord with traffic counts or VMT. Even if there are sufficient counts to support valid comparison with assigned volumes, the counts provide no information about vehicle occupancy or trip generation, distribution, purpose, and length. Analysts have little quantitative guidance for making any needed adjustments to the model set. Too often the later steps in a modeling chain (e.g., mode choice, assignment) are manipulated in an attempt to correct for errors in earlier stages. As a result, the mode-choice stage of a sequential four- step model may be misestimated because it is attempting to correct for error in the generation and distribution models. Rodier (2004) evaluated the official travel model of the Sacramento region for model error by running the 1991 model for 2000 with data from the actual 2000 observed travel survey, along with demographic and economic (employment) data, as inputs. In such a test, input error is eliminated, and only model error remains. Rodier found that trip generation was under- projected by 6 percent and VMT was overestimated by 6 percent as compared with actual counts. This test thus finds primarily model specification and model calibration error. The author also tested the accuracy of socioeconomic/ land use projections made in 1991 for 2000. This test showed that trip gen-

OCR for page 65
METROPOLITAN TRAVEL FORECASTING Current Practice and Future Direction 74 eration was 2 percent higher and VMT 12 percent higher than counts and actual 2000 survey data. This type of test finds specification, calibration, and input errors, all acting together. The household and employment projections made in 1991 turned out to be 8 percent and 9 percent higher, respectively, than actual figures for the whole region and so were a major source of error. In both of Rodier’s tests, errors were much higher for trip generation for the home–shopping trip purpose, and mode shares were the most incongruent for 3+ shared-ride trips and especially for walking trips. This is one of the most useful papers to date on modeling error using both historical forecast- ing and sensitivity tests. Very few MPOs conduct such exercises, but all MPOs should do so as part of model validation. Failure to Maintain Consistency Among All Elements of a Forecast The effects of a lack of consistency among the various elements of the mod- eling chain have often been overlooked. In some cases, this neglect has been due to a limitation of the model application software; in other cases, those developing or applying the model set are unaware of the potential problems. Scrutiny of forecasts made for Transit New Starts projects has demonstrated that a lack of consistency in generalized cost relationships (e.g., time, dis- tance, tolls) among various elements of a model can lead to counterintuitive and likely incorrect results (AECOM Consult 2005). There may be a disconnection between land use/growth forecasts and transportation plans. This disconnection relates to both the location and nature of the growth. Over the years, many MPOs have investigated the use of sys- tematic procedures for forecasting the location of growth in households and employment. Some have implemented and are using formal land use models that account not only for attributes of the transportation system but also for other factors that are expected to affect location decisions. In many other agen- cies, however, growth projections are formulated by the component jurisdic- tions without regard for expected transportation system improvements or congestion. Reports of allocation of “forecasts by negotiation” are common. Many agencies have begun to include in their model sets factors intended to reflect the influence of subarea development patterns, including density, activity mix, and design, on trip generation, distribution, and mode share. Given the small sample sizes of household surveys, most of these procedures are based on limited data. The impact of development patterns on travel is

OCR for page 65
Shortcomings of Current Forecasting Processes 75 not yet well established, but agencies are in some cases being asked to con- sider these effects in formulating plans and evaluating projects. In all but the most uncongested systems, the transportation network condi- tions assumed for purposes of initiating the forecasting process are not the con- ditions that would actually apply in view of the volumes of travelers and vehicles about to be forecast by the models. To compensate, it is common to feed back congested travel times from the forecast output to successive iterations of trip distribution, mode choice, and network loadings. As the modeled networks become more congested, feedback of this type becomes more important. Use of Models Without Regard for Their Limitations As noted earlier, travel models were originally developed for macro-scale regional planning. With many adjustments and new components, they have been adapted for the study of many other issues, including transit station boardings and projections of regional emissions. In the committee’s experi- ence, agencies have reported future-year facility volumes on the basis of data taken directly from the model outputs. Unless the models have been carefully restructured or estimated with the objective of addressing such issues, the resulting forecasts may not be valid. Peer Review Given the complexity of the modeling enterprise, it may be difficult to avoid altogether errors in modeling practice such as those catalogued above. Independent, rigorous, regular peer reviews of MPO models and practice are one means of reducing the incidence of these errors and assuring stakeholders of the quality of travel forecasts. Peer review has been ongoing for many MPOs on an ad hoc basis. The Federal Highway Administration’s Travel Model Improvement Program has provided financial support for peer review of models as well. LACK OR QUESTIONABLE RELIABILITY OF DATA Models can be responsive only to factors that have been included in their spec- ification. In some agencies, factors are omitted simply because data are insuffi-

OCR for page 65
Shortcomings of Current Forecasting Processes 79 for 2000.1 Considerable variation between the 20-year forecasts and the actual situation in 2000 can be seen. These data are not displayed as a critique of demo- graphic planning but to show the degree of uncertainty associated with such forecasts, regardless of how sophisticated the forecasting process in use may be. For most cities, the greatest variation was between forecast and actual val- ues for employment, which was significantly underpredicted for each of the six areas. It is instructive to note that the United States as a whole experienced a double recession in 1980–1981 (the period when these forecasts for 2000 were made) and that some parts of the country were particularly affected. Oregon, for example, lost 10 percent of its jobs during this recession, and it took 6 years to replace these jobs (Thompson 2004). The uncertainty associated with socioeconomic forecasts raises questions about the validity of travel demand modeling that produces deterministic point estimates of future travel. A better use of travel models might be for analysis of outcomes of a range of transportation alternatives, considering different sce- narios of future urban development. Such an approach would allow a city to best position itself for whatever policy makers believe the future may bring. Future Data Challenges The challenges of obtaining appropriate and sufficient data for modeling are magnified by such emerging issues as changes in lifestyle that affect the tradi- tional methods used to conduct home interview surveys, changes in census prod- ucts, and the need for data on daytime populations. Collection of Travel Data While MPOs today have data processing capabilities far superior to those applied in the original urban transportation studies, technological develop- ments and other considerations have combined to make the methodology of home interview surveys more problematic. In-person (or in-home) interviews have become very expensive and difficult to conduct in many urban settings, and interviews are now conducted more commonly by phone or through self- reporting travel diaries. Both of the latter approaches are less likely to elicit comprehensive trip reports than in-person interviews. While automated computer-assisted telephone interviewing helps hold down survey costs, per- mits real-time checking for data inconsistencies, and allows respondents to Percent variation between actual and forecast data is calculated as (forecast − actual)/actual. 1

OCR for page 65
METROPOLITAN TRAVEL FORECASTING Current Practice and Future Direction 80 be prompted in the same way as during an in-person interview, telephone interviewing has disadvantages compared with the in-person approach. The method used most commonly to select sample households for survey- ing is now random phone number selection, which limits the households in the sample to the subset with land-line telephone numbers. Changes in com- munications technology have made this method of selecting households even more questionable because many—typically those with younger persons—now depend solely on cellular phones, which cannot legally be contacted through automated dialing. Screening of calls with voice mail, answering machines, and caller identification has also reduced the effectiveness of phone interviewing. Moreover, contacting households by phone means that those whose members remain at home and can be contacted by phone are more likely to be sampled. Obtrusive telemarketing has an impact as well because many individuals will not respond positively to any phone solicitation, regardless of how well inten- tioned. Finally, in-person interviewing has the added advantage of enabling observation. Thus even if one does not interview the respondent, some infor- mation about the household can be assumed from observing the neighborhood, the type and condition of housing, the number of automobiles, and so on.2 Data from Census Products The Census Bureau no longer intends to collect long-form data from a large sample of housing units during decennial censuses; however, roughly com- parable long-form data will be available from the ACS (U.S. Census Bureau 2006). The ACS estimates will have higher standard errors than past decen- nial census long-form estimates because of smaller housing unit samples, even with 3- and 5-year sample accumulations. The ACS will provide transporta- tion planners with intercensus-year data on households, persons, and com- muters that previously were available only every 10 years. Introduction of the ACS will also affect the Public Use Microdata Samples and the Census Transportation Planning Package special tabulation of long-form data, which are extensively used by MPOs for model development (Eash 2005). Data on Daytime Populations Travel models are used for typical travel behaviors but are increasingly being used as well for planning of evacuations and relief efforts due to natural disas- ters, immunization programs, and risk assessments for homeland security. These 2 Personal communication, J. Zmud to T. Palmerlee, March 8, 2007.

OCR for page 65
Shortcomings of Current Forecasting Processes 81 new purposes bring their own data needs. An example is the estimation of day- time population—the number of people who are present in an area during nor- mal business hours. There are means of roughly calculating daytime populations from Census Bureau information on resident populations and workers com- muting into and out of an area (U.S. Census Bureau 2007). The time of day at which commuting takes place complicates the calculation, especially for employ- ment centers with a substantial number of shift workers. Further complication derives from the travel of such groups as students, tourists, and shoppers. Information sources on the various components of daytime populations are lim- ited, posing a challenge for these new uses of travel demand models. BIASES ARISING FROM THE INSTITUTIONAL CLIMATE Forecasts of costs, traffic, and revenue are made for the purpose of assessing courses of action. They are used regularly in planning and designing trans- portation facilities and policies. The practice of using travel demand forecasts for policy assessment is based on the understanding that large capital invest- ments and long-term commitments of public resources to operating and maintaining networks of facilities are always controversial. Objective analy- sis is needed to select wisely among alternative investment strategies. Both capital and operating costs of facilities are forecast during the process of plan- ning networks of transportation facilities. Forecasting often occurs in a politically contentious environment. Some communities desperately want facilities expanded to serve them; others orga- nize in fierce opposition to certain projects or to particular design character- istics that are proposed. Some interest groups therefore wish to exaggerate the expected traffic on a planned facility, while others seek to minimize estimated use. Forecasts are needed to facilitate compromises among approaches advo- cated by different interest groups. Travel and cost forecasts should not be expected to avoid or resolve political differences or debates. Rather, they are intended to inform and facilitate debate and to contribute to rational decision making and compromise, especially in complex and politically charged situa- tions. Forecasts are always subject to error and uncertainty, but they should be prepared honestly, data should not be falsified, and assumptions should be chosen on defensible and technical grounds and not because they favor cer- tain outcomes over others. Over the past 20 years, researchers have investigated the extent to which travel demand forecasts are objective or influenced by politics. In a well-known

OCR for page 65
METROPOLITAN TRAVEL FORECASTING Current Practice and Future Direction 82 and controversial report, Pickrell (1990) argued that in the United States, the majority of a sample of rail transit projects he studied were forecast to have ridership levels higher than those actually achieved when the projects were completed, while the vast majority of those projects experienced higher cap- ital and operating costs than had been forecast at the time funds were com- mitted. Thus, actual costs per rider turned out to be much higher than had been forecast. Other authors, including Richmond (2005), have argued that the outcomes of such forecasts were politically inspired; for reasons that could be explained and understood in terms of consultants’ behavior, they deliber- ately departed from reasonable expectations. Wachs (1990, 2001) examined forecasting for transportation projects as a complex phenomenon prone to both error and deliberate distortion. The Federal Transit Administration (FTA) evaluated the performance of 10 projects in 1990 (including those in the Pickrell study) and 19 other new projects in 2003. It was found that in 1990, none of the 10 new starts (all rail projects) achieved even 80 percent of forecast ridership; only one exceeded 70 percent (Figure 5-2). By 2003, the accuracy of forecasting had improved. Of 19 new starts (again all rail), eight achieved 80 percent of forecast ridership. Recently, a group of European scholars led by Professor Bent Flyvbjerg from the University of Aalborg has added fuel to the fire that has characterized this debate. This team studied hundreds of projects in many countries, includ- ing highways, rail projects, and bridges built over more than 50 years (Flyvbjerg et al. 2002, 2003, 2005, 2006). They found that costs are far more likely to be 100 <80% of forecast ridership 80 80% or more of forecast ridership Percent of Projects 60 40 20 0 1990 2004 Year of Study FIGURE 5-2 New start rail transit forecasts and actual ridership, 1990 and 2004.

OCR for page 65
Shortcomings of Current Forecasting Processes 83 underestimated than overestimated prior to construction, while patronage or use of facilities is far more likely to be overestimated than underestimated. If estimates are truly unbiased, overestimation and underestimation should be roughly equally likely. Of interest, the magnitude of forecast errors has not been declining over time. This suggests that, with some exceptions such as FTA New Starts, the performance of travel demand models and transportation cost esti- mates is not improving despite the efforts of many transportation researchers to improve the techniques employed. Forecast errors are also persistent across modes of transportation (roads and rail projects) and geography, though on average they are larger for rail than for highway projects. The above findings can be interpreted in different ways, leading trans- portation researchers and analysts to suggest alternative courses for corrective action. The first course is to undertake deeper and continuing research to iso- late the specific causes of divergence between forecasts and actual performance. Some have characterized the apparent optimism bias in forecasts as innocent and unsurprising. Facilities are less likely to be built, it is said, if their fore- cast costs are high and expected use is low, leading to the phenomenon of errors in one direction dominating facilities that have been built. Other research suggests, however, that optimism bias is hardly the result of inno- cence; in some cases, researchers have been able to document “strategic mis- representation” in the form of “adjusted” coefficients and “refined” parameters from one model run to the next. It is, of course, quite possible that some of the observed divergence is unintentional while some is deliberate. The con- duct of research on the causes of discrepancies between forecasts and actual performance is hindered by the fact that funds are rarely made available by public bodies in any country for follow-up analyses of the performance of forecasts after facilities have been built. It is both necessary and possible to chart a responsible course by develop- ing standards and procedures for evaluating forecasts of patronage, revenue, and costs in association with the planning of new transportation investment projects. The Department for Transport in Great Britain has issued a white paper on procedures for controlling optimism bias in forecasting (Flyvbjerg et al. 2004). Requirements that assumptions be reported and explained, that critical external peer review of forecasts be performed, and that standards for the use of data and the making of assumptions in forecasting be published are all helpful. In the United States, FTA is gradually developing a set of guide- lines and procedures designed to ensure that best practices are routinely employed in forecasting for new starts of urban rail systems. These strategies

OCR for page 65
METROPOLITAN TRAVEL FORECASTING Current Practice and Future Direction 84 would, at the very least, allow egregious deviations from objectivity and good practice to be recognized and criticized. A second promising course of action is the development of “reference class forecasting,” based on research that led to the awarding of the 2002 Nobel Prize in economics to Daniel Kahneman (Kahneman 1994; Lovallo and Kahneman 2003). Kahneman has argued that projects such as rail extensions, bridges, and highways should be evaluated on the basis of “outside” as well as “inside” views. That is, forecasts of patronage and costs should be placed within a range of variation established by previous projects of a similar class. If forecasts lie well outside a range thus established, they should be considered suspect and required to undergo further analysis. Several countries have applied these insights in developing guidelines for the evaluation of forecasts of traf- fic, revenue, and costs. While reference class forecasting holds promise, data limitations may make it impossible to obtain accurate representations of forecasts and actual results for previous projects to be included in the class. Moreover, cost overruns are often a function of changes in the scope of a project that evolve as a facility is being built. It is not clear how to account for this phenomenon appropriately in reference class forecasting. The divergence between forecasts and the actual performance of trans- portation projects is a complex and multidimensional problem. While it is possible to state that forecasts should be as free as possible from deliberate dis- tortion and misrepresentation, it remains difficult to prescribe mechanisms that can ensure this outcome. SUMMARY FINDINGS AND RECOMMENDATIONS The four-step or in some cases three-step trip-based modeling process is used by the vast majority of MPOs. The many shortcomings of this process have long been recognized. The weaknesses of current practice can be categorized as follows: (a) inherent weaknesses of the models, (b) errors introduced by modeling practice, (c) lack or questionable reliability of data, and (d) biases arising from the institutional climate in which the models are used. Inherent Weaknesses of the Models Critiques of the ability of the current modeling process to address the issues with which MPOs must deal are numerous. Most fundamentally, on the demand side, the process is not behavioral in nature; that is, it is not based on

OCR for page 65
Shortcomings of Current Forecasting Processes 85 a coherent theory of travel behavior and is not well suited to representing trav- elers’ responses to the complex range of policies typically of interest to today’s planners. On the supply side, the process is unable to represent dynamic con- ditions. The conventional travel demand models make use of networks, both highway and transit, in which impedances are averages over an extended period, reflect no uncertainty or unreliability, and are not representative of the conditions that would be expected or found by an individual traveler at the time a trip choice is made. The issues that the current widely used metropol- itan travel demand forecasting process cannot adequately characterize as a con- sequence of these deficiencies include the following: • Road pricing, including high-occupancy travel lanes; • Time-specific policies, such as parking, work schedules, or scheduling of truck deliveries; • Hourly speeds or traffic volumes; • Improvements to traffic operations; • Nonmotorized travel; • Peak spreading and highly congested networks; and • Goods movement. Poor representation of uncertainty is another deficiency. Most travel forecasting models produce a single answer, although the model is estimated, calibrated, and validated on the basis of data sets subject to sampling and other errors. There are many sources of error and uncertainty in travel demand forecasting, but end users of most travel forecasts would not be aware of these limitations. Errors Introduced by Modeling Practice A primary concern is the lack of sufficient data for proper validation of mod- els after the estimation and calibration of model parameters. As noted in the previous chapter, validation is often hampered by a lack of independent data sources, and many MPOs validate against much of the same data used to develop the models. Too often the later steps in a model chain (e.g., mode choice, assignment) are manipulated in an attempt to correct for errors in ear- lier stages. Moreover, scrutiny of forecasts made for Transit New Starts proj- ects has demonstrated that a lack of consistency in generalized cost relationships (e.g., time, distance, tolls) among various elements of a model can lead to counterintuitive and likely incorrect results.

OCR for page 65
METROPOLITAN TRAVEL FORECASTING Current Practice and Future Direction 86 Finally, travel models were originally developed for macro-scale regional planning. As new requirements have emerged, models have been used with- out regard to their limitations (with many adjustments and new components) for such purposes as forecasts of transit station boardings and projections of regional emissions. To ameliorate errors introduced by modeling practice, MPOs should conduct formal peer reviews of their modeling practice. Independent peer review of modeling practice is essential given the complexity of the modeling enterprise. Lack or Questionable Reliability of Data Errors in demographic forecasts can lead to the identification of incorrect loca- tions for trip origins and destinations, creating significant orientation errors in trip distribution and accessibility anomalies in transit forecasting. For exam- ple, considerable divergence is seen between 20-year forecasts of households, population, and employment and the actual situation 20 years later. These data show the degree of uncertainty associated with such forecasts, regardless of how sophisticated the forecasting process being used may be. There are also a number of emerging data challenges. They include the collection of travel data and data from census products, and estimates of urban daytime populations. MPOs, together with the federal government and the states, should determine data requirements for validating current travel forecasting models, meeting regulatory requirements, and developing freight models and advanced travel models. Biases Arising from the Institutional Climate Forecasts are always subject to error and uncertainty, but they should be pre- pared honestly, data should not be falsified, and assumptions should be cho- sen on defensible and technical grounds and not because they favor certain outcomes over others. Over the past 20 years, researchers have investigated the extent to which travel demand forecasts are objective or influenced by politics. Particularly in the areas of new transit and toll-road start-ups, there is evidence of a systematic bias toward patronage forecasts that are substantially higher and cost forecasts that are substantially lower than the actual performance of completed projects. This

OCR for page 65
Shortcomings of Current Forecasting Processes 87 phenomenon is known as “optimism bias.” To guard against this type of bias, MPOs and other planning agencies should conduct reasonableness checks of demand and cost forecasts for major projects. This can be accomplished by comparing forecasts with the performance of similar operational projects. Additional Recommendations Policy makers must have the ability to make informed decisions about future investments and public policies for the transportation system. In reviewing the findings presented in this chapter, the committee concludes that current travel forecasting models and modeling practice are inadequate for many of the purposes for which they are being used. The committee therefore recom- mends the development and implementation of new modeling approaches for forecasting demand that are better suited to providing reliable infor- mation for such applications as multimodal investment analyses, opera- tional analyses, environmental assessments, evaluation of a wide range of policy alternatives, toll-facility revenue forecasts, freight forecasts, and federal and state regulatory requirements. The committee acknowledges the evidence that current practice is also deficient in many respects and that introducing advanced models will not in itself improve practice. Therefore, steps must be taken to improve both current and future practice in metro- politan travel forecasting. Conclusion The focus of this chapter has been on the shortcomings of current travel fore- casting models for their intended uses. The next chapter reviews opportuni- ties for addressing these shortcomings and advancing the current state of the practice. REFERENCES Abbreviations FHWA Federal Highway Administration S&P Standard and Poor’s TRB Transportation Research Board

OCR for page 65
METROPOLITAN TRAVEL FORECASTING Current Practice and Future Direction 88 AECOM Consult, Inc. 2005. Research on Highway Congestion Relief. Working Paper. Federal Transit Administration, Washington, D.C. Cambridge Systematics, Inc. 2001. NCHRP Report 462: Quantifying Air-Quality and Other Benefits and Costs of Transportation Control Measures. TRB, National Research Council, Washington, D.C. Eash, R. 2005. Impacts of Sample Sizes in the American Community Survey. Presented at Conference on Census Data for Transportation Planning, Irvine, Calif., May. www.trb. org/conferences/censusdata/Resource-Impacts.pdf. FHWA. 2006. About FMIP. In Freight Model Improvement Program. www.fmip.gov/about/ index.htm. FitchRatings. 2003. Bliss, Heartburn, and Toll Road Forecasts. Project Finance Special Report, Nov. 12. Flyvbjerg, B., M. S. Holm, and S. Buhl. 2002. Underestimating Costs in Public Works Projects: Error or Lie? Journal of the American Planning Association, Vol. 68, No. 3. Flyvbjerg, B., N. Bruzelius, and W. Rothengatter. 2003. Megaprojects and Risk: An Anatomy of Ambition. Cambridge University Press, Cambridge, United Kingdom. Flyvbjerg, B., C. Glenting, and A. Kvist Ronnest. 2004. Procedures for Dealing with Optimism Bias in Transport Planning: Guidance Document. United Kingdom Department for Transport, London. Flyvbjerg, B., M. S. Holm, and S. Buhl. 2005. How (In)accurate are Demand Forecasts in Public Works Projects? Journal of the American Planning Association, Vol. 71, No. 2. Flyvbjerg, B., M. S. Holm, and S. Buhl. 2006. Inaccuracy in Traffic Forecasts. Transport Reviews, Vol. 26, No. 1. Hunt, J. D., R. Johnston, J. E. Abraham, C. J. Rodier, G. R. Garry, S. H. Putman, and T. de la Barra. 2001. Comparisons from Sacramento Model Test Bed. In Transportation Research Record: Journal of the Transportation Research Board, No. 1780, TRB, National Research Council, Washington, D.C., pp. 53–63. Kahneman, D. 1994. New Challenges to the Rationality Assumption. Journal of Insti- tutional and Theoretical Economics, Vol. 150, pp. 18–36. Lovallo, D., and D. Kahneman. 2003. Delusions of Success: How Optimism Undermines Executives’ Decisions. Harvard Business Review, July. Meyer, M. D., and E. J. Miller. 2001. Urban Transportation Planning: A Decision-Oriented Approach. McGraw-Hill, Boston, Mass. Pickrell, D. H. 1990. Urban Rail Transit Projects: Forecast Versus Actual Ridership and Costs: Final Report. U.S. Department of Transportation, Washington, D.C. Richmond, J. 2005. Transport of Delight: The Mythical Conception of Rail Transit in Los Angeles. University of Akron Press, Akron, Ohio. Rodier, C. J. 2004. Verifying Accuracy of Regional Models Used in Transportation and Air Quality Planning: Case Study in Sacramento, California, Region. In Transportation Research Record: Journal of the Transportation Research Board, No. 1898, Transportation Research Board of the National Academies, Washington, D.C., pp. 45–51. S&P. 2005. Traffic Forecasting Risk Study Update 2005: Through Ramp-Up and Beyond. In Standard & Poor’s 2006 Global Finance Yearbook. www2.standardandpoors.com/ spf/pdf/fixedincome/project_finance_2005_final.pdf.

OCR for page 65
Shortcomings of Current Forecasting Processes 89 Schofer, J. 2006. Summary Statement. In Expert Forum on Road Pricing and Travel Demand Forecasting, Proceedings, Office of the Secretary, U.S. Department of Transportation, Washington, D.C. Spear, B. 2006. A Summary of the Current State of the Practice in Modeling Road Pricing. In Expert Forum on Road Pricing and Travel Demand Modeling, Proceedings, Office of the Secretary, U.S. Department of Transportation, Washington, D.C. Thompson, J. 2004. Oregon’s Long Climb Back for Jobs. Issue Brief. Oregon Center for Public Policy, Silverton. TRB. 1995. Special Report 245: Expanding Metropolitan Highways: Implications for Air Quality and Energy Use. National Research Council, Washington, D.C. TRB. 2002. Special Report 264: The Congestion Mitigation and Air Quality Improvement Program: Assessing 10 Years of Experience. National Research Council, Washington, D.C. U.S. Census Bureau. 2006. American Community Survey: Design and Methodology. Technical Paper 67.1. U.S. Census Bureau. 2007. Estimated Daytime Population Calculation. www.census.gov/ population/www/socdem/daytime/daytimepopscalecalc.html. Vovsha, P., W. Davidson, and R. Donnelly. 2005. Making the State of the Art the State of Practice: Advanced Modeling Techniques for Road Pricing. In Expert Forum on Road Pricing and Travel Demand Modeling, Proceedings, Office of the Secretary, U.S. Department of Transportation, Washington, D.C. Wachs, M. 1990. Ethics and Advocacy in Forecasting for Public Policy. Business and Professional Ethics Journal, Vol. 9, Nos. 1–2. Wachs, M. 2001. Forecasting Versus Envisioning: A New Window on the Future. Journal of the American Planning Association, Vol. 67, No. 4.