Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
80 5.1 Introduction Much has been written and presented recently regard- ing model validation and reasonableness checking, includ- ing the FHWA Travel Model Validation and Reasonableness Checking Manual, Second Edition (Cambridge Systematics, Inc., 2010b); the Florida Department of Transportation (FDOT) FSUTMS-Cube Framework Phase II, Model Cali- bration and Validation Standards: Model Validation Guide- lines and Standards (Cambridge Systematics, Inc., 2007a); the final report for NCHRP Project 8-36B, Task 91, âVali- dation and Sensitivity Considerations for Statewide Mod- elsâ (Cambridge Systematics, Inc., 2010a); and the FHWAâs Shining a Light Inside the Black Box webinar series (Ducca et al., 2008). This chapter demonstrates how the information from Chapters 3 and 4 of this report can support the model valida- tion and reasonableness checking concepts and procedures presented in the aforementioned documents. It is intended to complement, not duplicate, other reference material on validation and reasonableness checking. The reader should review the references listed in the previous paragraph for more complete information on model validation and reason- ableness checking. There are two primary uses for the data provided in this report: â¢ Developing travel model components when no local data suitable for model development are available; and â¢ Checking the reasonableness of model components devel- oped using local data. In the first case, local data should be collected to validate the models or model components developed based on this report. In the second case, the data in this report can be used to supplement and support the validation and reasonableness checking process. 5.2 Model Validation Overview 5.2.1 Definitions It is important to provide clear definitions for the terms âvalidationâ and âreasonableness checkingâ as used in this report. Different references may provide different definitions or emphasize different aspects of model validation. The fol- lowing definitions of validation are used in the four refer- ences noted in Section 5.1: â¢ Validation is the application of the calibrated models and comparison of the results against observed data. Ideally, the observed data are data not used for the model esti- mation or calibration but, practically, this is not always feasible. Validation data may include additional data col- lected for the same year as the estimation or calibration of the model or data collected for an alternative year. Valida- tion also should include sensitivity testing. (Travel Model Validation and Reasonableness Checking Manual, Second Edition) â¢ Validation is the procedure used to adjust models to simu- late base year traffic counts and transit ridership figures. Validation also consists of reasonableness and sensitiv- ity checks beyond matching base year travel conditions. (FDOT FSUTMS-Cube Framework Phase II Model Calibra- tion and Validation Standards: Model Validation Guidelines and Standards) â¢ Validation is the process that determines whether or not a model is reasonably accurate and reliable while sensi- tivity assesses the ability of the model to forecast changes in travel demand based on changes in assumptions. (âValidation and Sensitivity Considerations for Statewide Modelsâ) â¢ Validation is âforecastingâ current travel patterns to dem- onstrate sufficient ability to reproduce highway counts and transit line volumes. (Shining a Light Inside the Black Box) C h a p t e r 5 Model Validation and Reasonableness Checking
81 A common theme in all of the above definitions is a com- parison against observed data, especially against locally col- lected travel data, traffic counts, and transit boardings. The data summarized in this report provide independently col- lected observed travel data. Of course, the data summarized in this report are not specific to any single location and, thus, do not fully satisfy the intent of model validation as defined above. The best use of the data in this report is to supplement local data. In areas with existing travel models, the data included in this report may be used for reasonableness checking. The observed travel data summaries and model parameters contained herein provide an independent source of data for comparing travel models estimated and calibrated using locally collected data to travel characteristics from other areas. 5.2.2 Model Validation and Reasonableness Checking Considerations The validation documents referenced in Section 5.1 pre- sent a number of considerations that should guide model validation and reasonableness checking: â¢ Model validation and reasonableness checking should encompass the entire modeling process from the develop- ment of input data required for model development and application to model results. â¢ Matching a specified standard such as âthe coefficient of determination for modeled to observed traffic volumes should be 0.89 or greaterâ is not sufficient to prove the validity or reasonability of a model. â¢ The intended model use affects model validation and rea- sonableness checking: â For models that will be used to assess short-term infra- structure improvements or design, validation efforts may focus on the ability of the model to reproduce exist- ing travel. â For models that will be used for planning and policy analyses, validation efforts may focus on the reasonable- ness of model parameters and sensitivities to changes in input assumptions. â¢ Planning for model validation and reasonableness check- ing is important to ensure that this important step is not overlooked and that data required to validate the models are collected. â¢ Variability and error are inherent in the travel modeling process. Variability and error occur in the input data used to estimate and apply travel models, in estimated or speci- fied model parameters, and in the data used to validate the models. 5.2.3 Uses of Data in This Report for Validation and Reasonableness Checking If the data and parameters included in Chapter 4 of this report are used to specify or enhance travel models for an area, the specification and collection of independent valida- tion data such as traffic counts, transit boardings, travel time studies, and special generator cordon counts (see Chapter 3) are required for model validation. Those data may be supple- mented with data from other sources such as the U.S. Census, ACS, LEHD, locally collected travel surveys, or other sources. The locally collected data may be used to perform traditional model validation tests such as comparisons of modeled to observed vehicle miles of travel, screenline crossings, traffic volumes on roadways, and transit boardings. If areas have existing travel models estimated from locally collected data, the data contained in Chapter 4 may be used for reasonableness checking of model parameters and rates for trip-based travel models. The information contained in Chapter 4 also can be used to check the reasonableness of more advanced modeling techniques such as activity-based travel models, provided the results from those models can be converted to the trips resulting from the tours and activities. 5.2.4 Layout of Chapter The remainder of this chapter provides an overview of the use of information contained in this report for model valida- tion and reasonableness checking. Section 5.3 focuses on vali- dation and reasonableness checking of existing travel models. Section 5.4 provides an example of model reasonableness checking of model components and overall validation of a travel model specified using information from Chapter 4. Section 5.5 provides cautions and caveats to using the data contained in this report for model validation and reasonable- ness checking. Although these data can provide useful infor- mation regarding the reasonableness of travel models, this information cannot be used to validate travel models. 5.3 Model Validation and Reasonableness Checking Procedures for Existing Models The general approach to model validation and reasonable- ness checking of existing models using information provided in this report focuses on answering the following questions: â¢ Are the rates and parameters developed for a specific model component for the region reasonable?
82 â¢ If the rates or parameters for a specific model component are different from what would be expected, are there other characteristics of the model being considered that would âexplainâ the differences? As discussed in Section 1.1, this report is the third of a series of NCHRP reports that summarize typical model rates and parameters. Thus, in some cases, results summarized in this report can be compared to those summarized in NCHRP Report 187 (Sosslau et al., 1978) and NCHRP Report 365 (Martin and McGuckin, 1998). Such comparisons might provide an idea of the stability or trends of specific model rates and parameters over time that may help identify the reasonable- ness of estimated or calibrated model parameters for a region. 5.3.1 Are the Estimated Model Rates for the Region Reasonable? Chapter 4 provides some aggregate summaries of travel data. The summaries are averages of individualsâ travel behav- iors summarized over different groupings of individuals, mar- ket segments, and regions. It should be possible to compare information reported in Chapter 4 to results from a travel model estimated for a region at some level of aggregation even if the underlying travel model for the region is unique. For example, suppose a region uses an activity-based travel model. Since the information reported in Chapter 4 is trip based, no direct comparison of model parameters is possible. However, many activity-based travel models produce travel forecasts for individuals that mimic typical travel surveys. Thus, it should be possible to summarize the results of the activity-based models to produce âtrip-basedâ summaries for statistics such as trip rates, average trip lengths, time of day of travel, mode shares, and so forth. ExampleâReasonableness of Trip Generationâ A âSuccessâ Story Tables 5.1 through 5.3 show a typical trip generation model estimated for an example large urban area with a population between 1 and 3 million people. Table 5.4 shows the total trip rates resulting from Tables 5.1 through 5.3. Tables 5.5 and 5.6 provide comparisons of the average trip rates by household size and by income group for the example Income Group Household Size Average 1 2 3 4 5+ Low (Less than $25,000) 0.5 1.4 1.4 1.4 2.7 0.8 Middle ($25,000â$99,999) 1.3 1.9 2.1 2.3 2.7 1.9 High ($100,000 or more) 1.0 1.9 2.6 2.5 2.1 2.2 Average 1.1 1.9 2.2 2.4 2.5 1.8 Table 5.1. Modeled home-based work trip production rates for example urban area. Income Group Household Size Average1 2 3 4 5+ Low (Less than $25,000) 1.5 2.6 5.4 5.5 5.6 2.2 Middle ($25,000â$99,999) 1.7 3.6 5.3 8.3 11.6 4.9 High ($100,000 or more) 1.9 3.2 5.3 10.5 11.6 6.2 Average 1.6 3.4 5.3 9.2 11.5 4.9 Table 5.2. Modeled home-based nonwork trip production rates for example urban area. Income Group Household Size Average1 2 3 4 5+ Low (Less than $25,000) 0.9 0.9 3.3 3.1 3.1 1.1 Middle ($25,000â$99,999) 1.5 2.8 3.3 4.0 3.8 2.8 High ($100,000 or more) 2.5 3.5 4.7 5.1 6.3 4.4 Average 1.4 2.9 3.7 4.5 4.6 3.0 Table 5.3. Modeled nonhome-based trip production rates for example urban area.
83 Income Group Household Size Average 1 2 3 4 5+ Low (Less than $25,000) 2.9 4.9 10.1 10.0 11.4 4.1 Middle ($25,000â$99,999) 4.5 8.3 10.7 14.6 18.1 9.6 High ($100,000 or more) 5.4 8.6 12.6 18.1 20.0 12.8 Average 4.1 8.2 11.2 16.1 18.6 9.7 HBW = home-based work; HBNW = home-based nonwork; NHB = nonhome based. Table 5.4. Total trip production ratesâHBW î± HBNW î± NHB for example urban area. Trip Purpose and Data Source Household Size 1 2 3 4 5+ Home-Based Work Hypothetical Region 1.1 1.9 2.2 2.4 2.5 NHTS 0.5 1.2 2.0 2.3 2.4 Home-Based Nonwork Hypothetical Region 1.6 3.4 5.3 9.2 11.5 NHTS 1.8 4.0 6.7 10.6 13.4 Nonhome based Hypothetical Region 1.4 2.9 3.7 4.5 4.6 NHTS 1.3 2.5 3.8 5.3 5.7 Total Hypothetical Region 4.1 8.2 11.2 16.1 18.6 NHTS 3.6 7.7 12.5 18.2 21.5 Source: 2009 NHTS. Table 5.5. Comparison of example region to NHTS trip production rates by household size. Trip Purpose and Data Source Income Range Less than $10,000 $10,000â $24,999 $25,000â $49,999 $50,000â $99,999 $100,000 or More Home-Based Work Hypothetical Region 0.8 1.9 2.2 NHTS 0.6 0.8 1.3 1.9 2.0 Home-Based Nonwork Hypothetical Region 2.2 4.9 6.2 NHTS 4.1 4.7 5.0 6.2 7.6 Nonhome based Hypothetical Region 1.1 2.8 4.4 NHTS 1.6 1.9 2.7 3.8 4.7 Total Hypothetical Region 4.1 9.6 12.8 NHTS 6.3 7.4 9.0 11.9 14.3 Source: 2009 NHTS. Table 5.6. Comparison of example region to NHTS trip production rates by income group.
84 urban area with the comparable rates from the NHTS as summarized in Section 4.4.4. For the example urban area, the home-based work average household trip rates are higher than the averages shown by the NHTS for all household sizes although they are close for households of three or more per- sons. For the home-based nonwork trip purpose, the trip rates by household size for the example urban area are all lower than the NHTS trip rates. For the nonhome-based trip purpose and for all trip purposes combined, the results were mixed with example urban area rates being higher than NHTS rates for the lowest two household sizes and lower for the top three household sizes. The comparison of trip production rates by income group shown in Table 5.6 is not quite as straightforward as the comparison of trips by household size as shown in Table 5.5. Unlike household sizes, income groups are affected by the year for which the incomes were reported, the income group breakpoints used in the survey and, possibly, by the region of the country for which the incomes were reported. For the 2009 NHTS data, the incomes were reported in 2008 dollars. Thus, for the example urban area, Consumer Price Index information was used to convert the income group dollar ranges from 1998 dollars to 2008 dollars. After the conversion, the income group breakpoints for the exam- ple urban area were reasonably close to the $25,000 and $100,000 breakpoints in the NHTS data. After the conver- sion of the income group breakpoint for the example area, the lowest-income group for the example area spanned two income groups for the NHTS data, as did the middle- income group. After the adjustments of the income groupings, the home- based work trip rates for the example urban area were higher than the comparable income groups in the NHTS data. The trip rates for the example urban area were at the low end or lower than the comparable income groups in the NHTS data for both the home-based nonwork and nonhome-based trip purposes. Results for total trip rates were mixed. Since the NHTS provides an agglomeration of trip rates for many urban areas throughout the country, there would be no reason to expect the trip rates from the example region to precisely match those obtained from the NHTS data. Nevertheless, it would be reasonable for the estimated trip rates for the region to reflect similar patterns to those shown in the NHTS data. The marginal trip rates for the example urban area by household size and by income group shown in Tables 5.5 and 5.6 reflect the NHTS trip rate patterns. While there are differences between the marginal trip rates for the example region and the NHTS data, the rates from the two sources reflect similar trends. Thus, while the NHTS data cannot be used to validate the trip rates for the example region, the comparison demonstrates an overall reasonable- ness of the trip generation model for the example region. ExampleâReasonableness of Trip Distributionâ A âNonsuccessâ Story The preceding example regarding trip generation rates provided a âsuccessâ story where the model in question was supported as being reasonable even though the trip gen- eration rates did not precisely match the rates summarized from NHTS data. The following example describes a situa- tion where simple comparisons to the summaries included in this report would have suggested that a regional model might not be reasonable. Additional analyses would be required to determine the reasonableness of the model. Trip-based travel models were developed for a midsized urban area (population between 500,000 and 1 million). The observed average trip duration for home-based work trips was summarized from the household survey as 35.4 minutes for all person trips by auto. This average was based on congested auto travel times. Based on data from the 2009 NHTS, as reported in Table C.10 in Appendix C, the average home-based work trip duration for an urban area with 500,000 to 1 million people was 22 minutes. Thus, the observed average home-based work trip duration for the region appeared to be too high. Such a conclusion led to additional analysis. The initial checks of the processing of the observed data, the mod- eled congested travel speeds used in conjunction with the reported trip interchanges to estimate the average trip dura- tion, and the trip durations reported by the travelers in the household survey confirmed the 35-minute average for the home-based work trip duration. The analyses also showed that the average trip durations for home-based nonwork and nonhome-based trips were within reasonable ranges based on summaries of NHTS data. Further investigation focused on the share of home-based work trips as a proportion of total trips. Reported home- based work, home-based nonwork, and nonhome-based trip shares were 11 percent, 54 percent, and 35 percent, respec- tively. For urban areas with 500,000 to 1 million people, the NHTS data showed these shares as 14 percent, 56 percent, and 30 percent. The low home-based work share coupled with the long average trip duration suggested that the region was different from other similar-sized urban areas. Anecdotal information from local planners provided a plau- sible explanation for the differences. Specifically, due to the state of the public school system at the time, many residents Other sources might be considered for checking the reason- ableness of home-based work trip rates. Specifically, CTPP/ ACS data may provide alternative sources for determining HBW trip rates.
85 enrolled their children in private and parochial schools. Since the private and parochial schools were often beyond walking distance, school children were driven to and from school by parents as part of the parentsâ work journeys. This anecdotal information was supported by the reported travel patterns in the regional travel survey. The local planners also were uncon- cerned regarding the 35-minute average trip duration for direct home-to-work trips due to general roadway congestion levels. The result of the analyses led to modifications in the design of the trip-based travel models for the region. The models were designed to explicitly account for the increased serve passenger trips made by parents to serve the school trips of their children. ExampleâModel Parameters (Trip Distribution) It can be useful to compare estimated model parameters to those developed in other regions as a reasonableness check. This is, quite often, a step used in the estimation of discrete choice models such as mode choice models. However, it also can be performed using more aggregate models. Suppose a region estimated the following gamma function parameters for a home-based work trip distribution model implemented using the gravity model: a = 5,280 b = -0.926 c = -0.087 A review of Table 4.5 contained in Chapter 4 does not pro- vide any clear indication regarding the reasonableness of the parameters. However, since the a parameter is simply a scale value, it can be modified to plot the various gamma functions over the same range of values. Figure 5.1 shows the resulting plot of the various functions. Again, while the data in Chap- ter 4 cannot validate the parameters estimated for the regional model, the information shown in Figure 5.1 suggests that the estimated function may be reasonable. However, some cau- tion might be warranted if the example region was medium sized. The example function is generally steeper at low travel times and produces friction factors that are lower than the other medium-sized region friction factors over most of the range of travel times. ExampleâTemporal Validation Some of the summaries contained in Chapter 4 can be compared to similar summaries contained in its predecessors, NCHRP Reports 187 and 365. For example, Table 5.7 compares average household trip rates from those two reports and sum- maries of 2009 NHTS data, while Table 5.8 compares shares of total trips by trip purpose. For urban areas with populations greater than 500,000, household-based average trip rates appear to be generally increasing over time. The rates appear to be generally decreasing for areas with populations less than 500,000. For shares of trips by trip purpose, home-based work shares are decreasing over time while nonhome-based shares Figure 5.1. Comparison of trip distribution gamma functions.
86 are increasing. For regions that are updating or redeveloping models, comparing aggregate results to trends that can be drawn from this report and its predecessors can be useful for checking model reasonableness. 5.4 Model Validation and Reasonableness Checking Procedures for Models or Model Components Developed from Information Contained in Chapter 4 The Travel Model Validation and Reasonableness Checking Manual, Second Edition recommends the development of a model validation plan when a model is developed or updated. The validation plan should outline model validation and rea- sonableness checks that will be performed along with the validation data that will be used as the bases for comparison for the model results. This recommendation holds true for models developed from locally collected travel survey data or models specified using rates borrowed from other regions or provided in Chapter 4. As an example, suppose an MPO for a region of 250,000 people was updating its travel model based on rates pro- vided in Chapter 4 and Appendix C. The existing travel model had been specified using data from NCHRP Report 365, and no travel survey data were available. A validation plan was developed and, based on that plan, available resources were focused on the collection of traffic counts (daily and by time of day). In addition, staff from the MPO and their families were asked to record travel times on their trips to and from work. Since the travel model being updated was based on NCHRP Report 365 rates, the MPO had developed a procedure to esti- a,c a,c Table 5.7. Comparison of household trip rates. Urbanized Area Population Percentage of Daily Person Trips by Trip Purpose NCHRP Report 187a (Published 1978) NCHRP Report 365a (Published 1998) 2009 NHTS Datab HBW HBNW NHB HBW HBNW NHB HBW HBNW NHB 50,000 to 100,000 16 61 23c 20c 57 c 23 c 15 54 31 100,000 to 200,000 20 57 23c 20c 57 c 23 c 15 54 31 200,000 to 500,000 20 55 25c 21c 56 c 23 c 15 54 31 500,000 to 1,000,000 25 54 21c 22 56 c 22 c 14 56 30 1,000,000 to 3,000,000 25 54 21c 22c 56 c 22 c 14 56 30 More than 3,000,000 25 54 21c 22c 56 c 22 c 14 56 30 a Shares by purpose are based on person trips in motorized vehicles. b Shares by purpose are based on person trips by all modes. c Because of differences between urban area categories in the three reports, the rates shown were chosen from the closest matching category. HBW = home-based work; HBNW = home-based nonwork; NHB = nonhome based. Source: Sosslau et al. (1978), Martin and McGuckin (1998), 2009 NHTS. Table 5.8. Comparison of shares of trips by trip purpose.
87 mate households by household size. Average trip production rates from Tables C.5 through C.7 were used to specify the trip rates shown in Table 5.9 for the example MPO model. Table 5.10 shows the trips per household resulting from the applications of the original model based on NCHRP Report 365 rates along with results from the application of the model summarized in Table 5.9 using the MPOâs socio- economic distributions of households by household size. Table 5.10 also shows the average trip rates for MPOs from Tables C.5 through C.7. The table also shows the modeled distributions of trips by trip purpose resulting from the origi- nal and updated models. Based on the information shown in Table 5.10, MPO modeling staff suspected that the model would result in more travel in the region than would be shown by the observed traffic counts. Trips were distributed using the friction factors for âMedium (A)â MPOs shown in Table 4.5. The informal travel time sur- vey of MPO staff did not suggest any substantial issues with the coded network speeds. Most staff reported observed travel times within Â±10 percent of the modeled travel times for their trips from home to work. The modeled average trip dura- tions are shown in Table 5.11 along with the average trip durations for urban areas of less than 500,000 population from Table C.10. The results shown in Table 5.11 also sug- gested that the model would show less travel in the region than would be shown by the observed traffic counts. When the modeled vehicle trips were assigned (after apply- ing mode split, auto occupancy, and time-of-travel model components), the resulting vehicle miles of travel were close to the vehicle miles of travel estimated from the traffic counts Trip Purpose Household Size 1 2 3 4 5+ HBW 0.5 1.2 2.0 2.3 2.4 HBNW 1.8 3.6 6.7 9.5 12.9 NHB 1.3 2.5 3.8 5.3 5.7 HBW = home-based work; HBNW = home-based nonwork; NHB = nonhome based. Table 5.9. Initial trip production rates for example urban area. Measure HBW HBNW NHB Total Trip Rates Original Modela 1.8 4.8 2.0 8.6 Updated Model 1.6 5.4 3.1 10.1 MPO Averages 1.4 5.1 3.0 9.6 Distribution of Trips by Purpose Original Modela 21% 56% 23% 100% Updated Modelb 15% 53% 32% 100% MPO Averagesc 15% 53% 32% 100% a Based on Martin and McGuckin (1998), Table 9. b Based on model shown in Table 5.9. c Tables C.5 through C.7. HBW = home-based work; HBNW = home-based nonwork; NHB = nonhome based. Table 5.10. Initial trip production reasonableness check for example urban area. Measure Trip Durations in Minutes HBW HBNW NHB Total Implied by Table C.10 20 18 18 18 Based on Model Application 18 16 â11% 18 17 â4% Percentage Difference â10% 0% HBW = home-based work; HBNW = home-based nonwork; NHB = nonhome based. Table 5.11. Initial trip distribution reasonableness check for example urban area.
88 collected for the model validation. Modeled screenline cross- ings were within 10 to 15 percent of the observed screenline crossings. Based on the information provided by the reason- ableness checks for the trip production and trip distribution models and the model validation results, both the trip pro- duction and trip distribution models were deemed to pro- duce reasonable results. 5.5 Cautions Regarding Use of This Report for Validation The examples shown in this chapter illustrate both the risk and value of using information contained in this report for model validation and reasonableness checking. Since the data contained in Chapter 4 are highly aggregated from nationally collected data, they can be used only for general reasonableness checking. As stated previously, agreement between modeled information for a specific region and the general information in this report for any single measure is insufficient to dem- onstrate that a model for the region is valid. Likewise, failure to reasonably match the general summaries contained in this report does not invalidate a regional travel model. However, failure to reasonably match a general summary contained in this report should lead to further investigation of a regional travel model to explain the difference from the general travel patterns resulting from typical traveler behavior. It also is important to verify that the data being compared are, in fact, comparable. A prime example of this issue is trip generation. Many regions summarize and forecast all person travel made in motorized vehicles, while others summarize and forecast all person travel. Efforts have been made in Chapter 4 to clearly identify whether all travel or only travel in motorized vehicles has been included in the summaries. Finally, differences in data collection and processing tech- niques can introduce variation in the summarized data. There is a high level of consistency in the collection and pro- cessing of the NHTS data summaries contained in Chapter 4. However, since different MPOs have collected data for their own regions and developed their own models from those data, summaries of MPO-reported data and parameters are subject to variation from the data collection and processing procedures.