National Academies Press: OpenBook

Performance Measures for Freight Transportation (2011)

Chapter: Chapter 6 - Data Considerations to Support Performance Measurement

« Previous: Chapter 5 - Freight Stakeholder Preferences
Page 52
Suggested Citation:"Chapter 6 - Data Considerations to Support Performance Measurement." National Academies of Sciences, Engineering, and Medicine. 2011. Performance Measures for Freight Transportation. Washington, DC: The National Academies Press. doi: 10.17226/14520.
×
Page 52
Page 53
Suggested Citation:"Chapter 6 - Data Considerations to Support Performance Measurement." National Academies of Sciences, Engineering, and Medicine. 2011. Performance Measures for Freight Transportation. Washington, DC: The National Academies Press. doi: 10.17226/14520.
×
Page 53
Page 54
Suggested Citation:"Chapter 6 - Data Considerations to Support Performance Measurement." National Academies of Sciences, Engineering, and Medicine. 2011. Performance Measures for Freight Transportation. Washington, DC: The National Academies Press. doi: 10.17226/14520.
×
Page 54
Page 55
Suggested Citation:"Chapter 6 - Data Considerations to Support Performance Measurement." National Academies of Sciences, Engineering, and Medicine. 2011. Performance Measures for Freight Transportation. Washington, DC: The National Academies Press. doi: 10.17226/14520.
×
Page 55
Page 56
Suggested Citation:"Chapter 6 - Data Considerations to Support Performance Measurement." National Academies of Sciences, Engineering, and Medicine. 2011. Performance Measures for Freight Transportation. Washington, DC: The National Academies Press. doi: 10.17226/14520.
×
Page 56
Page 57
Suggested Citation:"Chapter 6 - Data Considerations to Support Performance Measurement." National Academies of Sciences, Engineering, and Medicine. 2011. Performance Measures for Freight Transportation. Washington, DC: The National Academies Press. doi: 10.17226/14520.
×
Page 57
Page 58
Suggested Citation:"Chapter 6 - Data Considerations to Support Performance Measurement." National Academies of Sciences, Engineering, and Medicine. 2011. Performance Measures for Freight Transportation. Washington, DC: The National Academies Press. doi: 10.17226/14520.
×
Page 58
Page 59
Suggested Citation:"Chapter 6 - Data Considerations to Support Performance Measurement." National Academies of Sciences, Engineering, and Medicine. 2011. Performance Measures for Freight Transportation. Washington, DC: The National Academies Press. doi: 10.17226/14520.
×
Page 59
Page 60
Suggested Citation:"Chapter 6 - Data Considerations to Support Performance Measurement." National Academies of Sciences, Engineering, and Medicine. 2011. Performance Measures for Freight Transportation. Washington, DC: The National Academies Press. doi: 10.17226/14520.
×
Page 60
Page 61
Suggested Citation:"Chapter 6 - Data Considerations to Support Performance Measurement." National Academies of Sciences, Engineering, and Medicine. 2011. Performance Measures for Freight Transportation. Washington, DC: The National Academies Press. doi: 10.17226/14520.
×
Page 61
Page 62
Suggested Citation:"Chapter 6 - Data Considerations to Support Performance Measurement." National Academies of Sciences, Engineering, and Medicine. 2011. Performance Measures for Freight Transportation. Washington, DC: The National Academies Press. doi: 10.17226/14520.
×
Page 62

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

50 c h a p t e r 6 Data Considerations to support Performance Measurement Summary Of the many challenges to developing a nationwide freight performance measurement system, the greatest is the com- plexity of gathering adequate data. It is self-evident that per- formance measurement relies on data and that the measure- ment system can only be as sound as the data it consists of. As mentioned in earlier chapters, the availability of sound, consistent, sustainable data was an overriding consideration in the selection of measures for the first-generation Freight System Report Card. Although stakeholder interviews indi- cated a desire for additional measures, the measures selected for the report card were ones for which data are readily and consistently available. Although measures were selected for which data exist, the ongoing population of the report card will represent an enor- mous data challenge. This section examines the challenges of freight system data collection that will need to be addressed. It also includes two relevant case studies—one of the Freight Analysis Framework (FAF) and one of the Transportation Services Index (TSI). Both are highly relevant in that they are analogous efforts to integrate freight data from a wide array of sources into a common reporting format. Their experience illustrates, on a smaller scale, the type of effort necessary to develop a freight report card. Freight Data Issues State and federal practitioners have identified significant gaps in the freight data available for performance measure- ment. In A Concept for a National Freight Data Program: Spe- cial Report 2761 the shortcomings in federal freight data sets were summarized. [T]he current disjointed patchwork of freight data sources is costly to generate and maintain but does not provide decision makers with the data they require. To remedy this deficiency, a national freight data framework is needed to guide the develop- ment of a national freight database and related data collection and synthesis activities with the potential to meet users’ data requirements. The report notes that many users’ needs require freight data that are not available from any single source. Thus, it is frequently necessary to combine data from different sources. The combination of data from different sources, often known as “data fusion,” is frequently problematic. Much of the exist- ing data were developed by different entities, over different times with different generations of technology. The sources differ in their modal coverage, collection techniques, and data definitions. Significant concerns were identified in Special Report 276 regarding the use of the existing data for a com- prehensive national freight database: A further deficiency of existing sources of freight transporta- tion data is that some of the information required by decision makers is simply not available. For example, informed efforts to alleviate highway congestion require data on routes traveled, time of day, and the types of trucks and commodities caught in congestion—data that are rarely collected, at least in the United States. Both the committee’s discussions with users and the per- sonal experience of individual members revealed a sense of frus- tration with existing freight data. The disjointed array of data sources is cumbersome and difficult to use, lacking in geographic detail, and notably deficient in covering increasingly important motor carrier flows. Several users also expressed concern about the unnecessary burden on data providers, who may be asked to provide similar data to different organizations—sometimes in different formats. This heavy respondent burden is likely to hinder efforts to gather quality data. A pending NCFRP Project 12 has been scoped to further develop a national freight data architecture. Its objectives include developing the specifications for the content and structure of a freight data architecture, to identify the value and challenges of the potential architecture, and to specify institutional strategies to develop and maintain the archi- tecture. This architecture is intended to serve the needs of

51 public and private decision makers at the national, state, and local levels. A study conducted for the Washington State DOT2 iden- tified 32 different data sets that the state could include in its freight data system. Despite the number of sets that can supply some data, the report noted that “very little system- atic data exists to inform decision makers about the eco- nomic impact, system bottlenecks, and supply chains flowing through freight systems that support Washington State pro- ducers and delivery of goods to consumers.” In Texas, the authors of a study on potential freight perfor- mance measures summarized the state of current freight data for performance measures thus: Freight performance measures (FPM) in the U.S. are currently at a very early stage in their development. Some states have made a push to look into FPMs or to begin some data collection to assess what would be required for an integrated ITS-PM system. However, most states have not yet utilized their performance measures across modes. The general consensus is that the imple- mentation of a comprehensive set of FPMs requires far more data-collection capability than most states currently possess.3 The authors note that even the leading work that has been done has focused on broad goals and objectives, rather than specific performance metrics. Another study of data sources in Texas identified 31 separate databases that could be used for some aspect of freight system performance analysis.4 At least two major areas of data improvement will need to be addressed to implement a freight performance measure- ment report card. First are the issues related to the integration and governance of state and federal transportation data, or the processes by which data from different sources are syn- thesized and stored so that they can be analyzed by users and decision makers. The literature indicates that transportation data integration from a wide array of providers will present significant technical, policy, and logistical challenges. Second is the issue of the quality and quantity of freight- related data. The experience of other transportation agencies suggests that freight-related data has continually improved in recent years but still lacks the detail, breadth, and com- pleteness necessary for consistent, nationwide performance measurement. data integration and governance Data governance and data integration will be essential ele- ments of a freight performance measurement system. Data governance has been defined as “the overall management of the availability, usability, integrity, and security of the data employed in an enterprise. A sound data governance program includes a governing body or council, a defined set of proce- dures, and a plan to execute those procedures.”5 One author has noted, “Data governance is a system of decision rights and accountabilities for information-related processes, executed according to agreed-upon models that describe who can take what actions with what information, and when, under what circumstances, using what methods.”6 FHWA defines data integration as “The process of combin- ing or linking two or more data sets from different sources to facilitate data sharing, promote effective data gathering and analysis, and support overall information management activities in an organization.”7 Following are some of the governance and integration issues that will need to be addressed in deploying a dashboard for freight-related performance measures. 1. The development of common data definitions for organi- zations providing data to a national set of freight perfor- mance measures; 2. Development of data quality and accuracy standards; 3. Development of protocols to integrate multiple sources of data into the framework; 4. Development of strategies to close data gaps; 5. Development of strategies to assure data availability; a. From sources; b. From the report card or sets of measures themselves; 6. Time to access data from the framework; 7. Identification of real-time versus archived data needs; and 8. Sustainability of the data framework. The Data Integration Primer8 notes, “The data integration process can be extremely involved and challenging, especially for organizations that have a long history of stand-alone files or rarely share data across database systems. “ Although the Data Integration Primer focuses only upon asset manage- ment, its underlying principles apply to broader types of data integration efforts. It notes that a careful analysis of organi- zational needs should precede data integration efforts. Use cases (analyses of the activities to be performed) and cus- tomer requirements are clearly needed to ensure whether the data integration effort meets the performance measurement effort’s needs. Also, a wide range of stakeholders and practi- tioners should be involved to identify the different needs that different users have for the performance measure data to be eventually integrated. Freight data Quality and Quantity Many national transportation-related data systems that could feed a freight performance system are generated by data produced by the states. This is the case for such applica- tions as the Highway Performance Management System, the Fatality Analysis Reporting System, and the National Bridge

52 Inventory. As states have increasingly focused upon perfor- mance measures and performance benchmarking with peer states, the deficiencies in their performance data have become more apparent. Comparative Performance Measurement: Pavement Smoothness9 notes significant variation in how different state highway agencies gather a very basic piece of performance data, the International Roughness Index (IRI) data for pavements. This variation occurs even though the data are machine gathered, protocols exist to calibrate the machines, and standards exist to assess the data. The study estimated that up to a 15 percent variation exists between how states record the data. In addition, it found that wide variation exists as to how states could manipulate and extract the data for comparative analysis. These variations occurred even though all states must record IRI and that considerable effort has been expended to establish clear standards and pro- tocols to ensure national consistency. The study noted that factors such as the tire pressure of the test vehicle, speed of the vehicle, or the driver’s strict adherence to the wheel path all influenced whether the data were consistent, accurate, or replicable between states. The General Accounting Office10,11,12 noted the difficulty of accurately assessing the number of truck-related fatal crashes because of inconsistent data reporting by the states. It noted in 1999 that states failed to report 38 percent of all report- able crashes involving trucks and 30 percent of fatal crashes involving trucks. It attributed the data gaps to a lack of state laws compelling state and local officials to supply data to federal officials. In 2004 and 2007 follow-up studies, GAO reported that reporting had improved but still 21 percent of crash reports lack complete data. It reported that timeliness in reporting had improved from 32 percent reported on time in 2000 to 89 percent in 2007. GAO noted that the FMCSA spent $21 million in grants over three years to improve the data reporting practices of 34 states. The studies of IRI data and truck-crash data illustrate the complexities of using even one traditional performance mea- sure for comparative analysis. State DOT data practitioners describe complexities that are orders of magnitude greater when they described integrating data across a number of dif- ferent legacy systems. The state’s experiences were summa- rized in proceedings for the TRB Workshop on Challenges of Data for Performance Measures, in 2006.13 Summaries of several states’ observations of their own data challenges and needs were provided. California DOT data are stored in myriad databases that are disjointed and uncoordinated, have varying usability, and are inconsistent or duplicated in other databases. Lane miles in one database may include miles maintained that are not state highways or could include proposed relinquish- ments. This confusion leads to different answers to the same question, resulting in duplicative manual recreation of data. More importantly, users lose confidence in the data. Most performance measures are developed within the division supporting its respective performance measures and are not developed as part of an overall data collection program. Alaska DOT has a data steward role that includes collection, quality control, transformation, documentation, archiving, and access of transportation data. Some of the issues that the agency has to overcome are institutional, parochial, data stovepipes, technology changes, and evolving department business requirements. Minnesota DOT’s representatives noted that there were limited numbers of tools for policy, programs, and executive-level decision making. This is at least partly due to issues related to data quality, availability, systems integra- tion, and tools to retrieve data, analyze it, develop predic- tive models, conduct trade-off analysis, and report results in useful formats. Virginia DOT reported that the effort of creating a dash- board of performance measures was made simpler by having a data warehouse. In the development of performance mea- sures, the agency combined different kinds of data to produce a single measure. The data warehouse provided that one stop for the different data used to automate the generation of the performance measures. Where data do not exist, the busi- ness requirements are formalized for data needed before any changes are made to existing systems or before new systems are developed. For non-automated performance reports, data come from many sources, including spreadsheets, templates, and e-mail. A lack of standardization in the number and defi- nition of data fields collected has made statewide incident management reporting difficult. The agency is in the process of overhauling the system that tracks the operations. Washington DOT notes that there is consensus that the agency needs better or more complete data. Based on a direc- tive by the legislature, Washington DOT completed a study of 11 core technology systems. According to the study none of the 11 core systems met even 20 percent of the agency’s cur- rent and future business and technical requirements. WSDOT is currently addressing the unmet needs through tremendous manual effort and use of multiple ad hoc systems. Florida DOT notes that data intricacies in collection and storage can get lost in generalization of a large database. There were challenges with keeping the data current and repeatable and having consistent data and sources. The blended mea- sures may have data from various sources and new data need to be addressed.

53 Case Studies The complexities of data integration and of addressing data deficiencies were clearly evident in the development of two representative freight information systems, the FAF and the TSI. Although neither are performance measurement systems, both provide comprehensive data regarding freight volumes, origins, destinations, and other trend information. The level of effort that was necessary for these two systems provides an order-of-magnitude example of the complexities facing the development of a comprehensive freight perfor- mance measurement system. Freight analysis Framework case study The Freight Analysis Framework integrates data from a variety of sources to estimate commodity flows and related freight transportation activity among states, regions, and major international gateways. The first version of FAF pro- vides estimates for 1998 and forecasts for 2010 and 2020. The second version provides estimates for 2002 and the most recent year plus forecasts through 2035. The FAF Commodity Origin-Destination Database esti- mates tonnage and value of goods shipped by type of com- modity and mode of transportation among and within 114 areas, as well as to and from seven international trading regions throughout the 114 areas plus 17 additional international gate- ways. The 2002 estimate is based primarily on the Commodity Flow Survey and other components of the Economic Census. Forecasts are included for 2010 to 2035 in five-year increments. Officials of FAF report that at present the effort requires one full-time U.S. DOT staff person and two full-time con- sultants. Both Battelle Memorial Institute and the Oak Ridge National Lab support the ongoing FAF efforts. The initial FAF setup cost was about $1 million and was spent on acquiring private data. Because there were privacy issues with the data, the detailed analysis and input/output data could not be shared with users. The next phase cost $600,000 and was a two-year effort focused on construct- ing models. This allowed the agency to share the commodity data with users. The system captures data from the Bureau of Transportation Statistics (BTS), the Federal Aviation Administration, the U.S. Army Corps of Engineers, and the Energy Information Association, as well as trans-border U.S. Customs Service data, census data, and foreign trade data. Private-sector data come from ATA and AAR. FAF captures only “for hire” shipping and does not capture shippers who use internal fleets, such as Wal-Mart and others who trans- port their own goods. There are no precise data available about who uses the FAF data and how frequently. From experience, the pro- gram managers believe that MPOs, the state DOTs, and private- sector users have regularly consulted the data. They believe that private-sector firms such as GE, UPS, FedEx, and Wal-Mart have used it to help determine the location of warehouses and assembly sites and to choose shipping routes. Although the FAF data provide unprecedented new insight into the national freight network, FAF is not now scalable down to the local level. FAF is focused on the national and regional aspects of freight movement. It does not capture movement less than 50 miles and was not designed to pro- vide a local perspective. The managers of the FAF program said that augmenting the FAF data for local granularity would be very data intensive and probably expensive. The FAF pro- gram managers say they do not anticipate scaling the data down to the local level. The FAF program incorporates data from the following data systems: Commodity Flow Survey: This is a domestic shipper survey conducted by the U.S. Census Bureau. It has origin/destina- tion data for manufacturing, mining, and agriculture sectors. It is conducted every five years. The last one was conducted in 2007. The survey seeks sample data from shippers randomly identified from federal tax files. Vehicle Inventory and Use Survey (VIUS): This survey was conducted by the U.S. Census Bureau. Last done in 2002, it collected information about trucks to be used to compute and calibrate tonnage for various products. The data will be analyzed and incorporated into FAF. Highway Performance Management System: These data are obtained from state DOTs that collect data from samples of roadways statistically selected annually. The data address information about the performance, use, and operating char- acteristics of U.S. highways. Vehicle Travel Information System (VTRIS): This annual update provides data about the number of trucks weighed, weight by vehicle type, and the classification of vehicles mov- ing on the U.S. highway system. This information is used for calibration of tonnage of freight moved. Transborder Surface Freight Data: This information gives North American trade data by commodity and mode. The data include imports and exports to and from Canada and Mexico. This is updated monthly and annually. Waterborne Domestic and Foreign Commerce: This is domestic information updated annually and foreign trade information updated monthly from USACE. Oil Pipeline: Oil movement data by multistate region are obtained from the Energy Information Administration. Air Traffic Statistics: Air traffic, tonnage, and revenue ton- mile data are obtained from carriers quarterly from the FAA.

54 The managers of the FAF program say their experience holds significant lessons for development of a freight per- formance measurement process. They acknowledge current uncertainties about roles and responsibilities and a lack of clarity about the role of federal, state, and local agencies in providing data. In several states the relationship between the state and the local agencies is contentious. The authority and responsibility are tied to the availability of funds and the agency controlling the funds. A lack of clarity on roles, cou- pled with shortage of funds and lack of publicly available data at various points of the network, makes it difficult to have an integrated approach to national freight performance mea- sures, they indicated. Fund shortages have led to the cancel- lation of funding for the Vehicle Inventory and Use Program (VIUS). FAF1 used private data that could not be shared with users looking for input and output data. This lack of publicly available data led to FAF2. The measures were derived from FAF1 modeling that could be accomplished by using data that could be made public. The FAF data can play a significant role in monitoring and evaluating the nation’s freight system. FAF provides informa- tion about the volume and value of freight flow in the United States, and it provides information about the network over which the freight moves, as shown in Figures 6.1 and 6.2. The snapshot of information it provides can be compared across years and across the network to provide information about the performance of freight movement, quantities moved, and revenue generated. It also provides information about speed, 7 Waterborne Domestic and Foreign Commerce: This is domestic information updated annually and f reign trade info mation updated monthly from USACE. Oil Pipeline: Oil movement data by multistate region are obtained from the Energy Information Administration. Air Traffic Statistics: Air traffic, tonnage, and revenue ton-mile data are obtained from carriers quarterly from the FAA. The managers of the FAF program say their experience holds significant lessons for development of a freight performance measurement process. They acknowledge current uncertainties about roles and responsibilities and a lack of clarity about the role of federal, state, and local agencies in providing data. In several states the relationship between the state and the local agencies is contentious. The authority and responsibility are tied to the availability of funds and the agency controlling the funds. A lack of clarity on roles, coupled with shortage of funds and lack of publicly available data at various points of the network, makes it difficult to have an integrated approach to national freight performance measures, they indicated. Fund shortages have led to the cancellation of funding for the Vehicle Inventory and Use Program (VIUS). FAF1 used private data that could not be shared with users looking for input and output data. This lack of publicly available data led to FAF2. The measures were derived Figure 6.1. FAF can be used to understand some aspects of freight movement such as relative volumes and destinations of freight flows from locations, in this case Missouri. Figure 6.2. FAF illustrates California's import volumes. 7 aterborne o estic and Foreign Co erce: This is do estic infor ation updated annually and foreign trade infor ation updated onthly fro S CE. il Pipeline: il ove ent data by ultistate region are obtained fro the Energy Infor ation d inistration. Air Traffic Statistics: ir traffic, tonnage, and revenue ton- ile data are obtained fro carriers quarterly fro the F . The anagers of the F F progra say their experience holds significant lessons for develop ent of a freight perfor ance easure ent process. They ackno ledge current uncertainties about roles and responsibilities and a lack of clarity about the role of federal, state, and local agencies in providing data. In several states the relationship bet een the state and the local agencies is contentious. The authority and responsibility are tied to the availability of funds and the agency controlling the funds. lack of clarity on roles, coupled ith shortage of funds and lack of publicly available data at various points of the net ork, akes it difficult to have an integrated approach to national freight perfor ance easures, they indicated. Fund shortages have led to the cancellation of funding for the ehicle Inventory and se Progra ( I S). F F1 used private data that could not be shared ith users looking for input and output data. This lack of publicly available data led to F F2. The easures ere derived Figure 6.1. FAF can be used to understand some aspects of freight movement such as relative volumes and destinations of freight flows from locations, in this case Missouri. Figure 6.2. FAF illustrates California's i port volu es. Figure 6.2. FAF illustration of California’s import volumes. Figure 6.1. Exampl of FAF data useful for assessing freight movem nt.

55 reliability, and congestion of movement of freight through the nation. It does not provide geographic or temporal gran- ularity. In other words, it is annualized data available at the state and national level, not the local level. Transportation services index The Transportation Services Index (TSI) was created by the USDOT Bureau of Transportation Statistics (BTS), and it measures the movement of freight and passengers nationally. The index, which is seasonally adjusted, combines available data on freight traffic, as well as passenger travel, that have been weighted to yield a monthly measure of transportation services output (Figure 6.3). The TSI is a monthly measure of the volume of services performed by the for-hire transportation sector. The index covers the activities of for-hire freight carriers, for-hire pas- senger carriers, and a combination of the two. The TSI has been active since 2002 but is still under development and is therefore experimental. It is being examined for refinements in data sources, methodologies, and interpretations. The TSI provides insight into how the output of trans- portation services has increased or decreased from month to month. The index can be examined together with other economic indicators to produce a better understanding of the current and future course of the economy. The movement of the index over time can be compared with other economic measures to understand the relationship of transportation to long-term economic changes. The managers of the TSI note that it is the broadest mea- sure of U.S. domestic transportation output. The project started with a grant from BTS in 2002 and was brought in- house that same year. The first official release of TSI occurred in March 2004. Initially the project had 22 staff and several consultants working on the project. Over the course of time the process was streamlined and staff resources were reduced to five federal employees and two contractors. The products delivered by TSI are: • Freight Index • Passenger Index • Combined (Total) Index The process of refining the data and integrating it to pro- vide the three different indexes involves many detailed steps. Those include: Data Gathering The BTS staff gather monthly data for each mode of trans- portation from a range of government and private sources (Table 6.1). Forecasting Some data series were not complete through December 2003, the ending date through which the original TSI was published. Therefore, staff needed to forecast the one or two missing months, using a statistical technique known as an autoregressed moving average. As production of the TSI continues, the need to forecast missing data will be reduced. However, it is not uncommon in indexes of this type for monthly data to be delayed because of reporting or other problems and for preliminary data to be substituted. Deseasonalizing Because the principal purpose of the index is to reflect monthly shifts in transportation services output and to analyze short-term trends, it is essential that it be adjusted for the normal seasonal changes that affect the transporta- tion sector. Transportation is highly seasonal, and without adjustment the index would not give an accurate picture Figure 6.3. TSI trends.Figure 6.2. FAF illustration of California’s import volumes. 9 Data Gathering The BTS staff gather monthly data for each mode of transportation from a range of government and private sources (Table 6.1). Forecasting Some data series were not complete through December 2003, the ending date through which the original TSI was published. Therefore, staff needed to forecast the one or two missing months, using a statistical technique known as an autoregressed moving average. As production of the TSI continues, the need to forecast missing data will be reduced. However, it is not uncommon in indexes of this type for monthly data to be delayed because of reporting or other problems and for preliminary data to be substituted. Deseasonalizing Because the principal purpose of the index is to reflect monthly shifts in transportation services output and to analyze short-term trends, it is essential that it be adjusted for the normal seasonal changes that affect the transportation sector. Transportation is highly seasonal, and without adjustment the index would not give an accurate picture of underlying changes in transportation output. BTS has therefore deseasonalized the data using standard statistical methodologies. Indexing While physical measures are gathered for each mode, ultimately for combination and analysis, the data from the different modes must be converted into an index. BTS uses 1996 as the base year and indexes by dividing the current monthly value by the average value for the 12 months of 1996. Weighting and Chaining The final step in creation of the index is combining the individual mode indexes into the three summary indexes: the freight index, the passenger index, and the overall, or combined, TSI. The weighting is based on the relative economic value added of each mode. Not all ton-miles are equivalent in their economic importance, nor are all passenger-miles. For example, the average price paid per ton-mile for freight T ran sp o rta tio n S erv ices In d ex 60 70 80 90 100 110 120 130 Ja n-9 0 No v-9 0 S e p-9 1 Ju l-9 2 M a y -9 3 M a r-9 4 Ja n-9 5 No v-9 5 Se p-9 6 Ju l-9 7 M a y -9 8 M a r-9 9 Ja n-0 0 No v-0 0 Se p-0 1 Ju l-0 2 M a y -0 3 M a r-0 4 Ja n-0 5 No v-0 5 S e p-0 6 Ju l-0 7 M a y 2 0 0 8P F re ight Index P as s enger Index Figure 6.3. TSI trends.

56 10 moved by rail is less than the average price paid per ton-mile for freight shipped by truck because of differences in factors such as haul length, shipment volumes, and resultant economies of scale. By using an economic measure for weighting, the TSI staff recognizes these differences and makes the index more valuable as a transportation measure that can be used together with other economic measures, such as GDP. Value added is used for consistency with other indicators that are used in relation to GDP, for example, industrial production. By using value added, rather than gross revenues, for each sector, they seek to avoid double counting inputs (i.e., diesel fuel) to the transportation sector. Table 6.1. TSI source data. MEASURE MODE SOURCE Freight TSI Trucking American Trucking Association Air BTS and Carrier Websites Rail Association of American Railroads Water US Army Corps of Engineers Pipeline Energy Information Administration Passenger TSI Air Bureau of Transportation Statistics and carrier websites Rail Federal Railroad Administration Transit American Public Transportation Association Source: US Department of Transportation, Research and Innovative Technology Administration, Bureau of Transportation Statistics Because value-added data is available from the Bureau of Economic Analysis on an annual basis only, weights are determined annually and applied throughout the year. Valued added reflects the volume of physical transportation as well as the value of that volume. Because they have already measured monthly changes in that volume, it is necessary to ensure that changes in volume are not double-counted in the process of adjusting the weights for the index. This is accomplished through a mathematical process called chaining, which follows standard methodologies established by the U.S. Census Bureau for similar indexes. The “For-Hire Only” freight data are collected for all five modes: trucking, air, rail, water, and pipeline. Passenger data include air, rail, and transit. As with the FAF data, the producers of the index do not have statistics on who uses the data or for what purposes. Anecdotally, they know the TSI is used by Wall Street as a general indicator of the economy. It Source: U.S. Department of Tra sp rtation, Research and Innovative Technology Adminis ration, Bureau of Transportation S atistics. table . tSI source data. of underlying changes in transportation output. BTS has therefore deseasonalized the data using standard statistical methodologies. Indexing While physical measures are gathered for each mode, ulti- mately for combination and analysis, the data from the dif- ferent modes must be converted into an index. BTS uses 1996 as the base year and indexes by divi ing the current monthly value by the average value for the 12 months of 1996. Weighting and Chaining The final step in creation of the index is combining the individual mode indexes into the three summary indexes: the freight index, the passenger index, and the overall, or com- bined, TSI. The weighting is based on the relative economic value added of each mode. Not all ton-miles are equivalent in their economic importance, nor are all passenger-miles. For example, the average price paid per ton-mile for freight moved by rail is less than the average price paid per ton-mile for freight shipped by truck because of differences in factors such as haul length, shipment volumes, and resultant econo- mies of scale. By using an economic measure for weighting, the TSI staff recognizes these differences and makes the index more valuable as a transportation measure that can be used together with other economic measures, such as GDP. Value added is used for consistency with other indicators that are used in relation to GDP, for example, industrial pro- duction. By using value added, rather than gross revenues, for each sector, they seek to avoid double counting inputs (i.e., diesel fuel) to the transportation sector. Because value-added data is available from the Bureau of Economic Analysis on an annual basis only, weights are deter- mined annually and applied throughout the year. Valued added reflects the volume of physical transportation as well as the value of that volume. Because they have already measured monthly changes in t at volume, it is necess ry to ensure that changes in volume are ot d uble-cou ted in the process of adjusting the weights for the index. This is accomplished through a mathematical process called chaining, which fol- lows standard methodologies established by the U.S. Census Bureau for similar indexes. The “For-Hire Only” freight data are collected for all five modes: trucking, air, rail, water, and pipeline. Passenger data include air, rail, and transit. As with the FAF data, the producers of the index do not have statistics on who uses the data or for what purposes. Anecdotally, they know the TSI is used by Wall Street as a general indicator of the economy. It is used to evaluate the performance of the transportation sector by stock analysts. It is used as a forecaster of the economy. Companies such as Global Insight use this information as a factor in their analysis to provide economic projection and forecasting information to clients such as GE and Wal-Mart. It is used by companies such as AllianceBernstein to provide research and information on investment related to services globally. This information is also published on the White House website.

57 TSI officials note that the index is not scalable down to the local level. The TSI was intended to be a national-level index. Scaling it down to the local level poses many difficulties, fore- most being availability of data. Trucking information at the local level is not available, nor is railroad freight information. There was a request from Fannie Mae for quarterly regional information. Given the current processes and sources of data collection, analysis, seasonal adjustments, and indexing and the weighting and chaining process involved in generating the TSI, there is no plan to scale the national index to a local or regional level. The systems and processes involved are detailed, often requiring manual manipulation of data and collecting of data from air carrier websites and revising the data for three months prior to making it available in a stable state. The TSI team has gone through significant streamlining of the process and data analysis, making it possible to generate the reports in a timely manner. The TSI staff report that the level of effort involved is signifi- cantly high. There are also some current uncertainties about roles and responsibilities. Even in its current state, reports are published as tentative for the last three months. After monitor- ing changes for a quarter, the earliest month is moved from pre- liminary to a final state and the latest monthly report is added in a preliminary state. In this way the current three months of data are always shown in a “preliminary” state. Trucking Monthly truck ton-mile data is not available through a federal agency, so the data are obtained from the American Trucking Association using a calculated truck tonnage index. When the official data become available the preliminary values are replaced. There is a small cost associated with pur- chase of these data. Air Aviation data are collected from the airline websites and the Office of Airline Information (OAI). Often times the data are not readily available from the OAI dataset. The data change frequently, and the TSI team have to be prepared to include the changes and to replace data as the data become officially available from the airlines. Rail The data are obtained from FRA and do not include data from Amtrak and the Alaskan Railway Corp. Commuter rail is included in transit. Tsi challenges and lessons learned Among the challenges that the TSI effort faces is the need for continuous effort to educate the management, the public, and other potential users on the value of the measures. The TSI team has a media person who is focused on educating and communicating the use and value of the TSI. The TSI experience also suggests that long-term fund- ing and the ability to recruit expertise will be necessary to establish a comprehensive freight performance measurement system. As noted, the TSI project started with a team of 22 people. After the initial start-up effort, the staff was reduced to five federal employees and two consultants. The TSI experience also illustrates that process and quality reviews are integral where data from varied sources have to be collected, analyzed, scrubbed, filtered, and then combined to create the index. Data availability has to be studied and various alternative sources of data need to be tapped. The TSI team notes that 50 percent of the data is lost through the process of data scrubbing, cleaning, and filtering prior to being included in the published TSI. Where possible, receiv- ing processed data from the source reduces some of the data scrubbing efforts. One such example of scrubbed data is the rail data received from FRA. Also, making sure that the required data will be available through the life of the mea- sure is important. Moreover, sometimes data is not available timely to complete all necessary tasks required to meet the tight windows of generating the monthly reports. At least one set of trade association data was only available, forcing a three-month lag for the TSI. Data Considerations for the Freight Report Card Based upon the findings of the literature, the case studies, and the interviews with stakeholders, the following data- quality considerations will need to be addressed in the devel- opment of a Freight System Report Card. use common definitions for common understanding In order for stakeholders to generate and to use the data needed to create a set of national freight performance mea- sures, there needs to be clarity regarding what each measure and each piece of data means. Clarity of definitions—not only for each measure, but also for the data that feeds each measure—will promote a common understanding of the data and the measure among all shareholders. This can be accomplished by defining the metadata, that is, data that describe data. There are many variations to the definition of metadata, but a common definition is one provided by Webopedia, which defines metadata as “Data that describes how and when and by whom a particular set of data was collected, and how the data are formatted.” The TRB Final Metadata table 6.1. tSI source data.

58 Working Group Report 2006 states one of the many values of metadata thus: Metadata provides information necessary for data to be under stood and interpreted by a wide range of users . . . metadata are particularly important when the data users are physically or administratively separated from the data producers. Metadata also reduce the workload associated with answering the same questions from different users about the origin, transformation, and character of the data.14 Metadata management is not an easy task, but it is essential when working with data from multiple sources and is easier to implement if formalized at the start of a project rather than enforced after the data has been pulled together from a vari- ety of different sources. Agencies have worked independent of each other for decades and each has its own data structures, naming conventions, and formats. In the past decade, with public agencies collaborating and conducting peer studies informally, they have moved toward similar understand- ing and definitions of data in many areas of transportation. However, there is much that is still needed. In some of the newer areas, such as Geographic Information Systems, there is much more standardization. In order for the performance measures for the freight transportation system to be success- ful, the metadata for the framework should be defined early in the process. ensure data Quality Data quality is the essential component that makes data valuable to users. This includes accuracy, consistency, timeli- ness, and completeness. Data quality, as defined by the British Columbia Government Information Resource Management Glossary, is “the state of completeness, validity, consistency, timeliness and accuracy that makes data appropriate for a specific use.” Data quality refers to how closely the data can portray the real phenomena. The quality of the data is what determines if a decision maker will rely on the data to make decisions. While the importance of having high-quality data is intui- tively clear, it takes considerable effort to ensure that the qual- ity of data is maintained. Because the quality of data will have a significant impact on decision making, a process will need to be implemented to systematically ensure quality checks of the data being used to populate the freight performance man- agement framework. draw on Multiple data sources Organizations developing and deploying new applica- tions routinely use available newer technologies, databases, and programming languages, in addition to those already in use. The result is a hybrid of databases and technologies within an organization; each often having varying standards, formats, and quality. Within an organization, even when the data are managed by one central group, the data often come from multiple databases that do not necessarily communicate with each other. This issue is magnified when data manage- ment is decentralized. The explosion of data leads to many challenges with sorting, selecting, and retrieving relevant data, including scrubbing, preprocessing, and integrating data from multiple sources. With data being stored in different systems that use different technologies and databases, the challenges of communicating between databases also has to be addressed. Therefore, dealing effectively with multiple sources of data becomes a major issue when working across agencies and crossing over to accessing data from the private sector. In measuring the performance of a multimodal freight system, formal mechanisms will need to be put in place to ensure that data derived from multiple sources or silos, cov- ering a range of technologies, systems, and databases, are adequately preprocessed and integrated prior to populating the framework. adopt data standards and Formats Data standards and formats play a very important role when integrating data from different sources. Some of these may seem very simple and conceptually easy to resolve, but in dealing with millions of records from multiple sources, the issues get compounded. All of these issues are resolvable, but the time required to address each of them adds to the time required for the overall analysis and preprocessing time. One simple example of data formats involves a freight cost of $5,000.50 recorded in several databases. It would be com- mon for one to store this information in a text format (five thousand dollars and 50 cents), another to save it in currency format but capture it as “Dollars 5000” and in yet another database to record it in a currency format, but with more detail, as “$5000.50.” Several detailed steps will have to be fol- lowed in this simple example to integrate cost information from these multiple sources. A data format for the final inte- grated data will first have to be established. Data from each source will then have to be processed for conversion to that final format before it can be integrated. The analysis and pre- processing needed for use of such data for establishing a per- formance management framework will be dependent on the number of sources, which could involve numerous public and private organizations. Appropriate attention will therefore be necessary to bring together data from private and public agencies covering the multiple modes, standards, and formats involved, to ensure that the data are preprocessed appropri- ately for conversion to the final format established for the performance measures framework.

59 address data integration As mentioned earlier, data integration is the process of the standardization of data definitions and data structures by using a common conceptual schema across a collection of data sources. Integrated data will be consistent and logically compatible in different systems or databases and can be used across time and users. Historically, data warehousing has been a technique suc- cessfully used by organizations to bring together data from multiple sources for reporting and decision making. The Ohio DOT, an agency that is advanced in the use of perfor- mance measures, has at least five different types of databases that use various programming languages, ranging from newer languages such as Java to older languages such as COBOL. It has successfully used data warehousing to bring together data from many different applications and many different data- bases to provide information to assist with decision making. The model used by the Virginia and Ohio DOTs, to create a data warehouse to provide information about performance of operations and assets, has been successful. The data ware- house approach also addresses the issue raised by Minnesota DOT about parochial systems and systems that duplicate data. In using a data warehouse, data can be extracted from different sources and the necessary logic can be applied to compute various statistical figures about performance of the measures (for example, percentage of time in a day that the traffic flow is below a specified service level). Alternatively, data may be summarized, integrated, or broken down and saved as more granular components. The granular informa- tion can then be used for the performance measure dash- board, decision support, or other reporting systems and to provide answers to ad hoc queries by users. Historical data required for trend analysis or computation of lagging and leading indicators of performance of measures can also be obtained from the data in the data warehouse. consider access Time The time taken to access information is important to the usability of any system, particularly one envisioned for high- way operation data as sought for this project. The informa- tion technology industry invests millions of dollars each year in researching user behavior to improve the user’s experience. If the goal is to make these performance measures available nationally for users to access for decision making, then one factor that needs to be considered for usability is the time taken from the moment a user commences an attempt to access the information to the moment when the user actually retrieves the information. As the volume of data in the sys- tem grows, the time taken to access the information will also increase. Long time periods to access information discourage users from using a system. The database design will have to take into consideration the access time and also design for both active and dormant data. The design should consider a tiered approach to data storage in which cheaper storage is used for less frequently used data, while frequently accessed data could be on high-performing disk storage. Backup and recovery processes should be formalized, tested, and imple- mented from the very beginning. plan for archived vs. real-Time data needs In addition to archived data, the measures for the perfor- mance of the freight transportation system could include real-time systems such as Intelligent Transportation Sys- tems (ITS). In the Freight Information Real-Time System for Transportation (FIRST), ITS was used by the Port Authority of New York and New Jersey from mid-2001 until December 2003 to provide real-time freight information.15 The Port of Vancouver has also successfully used ITS to improve freight movement. In both instances, data quality and availability of data were among the items listed that required attention for successful deployment of the systems. According to the USDOT, “ITS can facilitate the safe, efficient, secure, and seamless movement of freight. Applications being deployed provide for tracking of freight and carrier assets such as containers and chassis, and improve the efficiency of freight terminal processes, drayage operations, and international border crossings.” The architecture required to report summary data is dif- ferent from that required for real-time decisions or trend analysis. Any real-time or near real-time information that is required will need to consider additional factors such as data latency, frequency of refresh, and the frequency at which data need to be presented for each measure. Near real-time measures will require that data be captured, cleansed, and loaded in near real-time. plan for the sustainability of the Framework For continuity of decision making, it is important that any set of measures be sustained beyond the initial deployment and continue to provide timely and accurate information dur- ing the entire period of its use or life. The purpose of freight performance measures is to provide information to allow decision makers to make informed decisions and for users to see the performance of the freight transportation system not for the short term, but for several years. This can happen only if the framework is sustainable and available for the period of its intended use. Sustainability involves ensuring that timely, accurate data are available, that they can be easily accessed by

60 users without concerns of privacy, and the necessary infra- structure needed to support the data and framework is imple- mented and maintained for the life of the measures. continued research into additional data sources A consistent theme throughout this research has been how data limitations constrain expansion of freight performance measurement. The performance measures included in the Freight System Report Card are those that are possible given existing data sources. As noted in the Summary and in Chap- ter 7, Findings and Recommendations, further research into how to capture additional performance data—particularly related to multimodal freight efficiencies—is important. Bal- ancing the acquisition of such data with the cost and privacy of the private sector are among the most important of pos- sible future research areas. Endnotes 1 Donnelly, Rick (PB Consult, Inc.). A Framework for Development, prepared for the TRB Committee on Freight Transportation, 2003, pp. 6, 7. 2 Casavant, Ken, and Eric Jessup. Development of a Washington State Freight Data System, prepared for the Washington State DOT Office of Freight Stra- tegy and Policy, 2007. 3 Harrison, Rob, Mike Schofield, Lisa Loftus-Otway, Dan Middleton, and Jason West. Freight Performance Measures Guide, 2006. 4 Mani, Akshay, and Jolanda Prozzi. State-of-the-Practice in Freight Data: A Review of Available Freight Data in the U.S., Center for Transportation Research, The University of Texas at Austin, 2004. 5 SearchDataManagement.com, http://searchdatamanagement.techtarget. com/topics/0,295493,sid91_tax307458,00.html (accessed October 2, 2008). 6 Thomas, Gwen. “What’s Right for Your Organization.” June 18, 2008, http:// www.mainframezone.com/it-management/data- governance-structure- whats-right-for-your-organization/P5 (accessed Feb. 15, 2010). 7 FHWA Office of Asset Management. Data Integration Primer, 2001. 8 Data Integration Primer, 2001. 9 Spy Pond Partners, Comparative Performance Measurement: Pavement Smooth ness, prepared for AASHTO, 2008. 10 GAO. Motor Carrier Safety: A Statistical Approach Will Better Identify Com- mercial Carriers That Pose High Crash Risks Than Does the Current Federal Approach (GAO-07-585) Report to Congressional Requesters, June 2007. 11 GAO. Truck Safety: Motor Carriers Office Hampered by Limited Information on Causes of Crashes and Other Data Problems (GAO/RCED-99-182) Report to the Subcommittee on Transportation and Related Agencies, Committee on Appropriations, House of Representatives, June 1999. 12 GAO. Highway Safety: Further Opportunities Exist to Improve Data on Crashes Involving Commercial Motor Vehicles (GAO-06-102) Report to Con- gressional Committees, November 2005. 13 Transportation Research Circular E-C115: Challenges of Data Performance Measures: A Workshop. Transportation Research Board of the National Academies, Washington, D.C., 2006. http://onlinepubs.trb.org/onlinepubs/ circulars/ec115.pdf. 14 Transportation Research Board Metadata Working Group Report, Transpor- tation Metadata: Role of Data and information Technology Section, July 5, 2005. 15 FHWA. Freight Information Real-Time System for Transport, http://ops.fhwa. dot.gov/freight/documents/first.pdf (accessed May 25, 2010).

Next: Chapter 7 - Findings and Recommendations »
Performance Measures for Freight Transportation Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

TRB’s National Cooperative Freight Research Program (NCFRP) Report 10: Performance Measures for Freight Transportation explores a set of measures to gauge the performance of the freight transportation system.

The measures are presented in the form of a freight system report card, which reports information in three formats, each increasingly detailed, to serve the needs of a wide variety of users from decision makers at all levels to anyone interested in assessing the performance of the nation’s freight transportation system.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!