Cover Image

Not for Sale

View/Hide Left Panel
Click for next page ( 51

The National Academies of Sciences, Engineering, and Medicine
500 Fifth St. N.W. | Washington, D.C. 20001

Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 50
50 CHAPTER 6 Data Considerations to Support Performance Measurement Summary ment of a national freight database and related data collection and synthesis activities with the potential to meet users' data Of the many challenges to developing a nationwide freight requirements. performance measurement system, the greatest is the com- plexity of gathering adequate data. It is self-evident that per- The report notes that many users' needs require freight formance measurement relies on data and that the measure- data that are not available from any single source. Thus, it is ment system can only be as sound as the data it consists of. frequently necessary to combine data from different sources. As mentioned in earlier chapters, the availability of sound, The combination of data from different sources, often known consistent, sustainable data was an overriding consideration as "data fusion," is frequently problematic. Much of the exist- in the selection of measures for the first-generation Freight ing data were developed by different entities, over different System Report Card. Although stakeholder interviews indi- times with different generations of technology. The sources cated a desire for additional measures, the measures selected differ in their modal coverage, collection techniques, and data for the report card were ones for which data are readily and definitions. Significant concerns were identified in Special consistently available. Report 276 regarding the use of the existing data for a com- Although measures were selected for which data exist, the prehensive national freight database: ongoing population of the report card will represent an enor- mous data challenge. This section examines the challenges of A further deficiency of existing sources of freight transporta- tion data is that some of the information required by decision freight system data collection that will need to be addressed. makers is simply not available. For example, informed efforts It also includes two relevant case studies--one of the Freight to alleviate highway congestion require data on routes traveled, Analysis Framework (FAF) and one of the Transportation time of day, and the types of trucks and commodities caught in Services Index (TSI). Both are highly relevant in that they are congestion--data that are rarely collected, at least in the United analogous efforts to integrate freight data from a wide array States. Both the committee's discussions with users and the per- sonal experience of individual members revealed a sense of frus- of sources into a common reporting format. Their experience tration with existing freight data. The disjointed array of data illustrates, on a smaller scale, the type of effort necessary to sources is cumbersome and difficult to use, lacking in geographic develop a freight report card. detail, and notably deficient in covering increasingly important motor carrier flows. Several users also expressed concern about the unnecessary burden on data providers, who may be asked Freight Data Issues to provide similar data to different organizations--sometimes in different formats. This heavy respondent burden is likely to State and federal practitioners have identified significant hinder efforts to gather quality data. gaps in the freight data available for performance measure- ment. In A Concept for a National Freight Data Program: Spe- A pending NCFRP Project 12 has been scoped to further cial Report 2761 the shortcomings in federal freight data sets develop a national freight data architecture. Its objectives were summarized. include developing the specifications for the content and structure of a freight data architecture, to identify the value [T]he current disjointed patchwork of freight data sources is costly to generate and maintain but does not provide decision and challenges of the potential architecture, and to specify makers with the data they require. To remedy this deficiency, a institutional strategies to develop and maintain the archi- national freight data framework is needed to guide the develop- tecture. This architecture is intended to serve the needs of

OCR for page 50
51 public and private decision makers at the national, state, and has noted, "Data governance is a system of decision rights and local levels. accountabilities for information-related processes, executed A study conducted for the Washington State DOT2 iden- according to agreed-upon models that describe who can take tified 32 different data sets that the state could include in what actions with what information, and when, under what its freight data system. Despite the number of sets that can circumstances, using what methods."6 supply some data, the report noted that "very little system- FHWA defines data integration as "The process of combin- atic data exists to inform decision makers about the eco- ing or linking two or more data sets from different sources nomic impact, system bottlenecks, and supply chains flowing to facilitate data sharing, promote effective data gathering through freight systems that support Washington State pro- and analysis, and support overall information management ducers and delivery of goods to consumers." activities in an organization."7 In Texas, the authors of a study on potential freight perfor- Following are some of the governance and integration mance measures summarized the state of current freight data issues that will need to be addressed in deploying a dashboard for performance measures thus: for freight-related performance measures. Freight performance measures (FPM) in the U.S. are currently 1. The development of common data definitions for organi- at a very early stage in their development. Some states have made zations providing data to a national set of freight perfor- a push to look into FPMs or to begin some data collection to assess what would be required for an integrated ITS-PM system. mance measures; However, most states have not yet utilized their performance 2. Development of data quality and accuracy standards; measures across modes. The general consensus is that the imple- 3. Development of protocols to integrate multiple sources of mentation of a comprehensive set of FPMs requires far more data into the framework; data-collection capability than most states currently possess.3 4. Development of strategies to close data gaps; 5. Development of strategies to assure data availability; The authors note that even the leading work that has been a.From sources; done has focused on broad goals and objectives, rather than b.From the report card or sets of measures themselves; specific performance metrics. Another study of data sources 6. Time to access data from the framework; in Texas identified 31 separate databases that could be used 7. Identification of real-time versus archived data needs; and for some aspect of freight system performance analysis.4 8. Sustainability of the data framework. At least two major areas of data improvement will need to be addressed to implement a freight performance measure- The Data Integration Primer8 notes, "The data integration ment report card. First are the issues related to the integration process can be extremely involved and challenging, especially and governance of state and federal transportation data, or for organizations that have a long history of stand-alone files the processes by which data from different sources are syn- or rarely share data across database systems. " Although the thesized and stored so that they can be analyzed by users and Data Integration Primer focuses only upon asset manage- decision makers. The literature indicates that transportation ment, its underlying principles apply to broader types of data data integration from a wide array of providers will present integration efforts. It notes that a careful analysis of organi- significant technical, policy, and logistical challenges. zational needs should precede data integration efforts. Use Second is the issue of the quality and quantity of freight- cases (analyses of the activities to be performed) and cus- related data. The experience of other transportation agencies tomer requirements are clearly needed to ensure whether the suggests that freight-related data has continually improved data integration effort meets the performance measurement in recent years but still lacks the detail, breadth, and com- effort's needs. Also, a wide range of stakeholders and practi- pleteness necessary for consistent, nationwide performance tioners should be involved to identify the different needs that measurement. different users have for the performance measure data to be eventually integrated. Data Integration and Governance Data governance and data integration will be essential ele- Freight Data Quality and Quantity ments of a freight performance measurement system. Data governance has been defined as "the overall management of Many national transportation-related data systems that the availability, usability, integrity, and security of the data could feed a freight performance system are generated by employed in an enterprise. A sound data governance program data produced by the states. This is the case for such applica- includes a governing body or council, a defined set of proce- tions as the Highway Performance Management System, the dures, and a plan to execute those procedures."5 One author Fatality Analysis Reporting System, and the National Bridge

OCR for page 50
52 Inventory. As states have increasingly focused upon perfor- question, resulting in duplicative manual recreation of data. mance measures and performance benchmarking with peer More importantly, users lose confidence in the data. Most states, the deficiencies in their performance data have become performance measures are developed within the division more apparent. Comparative Performance Measurement: supporting its respective performance measures and are not Pavement Smoothness9 notes significant variation in how developed as part of an overall data collection program. different state highway agencies gather a very basic piece of performance data, the International Roughness Index (IRI) Alaska DOT has a data steward role that includes collection, data for pavements. This variation occurs even though the quality control, transformation, documentation, archiving, data are machine gathered, protocols exist to calibrate the and access of transportation data. Some of the issues that machines, and standards exist to assess the data. The study the agency has to overcome are institutional, parochial, data estimated that up to a 15 percent variation exists between stovepipes, technology changes, and evolving department how states record the data. In addition, it found that wide business requirements. variation exists as to how states could manipulate and extract the data for comparative analysis. These variations occurred Minnesota DOT's representatives noted that there were even though all states must record IRI and that considerable limited numbers of tools for policy, programs, and effort has been expended to establish clear standards and pro- executive-level decision making. This is at least partly due tocols to ensure national consistency. The study noted that to issues related to data quality, availability, systems integra- factors such as the tire pressure of the test vehicle, speed of tion, and tools to retrieve data, analyze it, develop predic- the vehicle, or the driver's strict adherence to the wheel path tive models, conduct trade-off analysis, and report results all influenced whether the data were consistent, accurate, or in useful formats. replicable between states. The General Accounting Office10,11,12 noted the difficulty of Virginia DOT reported that the effort of creating a dash- accurately assessing the number of truck-related fatal crashes board of performance measures was made simpler by having because of inconsistent data reporting by the states. It noted a data warehouse. In the development of performance mea- in 1999 that states failed to report 38 percent of all report- sures, the agency combined different kinds of data to produce able crashes involving trucks and 30 percent of fatal crashes a single measure. The data warehouse provided that one stop involving trucks. It attributed the data gaps to a lack of state for the different data used to automate the generation of the laws compelling state and local officials to supply data to performance measures. Where data do not exist, the busi- federal officials. In 2004 and 2007 follow-up studies, GAO ness requirements are formalized for data needed before any reported that reporting had improved but still 21 percent of changes are made to existing systems or before new systems crash reports lack complete data. It reported that timeliness are developed. For non-automated performance reports, data in reporting had improved from 32 percent reported on time come from many sources, including spreadsheets, templates, in 2000 to 89 percent in 2007. GAO noted that the FMCSA and e-mail. A lack of standardization in the number and defi- spent $21 million in grants over three years to improve the nition of data fields collected has made statewide incident data reporting practices of 34 states. management reporting difficult. The agency is in the process The studies of IRI data and truck-crash data illustrate the of overhauling the system that tracks the operations. complexities of using even one traditional performance mea- sure for comparative analysis. State DOT data practitioners Washington DOT notes that there is consensus that the describe complexities that are orders of magnitude greater agency needs better or more complete data. Based on a direc- when they described integrating data across a number of dif- tive by the legislature, Washington DOT completed a study of ferent legacy systems. The state's experiences were summa- 11 core technology systems. According to the study none of rized in proceedings for the TRB Workshop on Challenges the 11 core systems met even 20 percent of the agency's cur- of Data for Performance Measures, in 2006.13 Summaries of rent and future business and technical requirements. WSDOT several states' observations of their own data challenges and is currently addressing the unmet needs through tremendous needs were provided. manual effort and use of multiple ad hoc systems. California DOT data are stored in myriad databases that Florida DOT notes that data intricacies in collection and are disjointed and uncoordinated, have varying usability, storage can get lost in generalization of a large database. There and are inconsistent or duplicated in other databases. Lane were challenges with keeping the data current and repeatable miles in one database may include miles maintained that are and having consistent data and sources. The blended mea- not state highways or could include proposed relinquish- sures may have data from various sources and new data need ments. This confusion leads to different answers to the same to be addressed.