Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 50
50
CHAPTER 6
Data Considerations to Support
Performance Measurement
Summary ment of a national freight database and related data collection
and synthesis activities with the potential to meet users' data
Of the many challenges to developing a nationwide freight requirements.
performance measurement system, the greatest is the com-
plexity of gathering adequate data. It is self-evident that per- The report notes that many users' needs require freight
formance measurement relies on data and that the measure- data that are not available from any single source. Thus, it is
ment system can only be as sound as the data it consists of. frequently necessary to combine data from different sources.
As mentioned in earlier chapters, the availability of sound, The combination of data from different sources, often known
consistent, sustainable data was an overriding consideration as "data fusion," is frequently problematic. Much of the exist-
in the selection of measures for the first-generation Freight ing data were developed by different entities, over different
System Report Card. Although stakeholder interviews indi- times with different generations of technology. The sources
cated a desire for additional measures, the measures selected differ in their modal coverage, collection techniques, and data
for the report card were ones for which data are readily and definitions. Significant concerns were identified in Special
consistently available. Report 276 regarding the use of the existing data for a com-
Although measures were selected for which data exist, the prehensive national freight database:
ongoing population of the report card will represent an enor-
mous data challenge. This section examines the challenges of A further deficiency of existing sources of freight transporta-
tion data is that some of the information required by decision
freight system data collection that will need to be addressed. makers is simply not available. For example, informed efforts
It also includes two relevant case studies--one of the Freight to alleviate highway congestion require data on routes traveled,
Analysis Framework (FAF) and one of the Transportation time of day, and the types of trucks and commodities caught in
Services Index (TSI). Both are highly relevant in that they are congestion--data that are rarely collected, at least in the United
analogous efforts to integrate freight data from a wide array States. Both the committee's discussions with users and the per-
sonal experience of individual members revealed a sense of frus-
of sources into a common reporting format. Their experience
tration with existing freight data. The disjointed array of data
illustrates, on a smaller scale, the type of effort necessary to sources is cumbersome and difficult to use, lacking in geographic
develop a freight report card. detail, and notably deficient in covering increasingly important
motor carrier flows. Several users also expressed concern about
the unnecessary burden on data providers, who may be asked
Freight Data Issues to provide similar data to different organizations--sometimes
in different formats. This heavy respondent burden is likely to
State and federal practitioners have identified significant hinder efforts to gather quality data.
gaps in the freight data available for performance measure-
ment. In A Concept for a National Freight Data Program: Spe- A pending NCFRP Project 12 has been scoped to further
cial Report 2761 the shortcomings in federal freight data sets develop a national freight data architecture. Its objectives
were summarized. include developing the specifications for the content and
structure of a freight data architecture, to identify the value
[T]he current disjointed patchwork of freight data sources is
costly to generate and maintain but does not provide decision and challenges of the potential architecture, and to specify
makers with the data they require. To remedy this deficiency, a institutional strategies to develop and maintain the archi-
national freight data framework is needed to guide the develop- tecture. This architecture is intended to serve the needs of
OCR for page 51
51
public and private decision makers at the national, state, and has noted, "Data governance is a system of decision rights and
local levels. accountabilities for information-related processes, executed
A study conducted for the Washington State DOT2 iden- according to agreed-upon models that describe who can take
tified 32 different data sets that the state could include in what actions with what information, and when, under what
its freight data system. Despite the number of sets that can circumstances, using what methods."6
supply some data, the report noted that "very little system-
FHWA defines data integration as "The process of combin-
atic data exists to inform decision makers about the eco- ing or linking two or more data sets from different sources
nomic impact, system bottlenecks, and supply chains flowing to facilitate data sharing, promote effective data gathering
through freight systems that support Washington State pro- and analysis, and support overall information management
ducers and delivery of goods to consumers." activities in an organization."7
In Texas, the authors of a study on potential freight perfor- Following are some of the governance and integration
mance measures summarized the state of current freight data issues that will need to be addressed in deploying a dashboard
for performance measures thus: for freight-related performance measures.
Freight performance measures (FPM) in the U.S. are currently 1. The development of common data definitions for organi-
at a very early stage in their development. Some states have made
zations providing data to a national set of freight perfor-
a push to look into FPMs or to begin some data collection to
assess what would be required for an integrated ITS-PM system. mance measures;
However, most states have not yet utilized their performance 2. Development of data quality and accuracy standards;
measures across modes. The general consensus is that the imple- 3. Development of protocols to integrate multiple sources of
mentation of a comprehensive set of FPMs requires far more data into the framework;
data-collection capability than most states currently possess.3 4. Development of strategies to close data gaps;
5. Development of strategies to assure data availability;
The authors note that even the leading work that has been a.From sources;
done has focused on broad goals and objectives, rather than b.From the report card or sets of measures themselves;
specific performance metrics. Another study of data sources 6. Time to access data from the framework;
in Texas identified 31 separate databases that could be used 7. Identification of real-time versus archived data needs; and
for some aspect of freight system performance analysis.4 8. Sustainability of the data framework.
At least two major areas of data improvement will need to
be addressed to implement a freight performance measure- The Data Integration Primer8 notes, "The data integration
ment report card. First are the issues related to the integration process can be extremely involved and challenging, especially
and governance of state and federal transportation data, or for organizations that have a long history of stand-alone files
the processes by which data from different sources are syn- or rarely share data across database systems. " Although the
thesized and stored so that they can be analyzed by users and Data Integration Primer focuses only upon asset manage-
decision makers. The literature indicates that transportation ment, its underlying principles apply to broader types of data
data integration from a wide array of providers will present integration efforts. It notes that a careful analysis of organi-
significant technical, policy, and logistical challenges. zational needs should precede data integration efforts. Use
Second is the issue of the quality and quantity of freight- cases (analyses of the activities to be performed) and cus-
related data. The experience of other transportation agencies tomer requirements are clearly needed to ensure whether the
suggests that freight-related data has continually improved data integration effort meets the performance measurement
in recent years but still lacks the detail, breadth, and com- effort's needs. Also, a wide range of stakeholders and practi-
pleteness necessary for consistent, nationwide performance tioners should be involved to identify the different needs that
measurement. different users have for the performance measure data to be
eventually integrated.
Data Integration and Governance
Data governance and data integration will be essential ele-
Freight Data Quality and Quantity
ments of a freight performance measurement system. Data
governance has been defined as "the overall management of Many national transportation-related data systems that
the availability, usability, integrity, and security of the data could feed a freight performance system are generated by
employed in an enterprise. A sound data governance program data produced by the states. This is the case for such applica-
includes a governing body or council, a defined set of proce- tions as the Highway Performance Management System, the
dures, and a plan to execute those procedures."5 One author Fatality Analysis Reporting System, and the National Bridge
OCR for page 52
52
Inventory. As states have increasingly focused upon perfor- question, resulting in duplicative manual recreation of data.
mance measures and performance benchmarking with peer More importantly, users lose confidence in the data. Most
states, the deficiencies in their performance data have become performance measures are developed within the division
more apparent. Comparative Performance Measurement: supporting its respective performance measures and are not
Pavement Smoothness9 notes significant variation in how developed as part of an overall data collection program.
different state highway agencies gather a very basic piece of
performance data, the International Roughness Index (IRI) Alaska DOT has a data steward role that includes collection,
data for pavements. This variation occurs even though the quality control, transformation, documentation, archiving,
data are machine gathered, protocols exist to calibrate the and access of transportation data. Some of the issues that
machines, and standards exist to assess the data. The study the agency has to overcome are institutional, parochial, data
estimated that up to a 15 percent variation exists between stovepipes, technology changes, and evolving department
how states record the data. In addition, it found that wide business requirements.
variation exists as to how states could manipulate and extract
the data for comparative analysis. These variations occurred Minnesota DOT's representatives noted that there were
even though all states must record IRI and that considerable limited numbers of tools for policy, programs, and
effort has been expended to establish clear standards and pro- executive-level decision making. This is at least partly due
tocols to ensure national consistency. The study noted that to issues related to data quality, availability, systems integra-
factors such as the tire pressure of the test vehicle, speed of tion, and tools to retrieve data, analyze it, develop predic-
the vehicle, or the driver's strict adherence to the wheel path tive models, conduct trade-off analysis, and report results
all influenced whether the data were consistent, accurate, or in useful formats.
replicable between states.
The General Accounting Office10,11,12 noted the difficulty of Virginia DOT reported that the effort of creating a dash-
accurately assessing the number of truck-related fatal crashes board of performance measures was made simpler by having
because of inconsistent data reporting by the states. It noted a data warehouse. In the development of performance mea-
in 1999 that states failed to report 38 percent of all report- sures, the agency combined different kinds of data to produce
able crashes involving trucks and 30 percent of fatal crashes a single measure. The data warehouse provided that one stop
involving trucks. It attributed the data gaps to a lack of state for the different data used to automate the generation of the
laws compelling state and local officials to supply data to performance measures. Where data do not exist, the busi-
federal officials. In 2004 and 2007 follow-up studies, GAO ness requirements are formalized for data needed before any
reported that reporting had improved but still 21 percent of changes are made to existing systems or before new systems
crash reports lack complete data. It reported that timeliness are developed. For non-automated performance reports, data
in reporting had improved from 32 percent reported on time come from many sources, including spreadsheets, templates,
in 2000 to 89 percent in 2007. GAO noted that the FMCSA and e-mail. A lack of standardization in the number and defi-
spent $21 million in grants over three years to improve the nition of data fields collected has made statewide incident
data reporting practices of 34 states. management reporting difficult. The agency is in the process
The studies of IRI data and truck-crash data illustrate the of overhauling the system that tracks the operations.
complexities of using even one traditional performance mea-
sure for comparative analysis. State DOT data practitioners Washington DOT notes that there is consensus that the
describe complexities that are orders of magnitude greater agency needs better or more complete data. Based on a direc-
when they described integrating data across a number of dif- tive by the legislature, Washington DOT completed a study of
ferent legacy systems. The state's experiences were summa- 11 core technology systems. According to the study none of
rized in proceedings for the TRB Workshop on Challenges the 11 core systems met even 20 percent of the agency's cur-
of Data for Performance Measures, in 2006.13 Summaries of rent and future business and technical requirements. WSDOT
several states' observations of their own data challenges and is currently addressing the unmet needs through tremendous
needs were provided. manual effort and use of multiple ad hoc systems.
California DOT data are stored in myriad databases that Florida DOT notes that data intricacies in collection and
are disjointed and uncoordinated, have varying usability, storage can get lost in generalization of a large database. There
and are inconsistent or duplicated in other databases. Lane were challenges with keeping the data current and repeatable
miles in one database may include miles maintained that are and having consistent data and sources. The blended mea-
not state highways or could include proposed relinquish- sures may have data from various sources and new data need
ments. This confusion leads to different answers to the same to be addressed.
