Guidance for Developing a Freight Transportation Data Architecture (2010)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

S U M M A R Y Introduction The movement of freight in the United States continues to grow, causing congestion along corridors and at network nodes such as seaports, land ports of entry, truck and rail terminals, and airports. It is critical to have accurate, comprehensive, and timely information about freight movements and the impact of these movements on the transportation network in order to make sound investment decisions to improve and optimize the freight transportation system. This report documents the results of a study to develop specifications for content and structure of a national freight data architecture that serves the needs of public and private decisionmakers at the national, state, regional, and local levels. It is worth noting that the purpose of NCFRP Project 12 was to develop requirements and specifications for a national freight data architecture, not to develop the data architecture (which would be a logical next step after identifying those requirements and specifications). The research team undertook the following activities to address these research needs: • Completed a review of systems, databases, and architectures that might be used as a poten- tial reference for the development of a national freight data architecture; • Conducted surveys and follow-up interviews, interviews with subject matter experts, and a peer exchange with freight transportation stakeholders; • Developed a formal definition for a national freight data architecture; • Identified high-level categories of data architecture components; • Identified potential implementation approaches; • Developed a list of specifications for a national freight data architecture; and • Identified challenges and strategies related to the implementation of a national freight data architecture. Data Sources, Systems, and Architectures Various listings, links, and summaries of systems, databases, architectures, and other related documents that pertain to freight transportation data are available in the literature. Although there is a wealth of sources of information that pertain to freight transportation, a comprehen- sive catalog of freight-related data sources at different geographic levels (including national, state, regional, and local levels) does not exist. As a reference, the research team conducted a review of a sample of freight-related data sources at the national level to complement or other- wise extend existing listings. This sample is obviously not comprehensive. For example, it does not reference datasets that state, regional, and local entities need to collect to supplement or augment national-level datasets. Although the sample of data sources evaluated does not Guidance for Developing a Freight Transportation Data Architecture 1

2include all the potential data sources that deal with freight transportation, it is useful because it provides a sample of the typical national-level data sources that would need to be included in a national freight data architecture. A few systems and architectures were of particular inter- est because of the lessons that could be derived from the processes that led to their develop- ment. The analysis included topics such as purpose, content, institutional arrangements used for developing and maintaining the system or architecture; challenges and issues faced in cre- ating and maintaining the architecture or system; strategies and methods for dealing with data integration issues; and adaptability to serve evolving purposes and data sources. Online Surveys, Interviews, and Peer Exchange The research team conducted a planner and analyst survey, a shipper survey, and a motor carrier survey (as well as follow-up interviews) to gather information about freight data uses and needs. The research team also conducted interviews with subject matter experts to address specific items of interest to the research. The purpose of the planner and analyst sur- vey was to gather information from government planners, analysts, and other similar freight-related stakeholders. Respondents were involved in all modes of transportation, including air, rail, truck, pipelines, and water. Respondents indicated that they use freight data to support the production of a wide range of public-sector transportation planning doc- uments, adding weight to the notion that the national freight data architecture should sup- port a variety of freight-related processes. Respondents reported using and/or needing data at various levels of geographic coverage and resolution. The feedback on unmet data needs complement similar findings in the literature. The purpose of the shipper survey was to gather general information from the shipper community regarding freight data uses and needs, as well as willingness to share data with other freight-related stakeholders. Feedback from respondents indicates that the shipper industry collects large amounts of data. Many shippers and logistics service providers trans- mit data electronically using electronic data interchange (EDI) technologies. However, accessing data from shippers and logistics service providers for transportation planning appli- cations, beyond aggregated data from commercial data providers and national survey cam- paigns such as the Commodity Flow Survey (CFS), is not necessarily straightforward. For example, while a data record might characterize a commodity as well as origin and destina- tion locations, the route data component may be missing unless the carrier movement data are included. In addition, the shipper stakeholders interviewed indicated they could not com- ment on their companies’ ability or willingness to share data for freight transportation plan- ning purposes (particularly on a load-by-load basis, given its proprietary and confidential nature). Subsequent feedback obtained at the peer exchange (see below) highlighted a num- ber of strategies to address this issue, including initiating discussions about data sharing at a sufficiently high administrative level—since low-ranking personnel might know the data, but frequently do not have the authority or permission to discuss data sharing options. Involv- ing trade associations rather than individual firms might also be beneficial. The purpose of the motor carrier survey was to gather information from the motor carrier community about freight data uses and needs, as well as willingness to share data with exter- nal freight-related stakeholders. Carriers handle large amounts of disaggregated data during the course of their business operations. Increasingly, carriers use EDI standards and applica- tions. However, the amount of shipment information detail varies according to the type of car- rier. For example, truckload (TL) carriers, who tend to bill customers on a per-mile basis or by using a flat rate, rarely collect detailed commodity data. In addition, TL carriers are less likely to collect data on tonnage hauled or tare-level data. By comparison, less-than-truckload (LTL) carriers typically bill customers using a rate structure based on shipment weight, origin,

destination, and freight classification. However, LTL carriers tend to favor a freight-all-kinds rating structure that assigns a general freight classification to all shipments regardless of freight commodity or type. As opposed to TL carriers, LTL carriers are more likely to track total ton- nage. Motor carrier reservations about sharing proprietary and confidential data were related to the need to develop mechanisms to protect proprietary and confidential information and to maintain the anonymity of carriers and customers. In general, carriers would need to know in advance the specific uses of the data and, in return, would expect information in the form of industry benchmarking metrics. It is worth noting that developing metrics of interest to the private sector is part of the scope of work of NCFRP Project 3 “Performance Measures for Freight Transportation.” In conjunction with the 2009 North American Freight Flows Conference held in Irvine, CA, the research team organized a peer exchange to discuss preliminary research findings; request feedback; and facilitate a dialogue on implementation strategies to develop, adopt, and maintain a national freight data architecture. Participants included representatives of federal, state, regional, university, and private-sector agencies. To encourage participation and discussion, attendees received background materials such as relevant research topic sum- maries and breakout group agendas and discussion objectives. Feedback from peer exchange participants included recommendations for changes to initial research findings as well as a list of issues, challenges, and strategies to consider during the implementation of the national freight data architecture. National Freight Data Architecture Definition Taking into consideration the results of the literature review, as well as feedback from sur- veys, follow-up interviews, and the peer exchange, the research team developed the following generic definition for a national freight data architecture: The national freight data architecture is the manner in which data elements are organized and inte- grated for freight transportation-related applications or business processes. The data architecture includes the necessary set of tools that describe related functions or roles, components where those roles reside or apply, and data flows that connect roles and components at different domain and aggregation levels. Depending on the specific level of implementation chosen for the data architecture, this generic definition could be fine-tuned as follows: • Single-application approach. In this case, the national freight data architecture would become the manner in which data elements are organized and integrated for a specific freight application or business process at the national level (e.g., commodity flows). • Intermediate approaches (depending on the number of applications). In this case, the national freight data architecture would become the manner in which data elements are organized and integrated for a specific set of applications at the national, state, regional, and local levels. A large number of intermediate approaches is possible, depending on the business processes and geographic levels involved. For example, an intermediate implementation approach could include commodity flows at the national, state, and regional levels. Another, more encompassing, intermediate implementation approach could include commodity flows, safety, and pavement impacts at the national, state, regional, and local levels. • Holistic, all-encompassing approach. In this case, the national freight data architecture would become the manner in which data elements are organized and integrated for all freight transportation-related applications or business processes at the national, state, regional, and local levels. 3

4For any of these implementation options, the data architecture would include the necessary set of tools that describe related functions or roles, components where those roles reside or apply, and data flows that connect roles and components. National Freight Data Architecture Value From the documentation and information gathered during the research, the research team identified the following list of benefits that, together, provide a statement of value for the national freight data architecture: • Better understanding of the different business processes that affect freight transportation at different levels of coverage and resolution; • Better understanding of the supply chain, which should help transportation planners to identify strategies for improving freight transportation infrastructure; • Better understanding of the role that different public-sector and private-sector stakeholders play on freight transportation; • Better understanding of the need for standards to assist in data exchange; • Systematic, coordinated development of reference datasets (e.g., comprehensive commod- ity code crosswalk tables); • Systematic inventory of freight transportation data sources; • Systematic inventory of user and data needs that are prerequisites for the development of freight data management systems; • Use as a reference for the identification of locations where there may be freight data redun- dancy and inefficiencies; • Use as a reference for requesting funding allocations in the public and private sectors; and • Use as a reference for the development of outreach, professional development, and training materials. In practice, the value of the national freight data architecture is also a function of the costs associated with its implementation. Quantifiable data about expected benefits and costs are currently not available (benefit-cost analyses need to occur both at the beginning and at dif- ferent phases of implementation of the national freight data architecture). However, it is clear from the documentation and information gathered during the research that the “do-nothing” alternative (i.e., not implementing the national freight data architecture) is costly, ineffective, and unsustainable. Therefore, the research team’s recommendation is to pursue the national freight data architecture following a scalable implementation path in which the national freight data architecture starts with one application at one or two levels of decisionmaking and then adds applications and levels of decisionmaking as needed or according to a predetermined implementation plan until, eventually, reaching the maximum net value. National Freight Data Architecture Components The research team identified the following categories of components to include in the national freight data architecture: • Physical transportation components, • Cargo or freight, • Freight functions or roles, • Business processes, • Data sources,

5• Freight-related data, • Freight data models, • Freight data standards, and • User interface and supporting documentation. Figure 1 shows a high-level modular conceptualization and lists different categories of com- ponents. The diagram recognizes the scalable nature of the national freight data architecture and enables the production of various diagram versions (as well as tabular representations) depend- ing on what implementation level to pursue. For example, for a single-application data architec- ture that only focuses on commodity flows at the national level, it may not be necessary to depict (at least not in detail) other freight functions and business processes. Similarly, not all data standards would need to be considered, and the requirements for user interfaces to support that data architecture would be relatively minor. The diagram in Figure 1 is only one example of potentially many different types of diagrams that can be used to depict interactions among freight transportation components. National Freight Data Architecture Recommendations and Specifications In addition to the list of categories and components, the research team put together a list of recommendations for the development and implementation of the national freight data architecture. For convenience, the recommendations are written in the form of specifica- tions to guide and monitor the implementation of the data architecture as follows: • Adopt a definition for the national freight data architecture that is generic, scalable, and understood and accepted by the freight transportation community (see proposed definition above); • Compare candidate data architecture concepts; • Develop implementation plan for national freight data architecture components; • Develop lists of components to include in the national freight data architecture; • Develop and implement protocols for continuous stakeholder participation; • Conduct data gap analysis; • Conduct data disaggregation need analysis; • Assume a distributed approach (as opposed to a centralized approach) to freight data repository implementations; • Use a systems engineering approach for developing the national freight data architecture; • Use standard information technology tools and procedures; • Develop and/or use standardized terminology and definitions for each data architecture component developed; • Implement strong privacy protection strategies; and • Establish integration points with other data architectures and standards. Readers should note that the list of specifications is preliminary and might need refine- ment during the process of building the data architecture. Challenges and Strategies The research team identified relevant issues and challenges that might block the implemen- tation of the national freight data architecture as well as candidate strategies for developing, adopting, and maintaining the data architecture. The challenges were in the following cate- gories: technical, policy, economic/financial, and stakeholder buy-in and consensus.

Tuesday , June 02 , 200 9 Page 1 Data Model Physical Transportation Components Cargo or Freight Freight Functions or Role s Fixe d i nfrastructur e m anager or operator Regulator Policymaker Researcher Shipper or receiver Third-party logistics or broker Business Processe s Congestion managemen t International trade Logistics management On-board security monitoring Planning and forecasting Roadside safety inspection Routing and dispatching Safety analysi s Transportation infrastructur e analysis, design, and construction Transportation operations Workforce development and training Dat a S ources Administrative record s by laws an d r egulation s -sector data -sector data Freight Dat a M odels Freight Data Standard s Commodity and produc t classification standards CP C HMIS HS NAPC S NMFC NS T 2 00 7 PL U SCTG STCC Industrial classification standard s ISIC NAIC S SI C SITC at a e xchange standards ANSI ASC X1 2 s tandards UN/EDIFACT standards OASI S U BL standard s FIP S P UB 161-2 National ITS standards FGDC-sponsore d s tandards (includin g m etadata) Other standards ITD S S DS MET S Vehicle classification standards User Interface an d S upporting Documentation -based information clearinghous e Freight-Related Dat a Freight-Related Dat a and condition , s peed, and delay dat a routing dat a Notes: 1) Categories and component s a re provided for illustration purposes, are not exhaustive, and may be subject to change. 2) Not all categories and components apply to al l f reight-relate d busines s p rocesses. Figure 1. National freight data architecture framework and components.

• Technical challenges. Technical challenges refer to issues (e.g., technological limita- tions, hardware and software incompatibilities, and standards incompatibilities) that might impede the successful implementation of the data architecture. Examples include the following: – Feasibility of different implementation approaches; – Data storage requirements; – Feasibility of updated data entry protocols to eliminate data redundancies and support standardized data entry procedures; – Conversion of commodity code classifications; – Data life cycle and usefulness to support the decisionmaking process by public and pri- vate stakeholders; – Variability in data quality control practices, which affects data accuracy, completeness, and timeliness; – Differences in terminology, data item definitions, and data implementations among freight data stakeholders; – Prioritization of data architecture components; – Integration between shipper and carrier data to characterize commodity movements properly; and – Data confidentiality and security concerns. • Policy challenges. The national freight data architecture might fail if required policies, both in the public and private sectors, fail or are not feasible. Examples of policy chal- lenges include the following: – Homeland security concerns, which might limit the dissemination of certain freight- related data; – Impact on current private-sector data collection initiatives; and – Competitive and proprietary (privacy) concerns with the concept of public-sector agencies having access to private-sector data. • Economic and financial challenges. The national freight data architecture might fail if the perceived costs associated with its implementation exceed the benefits that stakehold- ers would receive. Examples of economic and financial challenges include the following: – Cost of data collection, storage, and quality assurance; – Benefits and costs related to data disaggregation requirements for different business processes; – Data life cycle and usefulness to support the decisionmaking process by public and pri- vate stakeholders; – Cost to acquire private-sector data; and – Cost to implement robust data confidentiality and data security measures. • Stakeholder buy-in and consensus. The national freight data architecture might fail if there is no stakeholder buy-in or consensus about the potential benefits that could result from implementing the data architecture. Examples of related issues include the following: – Reluctance of stakeholders to participate if there is no clarity regarding justification and anticipated benefits; – Confidentiality clauses in supply chain contracts, which might impede data sharing for transportation planning purposes; – Perception that data collected as part of a national freight data collection program could validate projects of national significance at the expense of small or rural communities; – Ability of carriers to provide data about loads they move; – Risk of low stakeholder participation, which could decrease data reliability; and – Adequacy of data standards. 7

8Strategies to ensure a successful implementation of the national freight data architecture include the following: • Implementation levels – Develop and compare candidate data architecture concepts, – Identify business process and implementation level priorities, – Develop high-quality data architecture concepts and applications that address the needs of the highest priority items first, and – Identify data needs at the finest disaggregation level and implement data collection and data storage plans at that level. • Relationships with leaders, champions, and stakeholders – Identify data architecture leaders and champions; – Engage the national freight data architecture champions early; – Maintain good communication channels with the various stakeholders during all phases of the development and implementation of the national freight transportation data architecture; – Identify funding mechanisms for the implementation of the data architecture; – Develop brochures, presentations, and other materials that explain the national freight data architecture, its scope, high-level components, and what it expects to accomplish; – Deliver effective messages on how the national freight data architecture will assist stake- holders in the identification of strategies to address a variety of freight-related issues ranging from data collection to analysis and reporting; – Deliver messages that provide clear, concise answers to the various challenges high- lighted in the previous section; – Articulate benefits of participation by the private sector; and – Identify opportunities for partnerships with the private sector (e.g., through public- private partnerships) to make data accessible for transportation planning purposes in a cost-effective manner. • Performance measures and effectiveness – Develop criteria for measuring effectiveness in the implementation of the national freight data architecture, – Identify major progress milestones, – Tie the implementation of the national freight data architecture to the development of metrics or performance measures that could benefit the entire freight transportation community, and – Accelerate the implementation of programs such as EFM and the freight performance measurement program. • Lessons learned from the implementation and maintenance of existing freight-related systems and architectures – Develop systems that are relevant to stakeholders, include adequate stakeholder par- ticipation, and provide incentives to encourage participation—particularly in the case of state and local entities; – Clearly define expected outcomes and development and coordination plan; – Articulate programs well; provide clear, uniform guidance; and provide good docu- mentation; – Develop applications that rely on widely used data standards; – Develop and compare candidate architecture concepts; – Consider federal legislation to support and develop the program; – Develop tools to measure benefits and costs early;

– Integrate archived data needs into frameworks and architectures early and develop data programs that use industry standards; – Implement interagency data exchange programs with centralized data coordination; – Use available data sources and develop long-term plans while keeping systems flexible to respond to changes and new data sources; – Schedule major and regular revisions effectively while avoiding scope creep; – Develop systems that are consistent with input data limitations; – Develop applications with backward compatibility; – Evaluate data disaggregation level requirements to ensure statistical significance; – Provide adequate resources for data collection, fully understand the implications of small sample sizes, and continue to involve the U.S. Census Bureau for the use of sur- vey instruments; – Emphasize data access, quality, reliability, confidentiality, and integrity; – Participate in the standards development process; – Create crosswalks to ensure compatibility of survey data internally over time and exter- nally across other datasets; – Involve stakeholders early and often through various mechanisms and technologies; and – Develop and implement professional capacity and training programs early. One of the strategies for implementation mentioned is to develop and compare candidate data architecture concepts. Peer exchange participants highlighted that implementing a comprehen- sive data architecture at once with no testing of options prior to making a decision about the cor- rect approach would be too risky. Participants also favored the concept of developing and com- paring several alternative approaches. A recommendation from peer exchange participants was to use NCFRP as an avenue for funding the development of alternative data architecture concepts. Participants indicated that the request for proposals should outline clear objectives while leaving the definition of approaches to the research team(s) selected. An idea discussed was to develop the data archi- tecture around scenarios or themes, such as business areas or processes, levels of govern- ment, or economic activity. Activities in connection with each scenario or theme would include structuring a competition for research teams (each of which would include a uni- versity partner, a private-sector partner, and a government-level partner) to develop and test competing data architecture concepts, making sure to include multimodal components in the scenarios and tests, and conduct a follow-up evaluation. 9