Guidance for Developing a Freight Transportation Data Architecture (2010)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

76 Introduction The overarching theme behind NCFRP Project 12 was the need for accurate, comprehensive, timely freight transporta- tion data at different levels, as well as the need for a holistic approach to freight transportation data. More specifically, NCFRP Project 12 was designed to identify specifications for a national freight data architecture that facilitates freight-related statistical and economic analyses; supports the decisionmaking process by public and private stakeholders at the national, state, regional, and local levels; and enables the acquisition and maintenance of critical data needed to identify freight-related transportation needs. Specific NCFRP Project 12 objectives included the following: • Develop specifications for content and structure of a national freight data architecture that serves the needs of public and private decisionmakers at the national, state, regional, and local levels; • Identify the value and challenges of the potential data architecture; and • Specify institutional strategies to develop and maintain the data architecture. The research team undertook the following activities to address these research needs: • Completed a review of systems, databases, and architectures that might be used as a potential reference for the develop- ment of a national freight data architecture; • Conducted surveys and follow-up interviews, interviews with subject matter experts, and a peer exchange with freight transportation stakeholders; • Developed a formal definition for a national freight data architecture; • Identified high-level categories of data architecture com- ponents; • Identified potential implementation approaches; • Developed a list of specifications for a national freight data architecture; and • Identified challenges and strategies related to the implemen- tation of a national freight data architecture. It is worth noting that the purpose of NCFRP Project 12 was to develop requirements and specifications for a national freight data architecture, not to develop the data architecture (which would be a logical next step after identifying those requirements and specifications). Data Sources, Systems, and Architectures A variety of listings, links, and summaries of systems, data- bases, architectures, and other related documents that pertain to freight transportation data are available in the literature. Although there is a wealth of sources of information that pertain to freight transportation, a comprehensive catalog of freight-related data sources at different geographic levels (including national, state, regional, and local levels) does not exist. As a reference, the research team conducted a review of a sample of freight-related data sources at the national level to complement or otherwise extend existing listings. Previously shown Table 1 provides a list of freight data sources reviewed in this report. Appendix A of the contractor’s final report (avail- able on the project web page) includes an augmented version of this table, with data sources expanded at the dataset level. This appendix also includes a detailed description of each data source. The list of freight data sources in Table 1 is obviously not comprehensive. For example, it does not reference datasets that state, regional, and local entities need to collect to sup- plement or augment national-level datasets. Although the list in Table 1 does not include all of the potential data sources that deal with freight transportation, it is useful because it provides a sample of the typical national-level data sources C H A P T E R 5 Conclusions and Recommendations

77 that would need to be included in a national freight data archi- tecture. A few systems and architectures in Table 1 were of par- ticular interest because of the lessons that could be derived from the processes that led to their development. The analysis included topics such as purpose, content, institutional arrange- ments used for developing and maintaining the system or architecture; challenges and issues faced in creating and main- taining the architecture or system; strategies and methods for dealing with data integration issues; and adaptability to serve evolving purposes and data sources. Online Surveys, Interviews, and Peer Exchange The research team conducted a planner and analyst sur- vey, a shipper survey, and a motor carrier survey (as well as follow-up interviews) to gather information about freight data uses and needs. The research team also conducted inter- views with subject matter experts to address specific items of interest to the research. The purpose of the planner and analyst survey was to gather information from government planners, analysts, and other similar freight-related stake- holders. The invitation to participate in the survey included groups such as AASHTO committees, TRB committees, and AMPO. Respondents were involved in all modes of trans- portation, including air, rail, truck, pipelines, and water. Not surprisingly (given that respondents were typically public- sector planners), most respondents indicated that they use freight data to support the production of public-sector trans- portation planning documents. However, respondents also reported using data for various other freight-related appli- cations, adding weight to the notion that the national freight data architecture should support various freight-related pro- cesses. Respondents reported using and/or needing data at var- ious levels of geographic coverage and resolution. The feed- back on unmet data needs complement similar findings in the literature. The purpose of the shipper survey was to gather general information from the shipper community regarding freight data uses and needs, as well as willingness to share data with other freight-related stakeholders. The survey included repre- sentatives of companies of various sizes, including third-party logistics, freight forwarders, manufacturers, retailers, and sup- pliers. The shipper industry collects large amounts of data. Many shippers and logistics service providers transmit data electronically using EDI technologies. These stakeholders use EDI regularly for load tendering, tracking, and billing pur- poses. However, accessing data from shippers and logistics service providers for transportation planning applications (beyond aggregated data from commercial data providers and national survey campaigns such as CFS) is not necessarily straightforward. For example, although a data record might characterize a commodity as well as origin and destination locations, the route data component may be missing. In addition, the shipper stakeholders interviewed indicated they could not comment on their companies’ ability or willing- ness to share data for freight transportation planning purposes (particularly on a load-by-load basis, given its proprietary and confidential nature). Subsequent feedback obtained at the peer exchange (see below) highlighted a number of strategies to address this issue, including initiating discussions about data sharing at a sufficiently high administrative level—since low- ranking personnel might know the data, but frequently do not have the authority or permission to discuss data sharing options. Involving trade associations rather than individual firms also might be beneficial. A business model also might emerge in which data providers would forward sample data to a designated agency on a predetermined schedule for devel- oping a commodity flow database at the national level. The data would be stripped of certain identifiers to address privacy and confidentiality concerns. Although the data would not be available for free (since filtering, forwarding, storing, and pro- cessing the data would involve real costs), it is anticipated that the cost of collecting the data would be a fraction of the cost to conduct normal surveys. The purpose of the motor carrier survey was to gather infor- mation from the motor carrier community about freight data uses and needs, as well as willingness to share data with exter- nal freight-related stakeholders. Survey respondents repre- sented all major sectors of the motor carrier industry, includ- ing TL, LTL, and specialized. As in the case of shippers, motor carriers expressed reservations about sharing proprietary and confidential data. In particular, their reservations were related to the need to develop mechanisms to protect proprietary and confidential information and to maintain the anonymity of carriers and customers. In general, carriers would need to know in advance the specific uses of the data and, in return, would expect information in the form of industry benchmark- ing metrics. It is worth noting that developing metrics of inter- est to the private sector is part of the scope of work of NCFRP Project 3, “Performance Measures for Freight Transportation.” In practice, the type and amount of data provided by, or available through, carriers varies considerably, depending on factors such as carrier size, geographic locations, activity focus, and type of cargo transported. Carriers handle large amounts of disaggregated data during the course of their business oper- ations. Increasingly, carriers use EDI standards and applica- tions. However, most of this information is confidential and limited to the direct exchange of data between trading part- ners. In addition, the amount of shipment information detail varies according to the type of carrier. For example, TL carri- ers, who tend to bill customers on a per-mile basis or by using a flat rate, rarely collect detailed commodity data. In addition, TL carriers are less likely to collect data on tonnage hauled or

78 tare-level data, also attributable to industry-accepted billing practices. By comparison, LTL carriers typically bill customers using a rate structure based on shipment weight, origin, des- tination, and freight classification. However, there is anecdotal evidence that LTL carriers tend to favor a freight-all-kinds rat- ing structure that assigns a general freight classification to all shipments from a shipper regardless of freight commodity or type. As opposed to TL carriers, LTL carriers are more likely to track total tonnage. In conjunction with the 2009 North American Freight Flows Conference in Irvine, CA, the research team organized a peer exchange to discuss preliminary research findings; request feedback; and facilitate a dialogue on implementation strate- gies to develop, adopt, and maintain a national freight data architecture. Participants included representatives of fed- eral, state, regional, university, and private-sector agencies. To encourage participation and discussion, attendees received background materials such as relevant research topic sum- maries and breakout group agendas and discussion objectives. Feedback from peer exchange participants included recom- mendations for changes to initial research findings as well as a list of issues, challenges, and strategies to consider during the implementation of the national freight data architecture. National Freight Data Architecture Definition Taking into consideration the results of the literature review, as well as feedback from surveys, follow-up interviews, and the peer exchange, the research team developed the following generic definition for a national freight data architecture: The national freight data architecture is the manner in which data elements are organized and integrated for freight transportation-related applications or business processes. The data architecture includes the necessary set of tools that describe related functions or roles, components where those roles reside or apply, and data flows that connect roles and components at different domain and aggregation levels. Depending on the specific level of implementation chosen for the data architecture, this generic definition could be fine- tuned as follows: • Single-application approach. In this case, the national freight data architecture would become the manner in which data elements are organized and integrated for a spe- cific freight application or business process at the national level (e.g., commodity flows). • Intermediate approaches (depending on the number of applications). In this case, the national freight data architec- ture would become the manner in which data elements are organized and integrated for a specific set of applications at the national, state, regional, and local levels. A large num- ber of intermediate approaches is possible, depending on the business processes and geographic levels involved. For example, an intermediate implementation approach could include commodity flows at the national, state, and regional levels. Another, more encompassing, intermediate imple- mentation approach could include commodity flows, safety, and pavement impacts at the national, state, regional, and local levels. • Holistic, all-encompassing approach. In this case, the national freight data architecture would become the man- ner in which data elements are organized and integrated for all freight transportation-related applications or busi- ness processes at the national, state, regional, and local levels. For any of these implementation options, the data architec- ture would include the necessary set of tools that describe related functions or roles, components where those roles reside or apply, and data flows that connect roles and components. National Freight Data Architecture Value From the documentation and information gathered dur- ing the research, the research team identified the following list of benefits that, together, provide a statement of value for the national freight data architecture: • Better understanding of the different business processes that affect freight transportation at different levels of cov- erage and resolution; • Better understanding of the supply chain, which should help transportation planners to identify strategies for improving freight transportation infrastructure; • Better understanding of the role that different public- sector and private-sector stakeholders play on freight transportation; • Better understanding of the need for standards to assist in data exchange; • Systematic, coordinated development of reference datasets (e.g., comprehensive commodity code crosswalk tables); • Systematic inventory of freight transportation data sources; • Systematic inventory of user and data needs that are pre- requisites for the development of freight data management systems; • Use as a reference for the identification of locations where there may be freight data redundancy and inefficiencies; • Use as a reference for requesting funding allocations in the public and private sectors; and • Use as a reference for the development of outreach, profes- sional development, and training materials.

In practice, the value of the national freight data architec- ture is also a function of the costs associated with its imple- mentation. Quantifiable data about expected benefits and costs are currently not available (benefit-cost analyses need to occur both at the beginning and at different phases of imple- mentation of the national freight data architecture). How- ever, it is clear from the documentation and information gathered during the research that the “do-nothing” alterna- tive (i.e., not implementing the national freight data architec- ture) is costly, ineffective, and, unsustainable. Therefore, the research team’s recommendation is to pursue the national freight data architecture following a scalable implementation path in which the national freight data architecture starts with one application at one or two levels of decisionmaking and then adds applications and levels of decisionmaking as needed or according to a predetermined implementation plan until, eventually, reaching the maximum net value. National Freight Data Architecture Components The research team identified the following categories of com- ponents to include in the national freight data architecture: • Physical transportation components, • Cargo or freight, • Freight functions or roles, • Business processes, • Data sources, • Freight-related data, • Freight data models, • Freight data standards, and • User interface and supporting documentation. Figure 10 (shown previously) presents a high-level modular conceptualization and list of different categories of compo- nents. The diagram recognizes the scalable nature of the national freight data architecture and enables the production of a variety of diagram versions (as well as tabular represen- tations) depending on what implementation level to pursue. For example, for a single-application data architecture that only focuses on commodity flows at the national level, it may not be necessary to depict (at least not in detail) other freight functions and business processes. Similarly, not all data stan- dards would need to be considered, and the requirements for user interfaces to support that data architecture would be relatively minor. The diagram in Figure 10 is only one example of potentially many different types of diagrams that can be used to depict interactions among freight transportation components. An example of a different type of diagram previously shown is Figure 12, which presents a high-level conceptual model that focuses on relationships between different individual data architecture components for specific business processes. National Freight Data Architecture Recommendations and Specifications In addition to the list of categories and components, the research team put together a list of recommendations for the development and implementation of the national freight data architecture. For convenience, the recommendations are writ- ten in the form of specifications to guide and monitor the implementation of the data architecture. Chapter 4 provides a detailed discussion of each specifica- tion item. For compactness, only the title of each specification is listed here: • Adopt a definition for the national freight data architecture that is generic, scalable, and is understood and accepted by the freight transportation community (see proposed defi- nition above); • Compare candidate data architecture implementation concepts; • Develop implementation plan for national freight data architecture components; • Develop lists of components to include in the national freight data architecture; • Develop and implement protocols for continuous stake- holder participation; • Conduct data gap analysis; • Conduct data disaggregation need analysis; • Assume a distributed approach (as opposed to a central- ized approach) to freight data repository implementations; • Use a systems engineering approach for developing the national freight data architecture; • Use standard information technology tools and procedures; • Develop and/or use standardized terminology and defini- tions for each data architecture component developed; • Implement strong privacy protection strategies; and • Establish integration points with other data architectures and standards. Challenges and Strategies The research team identified relevant issues and challenges that might block the implementation of the national freight data architecture, as well as candidate strategies for develop- ing, adopting, and maintaining the data architecture. The challenges were in the following categories: technical chal- lenges, policy challenges, economic and financial challenges, and stakeholder buy-in and consensus. Chapter 4 provides a detailed description of each challenge. Only a summarized list of challenges is provided here. 79

80 • Technical challenges. Technical challenges refer to issues (e.g., technological limitations, hardware and software incompatibilities, and standards incompatibilities) that might impede the successful implementation of the data architecture. Examples include the following: – Feasibility of different implementation approaches; – Data storage requirements; – Feasibility of updated data entry protocols to eliminate data redundancies and support standardized data entry procedures; – Conversion of commodity code classifications; – Data life cycle and usefulness to support the decision- making process by public and private stakeholders; – Variability in data quality control practices, which affect data accuracy, completeness, and timeliness; – Differences in terminology, data item definitions, and data implementations among freight data stakeholders; – Prioritization of data architecture components; – Integration between shipper and carrier data to charac- terize commodity movements properly; and – Data confidentiality and security concerns. • Policy challenges. The national freight data architecture might fail if required policies, both in the public and private sectors, fail or are not feasible. Examples of policy challenges include the following: – Homeland security concerns, which might limit the dis- semination of certain freight-related data; – Impact on current private-sector data collection initia- tives; and – Competitive and proprietary (privacy) concerns with the concept of public-sector agencies having access to private-sector data. • Economic and financial challenges. The national freight data architecture might fail if the perceived costs associated with its implementation exceed the benefits that stakehold- ers would receive. Examples of economic and financial challenges include the following: – Cost of data collection, storage, and quality assurance; – Benefits and costs related to data disaggregation require- ments for different business processes; – Data life cycle and usefulness to support the decision- making process by public and private stakeholders; – Cost to acquire private-sector data; and – Cost to implement robust data confidentiality and data security measures. • Stakeholder buy-in and consensus. The national freight data architecture might fail if there is no stakeholder buy- in or consensus about the potential benefits that could result from implementing the data architecture. Examples of related issues include the following: – Reluctance of stakeholders to participate if there is no clarity regarding justification and anticipated benefits; – Confidentiality clauses in supply chain contracts, which might impede data sharing for transportation planning purposes; – Perception that data collected as part of a national freight data collection program could validate projects of national significance at the expense of small or rural communities; – Ability of small carriers to provide data about loads they move; – Risk of low stakeholder participation, which could decrease data reliability; and – Adequacy of data standards. Strategies to ensure a successful implementation of the national freight data architecture include the following: • Implementation levels – Develop and compare candidate data architecture concepts, – Identify business process and implementation level priorities, – Develop high-quality data architecture concepts and applications that address the needs of the highest prior- ity items first, and – Identify data needs at the finest disaggregation level and implement data collection and data storage plans at that level. • Relationships with leaders, champions, and stakeholders – Identify data architecture leaders and champions; – Engage the national freight data architecture champions early; – Maintain good communication channels with the vari- ous stakeholders during all phases of the development and implementation of the national freight transporta- tion data architecture; – Identify funding mechanisms for the implementation of the data architecture; – Develop brochures, presentations, and other materi- als that explain the national freight data architecture, its scope, high-level components, and what it expects to accomplish; – Deliver effective messages on how the national freight data architecture will assist stakeholders in the identifi- cation of strategies to address a variety of freight-related issues ranging from data collection to analysis and reporting; – Deliver messages that provide clear, concise answers to the various challenges highlighted in the previous section; – Articulate benefits of participation by the private sector; and – Identify opportunities for partnerships with the private sector (e.g., through public–private partnerships) to make

data accessible for transportation planning purposes in a cost-effective manner. • Performance measures and effectiveness – Develop criteria for measuring the effectiveness in the implementation of the national freight data architecture, – Identify major progress milestones, – Tie the implementation of the national freight data archi- tecture to the development of metrics or performance measures that could benefit the entire freight transporta- tion community, and – Accelerate the implementation of programs such as EFM and the freight performance measurement program. • Lessons learned from the implementation and mainte- nance of existing freight-related systems and architectures (see Chapter 2 for detailed information about these systems and architectures) – Develop systems that are relevant to stakeholders, include adequate stakeholder participation, and provide incentives to encourage participation—particularly in the case of state and local entities; – Clearly define expected outcomes and development and coordination plan; – Articulate programs well; provide clear, uniform guid- ance; and provide good documentation; – Develop applications that rely on widely used data standards; – Develop and compare candidate architecture concepts; – Consider federal legislation to support and develop the program; – Develop tools to measure benefits and costs early; – Integrate archived data needs into frameworks and architectures early and develop data programs that use industry standards; – Implement interagency data exchange programs with centralized data coordination; – Use available data sources and develop long-term plans while keeping systems flexible to respond to changes and new data sources; – Schedule major and regular revisions effectively while avoiding scope creep; – Develop systems that are consistent with input data limitations; – Develop applications with backward compatibility; – Evaluate data disaggregation level requirements to ensure statistical significance; – Provide adequate resources for data collection, fully understand the implications of small sample sizes, and continue to involve the U.S. Census Bureau for the use of survey instruments; – Emphasize data access, quality, reliability, confidentiality, and integrity; – Participate in the standards development process; – Create crosswalks to ensure compatibility of survey data internally over time and externally across other datasets; – Involve stakeholders early and often through a variety of mechanisms and technologies; and – Develop and implement professional capacity and train- ing programs early. One of the strategies for implementation mentioned above is to develop and compare candidate data architecture concepts. Peer exchange participants highlighted that implementing a comprehensive data architecture at once with no testing of options prior to making a decision about the correct approach would be too risky. Participants also favored the concept of developing and comparing several alternative approaches. A recommendation from peer exchange participants was to use NCFRP as an avenue for funding the development of alter- native data architecture concepts. Participants indicated the request for proposals should outline clear objectives, while leaving the definition of approaches to the research team(s) selected. An idea discussed was to develop the data architecture around scenarios or themes, such as business areas or processes, levels of government, or economic activity. Activities in connec- tion with each scenario or theme would include structuring a competition for research teams (each of which would include a university partner, a private-sector partner, and a government- level partner) to develop and test competing data architecture concepts, making sure to include multimodal components in the scenarios and tests, and conduct a follow-up evaluation. 81