Guidebook for Managing Data from Emerging Technologies for Transportation (2020)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

1 Guidebook for Managing Data from Emerging Technologies for Transportation provides guidance, tools, and a big data management framework, and it lays out a roadmap for transportation agencies on how they can begin to shift— technically, institutionally, and culturally—toward effectively managing data from emerging technologies. Transportation safety and mobility, which enhance American productivity, have advanced over the past three decades. This advancement is due in large part to various transformational intelligent transportation technologies, including advanced traffic management systems, elec- tronic toll collection, traffic signal coordination, transit signal priority, and traveler informa- tion systems, to name a few. Further developments in communications and technology have led to more advanced infrastructure and vehicle capabilities, mobile applications, and a host of mobility service offerings, including connected vehicles, automated vehicles, on-demand and shared mobility services, crowdsourcing, the Internet of things (IoT), and new mobility initia- tives such as smart cities and communities. All are producing data at extraordinary volumes and speeds. A range of institutions, both public and private, have initiated demonstration and pilot projects of these technologies, and many have invested in associated data sets. As these activi- ties continue to expand, the amount of data is also expanding. Data from emerging tech- nologies have tremendous potential to offer new insights and to identify unique solutions for delivering services, thereby improving outcomes. However, the volume and speed at which the data are generated, processed, stored, and sought for analysis are unprecedented and will fundamentally alter the transportation sector. With increased connectivity between vehicles, sensors, systems, shared-use transportation, and mobile devices, unexpected and unparalleled amounts of data are being added to the transportation domain at a rapid rate, and these data are too large, too varied in nature, and will change too quickly to be handled by the traditional database management systems of most transportation agencies. Instead, modern, flexible, and scalable “big data” methods to manage these data need to be adopted by transportation agencies if the data are to be used to facilitate better decision-making. As many transportation agencies are already forced to do more with less while meeting higher public expectations, continuing with traditional data management systems and practices will prove costly for transportation agencies unable to shift. Guidebook for Managing Data from Emerging Technologies for Transportation provides guid- ance, tools, and a big data management framework and lays out a roadmap for transportation agencies on how they can begin to shift—technically, institutionally, and culturally—toward effectively managing data from emerging technologies. New concepts and methodologies con- cerning data management and use are introduced, along with industry best practices for big data. Examples, references, and quotations are provided from transportation agencies that are navigating the implementation of big data to extend beyond traditional siloed use cases, includ- ing their challenges and successes. There is discussion of common misconceptions within the transportation industry. Whether an agency is starting from scratch with a new technology data set, making a business case for emerging technology data, already working on a big data project, has an issue or problem that might be solved with emerging technology data, or is looking for C H A P T E R 1 Introduction

2 Guidebook for Managing Data from Emerging Technologies for Transportation a new enterprise data management solution, the steps and guidance are designed to walk the agency through the necessary data management policies, procedures, and practices to fully meet the needs of data from emerging technologies. This guidebook is for engineering and information technology (IT) analysts and profes- sionals working for state, regional, and local transportation agencies. The target audience is those at the mid-level manager level, including positions such as operations manager, ITS program manager, traffic operations/traffic management center manager, IT manager, and database administrator. It is the intent that these staff, in their positions, would play a primary role as champion in leading data initiatives for their agencies that support positive organiza- tional change. Successful implementation requires close cooperation between engineering and IT departments, which will provide mutual benefits and create a lasting and positive impact on the organization and its personnel. As such, as agencies progress through this guidebook, they are encouraged to revisit and/or assess existing processes and workflows that will need to be transitioned or replaced as a result of applying emerging technology data and modern data management processes within the organization. Figure 1 shows the various components of this guidebook. The guidebook comprises three major sections, which are described following the figure. • Laying the Foundation. For 11 characteristics of data systems and management, this section contrasts the traditional approach versus the modern big data approach and provides an overview of a recommended big data architecture. • Modern Big Data Management Framework. This section provides data industry best prac- tices and more than 100 recommendations for modern data management across the full big data life cycle, including the creation of data, storage of data, use of data, and sharing of data (Figure 2). This framework provides the details of “how-to” manage data from emerging technologies. • Roadmap to Managing Data from Emerging Technologies. This section presents an 8-step process and associated guidance for transportation agencies looking to begin or further efforts toward more effectively managing data from emerging technologies, with the goal of organization-wide change. For agencies just beginning, the Roadmap provides a starting point. For agencies already on their way, the Roadmap provides details on how to further those efforts and gain cross-organizational support. Laying the Foundation Modern Big Data Management Framework Roadmap to Managing Data from Emerging Technologies 100+ recommendations across the data lifecycle 8-step process toward organizational change Contrasts traditional vs. modern approach for 11 characteristics of data systems Presents modern big data architecture NCHRP Web-Only Document 282 Data Management Capability Maturity Self-Assessment (DM CMSA) Data Sources Catalog Tool Big Data Governance Role & Responsibilities • • • • • Frequently Asked Questions (FAQ) Supporting Resources & Tools Figure 1. Components and application of the guidebook.

Introduction 3 These three major sections will provide agencies with the foundation, recommendations, and steps to effectively implement a modern, flexible, scalable, and sustainable approach to managing data from emerging technologies. Supporting Resources and Tools In addition to these major sections of the guidebook, there are a number of supporting resources and tools for agencies implementing the guidebook. These resources and tools follow. • NCHRP Web-Only Document 282: Framework for Managing Data from Emerging Transportation Technologies to Support Decision-Making details the research activities. The activities include results from a comprehensive state-of-the-practice review (49 sources cited) both external and internal to the transportation industry; an online survey of 25 orga- nizations deploying emerging transportation technology projects; telephone interviews with 11 city and state transportation agencies involved in managing data from emerging tech- nologies; and a stakeholder workshop involving 17 representatives from 15 local, regional, and state agencies. The report defines data management and provides 65 modern big data management foundational principles organized by 15 data management “focus areas” to cover the full life cycle of big data. The revised final report presents a modern big data benchmark and assessment method- ology, built from the foundational principles of big data management, which was applied to the information gathered from agencies participating in the research to further assess the state of the practice in data management within the transportation industry. The report ends with a list of common challenges reported by agencies during the research, as well as a list of associated needs. This guidebook was developed specifically to address as many of these challenges and needs as possible. While there is some natural overlap in content between the revised final report and this guidebook, an effort has been made to preserve the brevity and applicability of the guidebook. It is recommended that agencies implementing the guidebook refer to the revised final report for further background and details regarding the management of big data. • Data Management Capability Maturity Self-Assessment (DM CMSA). The DM CMSA is built from the modern big data benchmark and assessment methodology presented in Web-Only Document 282: Framework for Managing Data from Emerging Transportation Technologies to Support Decision-Making. Questions within each of the 15 focus areas will guide transportation agencies through a self-assessment to gauge their current data manage- ment practices, as well as to identify specific areas for improvement along their path toward shifting from traditional data management practices to more modern data management prac- tices to handle data from emerging technologies. Figure 2. Big data life cycle.

4 Guidebook for Managing Data from Emerging Technologies for Transportation • Big Data Governance Roles and Responsibilities. Provides a list of recommendations to consider when developing a modern data governance approach, a description and frame- works for big data governance, and a tool for tracking the big data governance roles and responsibilities within an agency. • Data Sources Catalog Tool. Provides assistance to transportation agencies in cataloging existing and potential data sources. This tool is useful for better understanding the data assets of an agency, prioritizing data sources, and informing the selection of those sources that might offer the most value before pursuing further development. • Frequently Asked Questions (FAQs). Responses to frequently asked questions regarding big data implementation, management, governance, use, and security. As indicated in Figure 1, it is recommended that readers start by reviewing the Laying the Foundation chapter. These pages concisely contrast the traditional data systems and manage- ment approach/practices of most transportation agencies with their modern data systems and management counterparts and present a graphical representation and associated discussion of a recommended modern big data architecture. Agency staff at all levels, including executive leadership, should understand the concepts presented. From here, Figure 1 illustrates the cyclical and incremental nature in which agencies can apply this guidebook. The Framework and Roadmap are interrelated tools for agencies to achieve iterative and progressive transformation/change over time. How an agency proceeds will depend largely on the agency’s needs, organization, level of data maturity, drivers for change, who is leading the charge, and the goals and anticipated outcomes of the effort. One agency may start by diving into the technical details and best practices presented in the Frame- work to further its understanding of modern data management. Another agency may start by reviewing the steps, requirements, and resource needs associated with implementing the Roadmap. Yet another agency may begin by convening an internal group of stakeholders to conduct the DM CMSA in order to set baseline current practices and identify and prioritize areas for improvement. Once an agency begins its journey along the Roadmap, roadblocks faced can be overcome by using recommendations in the Framework or by reviewing/applying the resources and tools provided (e.g., DM CMSA). For example, an agency might find that procurement does not support use of the cloud (a foundational recommendation and best practice of modern data management). To support agency champions in getting beyond this barrier, the Framework and Roadmap provide arguments agency champions can use for why the cloud is needed. There is natural overlap between the various components of the guidebook. For example, agencies will find discussions in Steps 4 and 5 of the Roadmap that expand on recommenda- tions found within the Framework, as well as associated questions within the DM CMSA. The text in the guidebook has not been designed to be read from front to back; agencies should refer to the various sections and supporting resources/tools as needed to support short-term incremental progress and long-term organizational change. Terminology There are numerous technical terms within this guidebook. While definitions are provided for some technical terms, other technical terms are used in a context without specifically defining them. As there are multiple sources providing standard definitions to these and other terms commonly used in the field of information technology, rather than repeat these definitions herein, readers are encouraged to visit the following sites: Techopedia at https:// www.techopedia.com/dictionary and TechTerms at https://techterms.com/.