Leveraging Big Data to Improve Traffic Incident Management (2019)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

114 This report has addressed the NCHRP Project 17-75 research objectives, which were to (1) describe and assess current and emerging sources of data that could improve TIM, (2) describe potential opportunities to leverage Big Data that could advance the TIM state of the practice, (3) identify potential challenges for TIM agencies to leverage Big Data, and (4) develop Big Data guidelines for TIM agencies. The sections in this chapter summarize the findings of the research, set forth potential next steps for the research findings, and address recommendations, needs, and priorities for additional related research. 7.1 Summary of Findings The state of the practice in TIM shows significant advancement over the past decade, most notably through the development of regional and statewide TIM committees, the National TIM Responder Training Program efforts, the implementation of TIM legislation, and the collection and analysis of TIM data for performance measurement. Recent guidance, like that provided through efforts of the TRB and the FHWA—including FHWA’s ongoing “Every Day Counts” (EDC) initiative, now beginning its fifth round—reflects national efforts to advance the collection and use of TIM data. The findings from a review of the state of the practice in Big Data reinforce awareness that: • Big Data is not new; rather, Big Data technologies and techniques have been applied for nearly two decades by various companies; • Although Big Data is characterized by the five Vs of volume, velocity, variety, veracity, and value, not all datasets need to possess all five of these qualities to be considered Big Data; • Contrary to the relational database approach, Big Data analytics is not bound to a single set of tools to perform analyses; rather, Big Data analytics encompass a wide variety of proprietary and open-source tools that can be customized and modified by users; • The tools used for Big Data analytics allow for the rapid transfer, processing, storage, and analysis of extremely large datasets, have increased the ability to analyze divergent data (e.g., decades-old historical records and real-time streaming data), and make it possible to derive value from data that cannot be attained using traditional data mining approaches that typically rely on relational databases. Big Data applications in the field of transportation are more recent, having occurred within the past few years, and include applications in the areas of planning, parking, trucking, public transportation, operations, ITS, and other more niche areas. A significant gap exists between the current state of the practice in Big Data analytics (e.g., image recognition, graph analytics) and the state of DOT applications of data for TIM (e.g., manual use of Waze data for incident detection). C H A P T E R 7 Summary and Next Steps

Summary and Next Steps 115 A few TIM Big Data applications were identified, but these were largely applications that could be performed using relational databases. Local data and state data generally are not collected at the volumes that make using or applying Big Data approaches practical. Ways are available to expand on these initial approaches to Big Data for TIM, but the data must first be prepared, must be of sufficient size, and must cover a sufficient length of time to identify meaningful patterns and yield value. Big Data applications offer significant opportunities to improve TIM, as highlighted in Chapter 4 through contrasts made between traditional and Big Data approaches to common areas of TIM concerns. These example Big Data applications illustrate that, beyond offering improvement on current practices, the Big Data approach represents a radical change from traditional approaches. Big Data represents a paradigm shift that goes beyond data collection and analysis to include data storage, management, and security; the financial planning and procurement of IT services; the required skillsets of employees; and beyond. Opportunities to apply Big Data to TIM at a regional or state level are currently limited by the collection and availability of data and the capability maturity of analysts. Although a few existing Big Data datasets (e.g., data available from HERE, INRIX, and Waze) might be immediately leveraged for TIM, these datasets alone lack the detail needed for effec- tively mining and understanding the nuances of incident response and TIM, and access to the raw data remains limited. Many of the benefits of Big Data analytics for TIM will require collect- ing and integrating more TIM-specific, detailed data (e.g., crash data or CAD data), at minimum at a state level if not at the national level, to establish sufficient volume and variety for uncovering relationships and insights. Discussing these opportunities now can help agencies identify the low-hanging fruit for Big Data in TIM, and will help agencies see the benefits of taking the next steps toward undertaking a TIM Big Data initiative. The research identified many challenges and potential barriers that could impact the appli- cation of Big Data for TIM. At the forefront of these challenges are aspects of organizational culture—specifically, challenges that impede agencies’ willingness and ability to embrace the paradigm shift that Big Data requires. Reluctance to open and share data, as well as impedi- ments that stand in the way of using cloud infrastructure, are two central factors that will limit the growth and application of Big Data within an organization. The application of Big Data also requires sensitivity to organizational capabilities. The level of technical expertise among existing TIM stakeholders at local, regional, and state agencies will likely vary widely, with the result that the skills and resources needed to close the gap between current data practices and Big Data practices may not be sufficient to comfortably and efficiently apply Big Data. Further, individuals who have Big Data expertise are in limited supply and in high demand, which may hamper agencies’ ability to train or hire talent and purchase the requisite resources. Fundamental to Big Data analytics is having access to large amounts of varied data. An assess- ment of 31 different data sources showed that a large gap exists between the current state of TIM-related data and the application of this data for Big Data analytics. Although merging a few datasets may be tenable for agencies, building the large, highly detailed, integrated datasets needed for Big Data will require significant resources, as well as the expertise to apply non- traditional approaches. Challenges such as the lack of standards for data collection and storage, PII, legal restrictions, and agency culture and policies will limit the application of Big Data for TIM. Furthermore, although millions of data points are generated every second by traffic sensors and probes, incidents are infrequent by nature and therefore relatively small in number. This limits the application of Big Data to TIM unless the data is aggregated across multiple regions and organizations to increase its volume and variety.

116 Leveraging Big Data to Improve Traffic Incident Management The research suggests that the current state of the practice for TIM data collection, storage, and analysis is between the first and second tiers on the Big Data pyramid. At this point, very limited TIM data is being collected and shared among partner agencies, and a solid data lake as a foundation for the development of TIM business intelligence and TIM data science has yet to be built. Therefore, based on the research findings, guidelines for transportation and TIM agencies were developed to lay out the various changes that will be necessary for agencies to develop a usable Big Data store (data lake), implement agency-wide analytics and business intelligence, and pursue the development of an evolving and beneficial data science environ- ment. Expressed at their highest level, the guidelines suggest that agencies prepare to: • Adopt a deeper and broader perspective; • Collect more data; • Open and share data; • Use a common data storage; • Adopt cloud technologies for the storage and retrieval of data; • Manage the data differently; • Process the data; and • Open and share outcomes and products to foster data user communities. By embracing these guidelines and the actions suggested to accompany them, agencies can address and overcome the challenges that limit the move toward the use of Big Data for TIM. Applying these guidelines will thus help position transportation and TIM agencies for Big Data. 7.2 Next Steps Agencies are encouraged to begin following the guidelines and putting the research into practice by fully embracing low-cost, traditional best practices in data collection, cleaning, warehousing, and analysis with existing data sources. Agencies also are encouraged to concur- rently identify opportunities to ready their organizations for Big Data. Opening and sharing data—both internally and externally—are critical cultural shifts that need to be embraced. An incremental approach is recommended that begins with developing the culture, policies, and expertise to improve the usability and increase the use of current data, and that captures oppor- tunities to migrate from in-house servers to the cloud. These steps form the basis for positioning agencies to begin capitalizing on the opportunities afforded by Big Data. Migrating research and guidelines from ideas into practice can begin by linking research results and outputs to related products and by engaging stakeholders. This endeavor can be enhanced by using the TIM Performance Measurement (TIM PM) Website (http://nchrptimpm. timnetwork.org/). A product of NCHRP Project 07-20, “Guidelines for the Implementation of TIM Performance Measurement,” TIM PM offers a natural location to obtain and share TIM information. NCHRP Project 07-20 was the first to offer a standardized set of TIM data elements, as well as a standardized way to organize these data elements in a database schema, so that TIM performance could be measured and analyzed. The website is a natural location at which to include information on Big Data as a next step in the application of data to improve TIM. The FHWA’s TIM Capability Maturity Self-Assessment (TIM CMSA)(https://ops.fhwa.dot. gov/tsmoframeworktool/available_frameworks/traffic_incident.htm) tool is used by state and local TIM program managers to benchmark and evaluate TIM program strengths, weaknesses, successes, and areas for improvement on an annual basis, and to aid in the development of a targeted action plan for TIM. Data collection, integration, and sharing is a key part of the TIM CMSA. Given that Big Data is in everyone’s future, agencies should have an opportunity to

Summary and Next Steps 117 assess themselves on the foundational principles associated with readying their organizations for Big Data. The EDC-2 National TIM Responder Training Program and the post-course assessment tool that was developed under SHRP 2 Project L32(C) offer other opportunities to incorporate effective guidance on the importance of good data collection and sharing practices and the understanding of how the data can help to improve TIM by informing decision-making, resource utilization/management, real-time TIM activities, and program funding decisions. Finally, many states have statewide and/or regional TIM coalitions or committees that include participation and representation from the various TIM stakeholder disciplines. Regular coalition or committee meetings provide an opportunity for stakeholders to discuss TIM practices, share lessons learned, and discuss ways to improve TIM. These meetings also offer opportunities to introduce concepts that connect Big Data to TIM, embedding the knowledge at the responder level across the various disciplines. Receptiveness or interest may vary across responder communities, but the research for this project has made it clear that some responders recognize the importance of data and are willing to take pertinent information to their upper management. Presenting tangible examples of Big Data applications and outcomes specific to TIM operations can help spur interest and motivation to take action. 7.3 Suggestions and Priorities for Additional Related Research Big Data is here, and transportation agencies are encouraged to begin embracing the changes required to tackle it. Traditional organizational cultures and lack of data may be holding back full acceptance and adoption of the foundational principles of Big Data, but the emergence of connected vehicle, traveler, and infrastructure data will soon be driving the change. To capitalize on the wealth of information that can be derived from these and other data sources—and to prevent system failures caused by data overload—transportation agencies must ready themselves for Big Data. The technology is here, the tools are available, and the expertise can be found to assist transportation agencies in both understanding and applying these technologies and tools to everyday questions and problems. Effective strategies and techniques are needed to recognize and break down some of the barriers that still impede agencies’ understanding and adoption of Big Data technologies. Additional research could help agencies find ways to overcome cultural barriers to opening and sharing data, and to resolve legal or proprietary concerns. Once transportation and partner agencies have collected, opened, shared, and pooled enough (and varied) data in a cloud envi- ronment, further research can then be conducted using Big Data techniques to discover how Big Data can help to improve specific components of TIM programs.