Management and Use of Data for Transportation Performance Management: Guide for Practitioners (2019)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Insight

Introduction • Foundation • Reporting • Insight • Cases Step 5 Analyze & Use Data The Analyze & Use step begins once data are converted into information. Data consist of values and figures that on their own have limited value. Information is a result of data that have been processed, organized, and interpreted to provide insight. Analyzing and using data for TPM involves consumption of data by analysts, planners, managers, engineers, and operations personnel to inform decision making or direct real-time system management. In the Store & Manage step, reporting and analysis tools are selected and configured. In the Analyze & Use step, these tools are used to support decision making. Moving from installation of a tool to productive use of the tool requires, at a minimum, the following: • Identification of the intended users and uses for the tool; • Designation of one or more individuals to develop specialized expertise with the tool (or engagement of a consultant to play this role); • Training and support for additional users of the tool; and • Iterative application and adjustment to tool parameters and configuration. “Distinguishing the signal from the noise requires both scientific knowledge and self-knowledge; the serenity to accept the things we cannot predict, the courage to predict the things we can, and the wisdom to know the difference.” Nate Silver 53

Introduction • Foundation • Reporting • Insight • Cases Step 5.1 Analyze Trends Assemble data. Assemble historical performance data for as many years as possible. If there have been changes to measurement methods, document when these occurred, but do not discard the older data. Even if there are discontinuities in the trend line, each section of the line can still be instructive for understanding how performance changed within each applicable time period. Review and analyze the data. Plot the data to visualize variations over time. If appropriate, apply smoothing techniques such as moving averages to reduce noise. Use statistical techniques to distinguish the underlying trends in the data from seasonal variations and one-off variations due to events or other exogenous factors. Involve someone with expertise in statistics to be sure that the methods being applied are valid for the data being analyzed. For more information... 1. “Time-Series Review of Highway Performance Monitoring System Data,” Transportation Research Record (TRB, 2018) https://journals.sagepub.com/doi/10.1177/0361198118767415 2. Applying Safety Data and Analysis to Performance-Based Transportation Planning (FHWA, 2015) https://safety.fhwa.dot.gov/tsp/fhwasa15089/data_anl.pdf 3. Silver, N. The Signal and the Noise: Why So Many Predictions Fail—But Some Don't (Penguin Books, 2012) 4. Trends in Non-Fatal Traffic Injuries 1996-2005 (NHTSA, 2008) https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/810944 Maryland SHA used an incident timeline tool and graphics showing queue buildups and delay costs to help convince the responder community to change its policies for blocking lanes. Case E 54

Introduction • Foundation • Reporting • Insight • Cases Step 5.2 Identify Patterns & Causes Visualize data. Many people think of visualization as an end product— something produced after an analysis is complete to help communicate the results (or a story) to the public. While this is often the case, visualization can also be leveraged during the analysis life cycle as a way to better understand what is in your data, to identify outliers, and even to point out flaws that exist in your data. Interactive visual analytics can lead to insights earlier in the TPM process, sometimes more so than at the end. Interpret data. Involve a group of experienced analysts in interpreting the observed trends. Look for correlations between performance trends and factors such as changes in revenues or budget allocations, fuel prices, economic conditions, or legislation/regulation. Use statistical packages and available analytical tools to analyze correlations. Develop insights that can be communicated to stakeholders (see Step 6—Present & Communicate Data.) For more information... 1. Approaches to Presenting External Factors with Operations Performance Measures (FHWA, 2018) https://ops.fhwa.dot.gov/publications/fhwahop18002/fhwahop1800 2.pdf 2. Historical Performance Evaluation of Iowa Pavement Treatments Using Data Analytics (Midwest Transportation Center, 2017) https://intrans.iastate.edu/app/uploads/2018/03/iowa_pvmt_tx_hist orical_perfomance_eval_using_data_analytics_w_cvr.pdf 3. An Analysis of the Significant Decline in Motor Vehicle Traffic Fatalities in 2008 (NHTSA, 2010) https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/811346 Arizona found in its Long-Range Transportation Plan process that for some performance areas, good outcome-oriented performance curves can be established. Where this was not possible, however, ADOT relied on simple curves reflecting the percent of identified needs met at a given allocation level. The lesson was to “not let the perfect become the enemy of the good.” Case A 55

Introduction • Foundation • Reporting • Insight • Cases Step 5.3 Predict Future Performance Create predictive models. Develop realistic assumptions about future work based on available revenues, program budgets, and improvement programs. Use available analytical tools to predict future performance based on funding levels and/or specification of planned improvements. These include pavement and bridge management systems, safety analysis tools, travel demand models, and other specialized simulation tools. In addition, predictive analytics tools are available that make use of a variety of statistical techniques, including machine learning to predict future performance based on available data. Applying analytical tools typically involves initial calibration—adjusting model parameters so that predictions are in line with observed conditions. This is followed by an iterative process of testing different assumptions and reviewing results for reasonableness. Every model has limitations; it is the role of an analyst to understand and explain these limitations. Predictive models generally require specialized expertise to set up and use. Significant modeling tasks can be outsourced if this expertise is not available in-house. However, staff with analytical skills, patience, and interest in modeling can be trained to take on ownership and apply these tools as well as oversee work of contractors. There are variations in available predictive tools for different performance areas, and it can take time to develop robust modeling capabilities. Agencies can start with basic approaches to prediction that rely on expert judgment and rules of thumb. As long as methods are clearly documented and caveats are stated, these approaches can provide value. Caltrans developed a unified approach to presenting predictions of asset performance and need that combined results from mature pavement and bridge management system runs with analytical methods based on available data and expert judgement for other assets. The approach involved combining data from multiple disparate data sources that were at different levels of completeness and based on different analysis methodologies at varying levels of sophistication. Case B 56

Introduction • Foundation • Reporting • Insight • Cases For more information... 1. FHWA Pavement Management Quarterly Webinar— Pavement Performance Modelling (2018) https://connectdot.connectsolutions.com/p9octngm9vv9/ 2. Bhavsar, P., Safro, I, Bouaynaya, N., Polikar, R., & Dera, D. “Machine Learning in Transportation Data Analytics,” Data Analytics for Intelligent Transportation Systems, pp 283-307 (Elsevier Inc., 2017) 3. Predicting Travel Time Reliability Using Mobile Phone GPS Data (Microsoft Research, 2016) https://www.microsoft.com/en-us/research/wp-content/ uploads/2016/11/GPS_flow_sensing_travel_times.pdf 4. Paz, A., Veeramisti, N., & de La Fuente-Mella, H. Forecasting Performance Measures for Traffic Safety Using Deterministic and Stochastic Models (IEEE, 2015) https://www.researchgate.net/publication/308864167_Forecasting_ performance_measures_for_traffic_safety_using_deterministic_ and_stochastic_models 57

Introduction • Foundation • Reporting • Insight • Cases Step 5.4 Establish/Update Targets Integrate results of trend analysis and predictive analysis. Establish a baseline value based on the trend line. Integrate the results of trend analysis and performance predictions to set targets for future performance. Document the analysis. Documentation should include data sources, the steps taken to prepare and combine them, any key assumptions or parameters used (e.g., inflation rates), and observations about data anomalies or correlations. Good documentation will enable the analysis process to be repeated in the future by other staff members and will serve as a valuable resource if questions come up about the results. For more information... 1. Safety Target Setting Factsheet (Arkansas Department of Transportation, 2018) http://www.tpm-portal.com/wp- content/uploads/2016/02/ARDOT-Target-Setting-Safety.pdf 2. Safety Performance Management Targets for 2018 (California Department of Transportation/Office of Traffic Safety, 2018) http://www.dot.ca.gov/fed-liaison/docs/Safety-Performance- Management-Targets-for-2018.pdf 3. Performance Management Target Setting Webinar—Pavement and Bridge (PM2) (California Department of Transportation, 2018) http://www.dot.ca.gov/assetmgmt/documents/Webinar_Slides.pdf 4. NCHRP Report 706: Uses of Risk Management and Data Management to Support Target-Setting for Performance-Based Resource Allocation by Transportation Agencies (Transportation Research Board of the National Academies, 2011) http://onlinepubs.trb.org/onlinepubs/nchrp/nchrp_rpt_706.pdf 5. NCHRP Report 666: Target-Setting Methods and Data Management to Support Performance-Based Resource Allocation by Transportation Agencies: Volume I: Research Report: Volume II; Guide for Target-Setting and Data Management (Transportation Research Board of the National Academies, 2010) http://onlinepubs.trb.org/onlinepubs/nchrp/nchrp_rpt_666.pdf Arizona DOT’s Long-Range Plan process developed performance curves for different investment categories, including preservation, modernization, and capacity. These curves were used to analyze the impacts of a change in investment on different performance outcomes. Case A 58

Introduction • Foundation • Reporting • Insight • Cases Capabilities Checklist: Analyze & Use Data Basic o Analysts are aware of and taking advantage of existing commercial off- the-shelf, open source, and publicly available tools for analysis, visualization, forecasting, and scenario analysis. o Analysts are trained in use of data analysis and visualization tools. o Private-sector or university contractors are used to provide data analysis services as alternatives to setting up analysis capabilities in- house. o Data are available that are sufficiently accurate to meet analysis requirements. o Visualization and analysis tools are used to explore and discover data anomalies and limitations. o Data preparation and analysis tasks are well defined and planned to ensure sufficient calendar time and staff resources. o Analysts are able to identify trends and causal factors. o Data element meanings, data transformations, and analysis assumptions are documented. Advancing o Predictive models for key transportation performance measures are validated based on multiple cycles of application. o Targets are established based on predictive analysis relating revenues and programmed work to performance results. o Data mining is conducted to support “back-casting”—which involves starting with a future vision and analyzing current and historical data to estimate changes required to move from the current situation to the future vision. o Cooperative arrangements across agencies have been established to transform data into information (e.g., the state DOT performs analysis of travel-time reliability, computes measures for each facility, and provides the data for use by MPOs and local agencies). o Predictive analytics and machine learning techniques are applied for predicting asset failure probabilities and other performance measures. Do’s and Don’ts Do: Begin to analyze and visualize available data to help uncover potential quality issues. Document your analysis process so that someone else could trace your steps in the future. Explore 3rd-party tools to get your agency up and running quickly with data analytics capabilities.  Invest in developing in-house analysis capabilities or, if that isn’t feasible, engage consultants that specialize in data analysis. Treat your consultants as trusted team members. Don’t:  Wait until all of your data is “cleaned” or perfect before beginning an analysis.  Build an analysis tool in-house unless you are sure about what you are getting into and you are confident that you will have staff to support the tool in the future.  Be so detailed and stringent in your analysis requirements that costs escalate out of control.  Just go with your current on-call consultant or low-bid contractor (if they aren’t data analysis experts). 59

Introduction • Foundation • Reporting • Insight • Cases Step 6 Present & Communicate Data The Present & Communicate step involves developing effective ways of communicating the message and story behind the data. The process of communicating performance results is likely to lead to questions about the data and analysis. Data analysts should anticipate that there will be iteration between the communication and analysis steps. The need for data improvement or augmentation may also be identified as new questions arise. Over time, these improvements will strengthen the agency’s ability to make effective use of data to improve performance. “The greatest value of a picture is when it forces us to notice what we never expected to see.” John Tukey 60

Introduction • Foundation • Reporting • Insight • Cases Step 6.1 Develop & Communicate Performance Stories Tell the story. Once data are successfully translated into information, it is important to provide context and the “so what” surrounding that information. One of the most effective ways to accomplish this is through a storytelling approach. Information consumers (the audience) must buy into the story for the information to be effective. The focus of the story must not be on data and information, but on the message that the information is supporting. For example, when the New Jersey Department of Transportation and the Delaware Valley Regional Planning Commission were trying to convey the importance of specific roadway projects to senior managers and public officials, they were challenged to communicate about complex performance measures related to reliability, safety, congestion trends, economic impacts, and more. After many unsuccessful attempts at producing thorough reports for decision makers, they tried an information visualization approach. They developed “elevator pitch” brochures that conveyed, primarily through graphics, the performance measures related to individual projects. The visualizations contained within the brochures could be easily interpreted by both engineers and the public. Accompanying narratives were short, and the brevity of the brochures meant that more people ultimately read and understood the message. When making a new investment in sensor infrastructure to support TPM, the message should not be that 200 more sensors will provide more data about congestion. Instead, the message should be that the new sensors will allow operators to more quickly identify traffic slowdowns, which will enable signal timing changes to ensure that inbound commuters make it to work on time. The audience must be able to relate to the outcomes and understand how they are being affected by changes in performance of the system. In 2012, an article in Governing magazine reported, “The [Gray] Notebook tells Washington citizens pretty much whatever they might want to know about how their transportation system is working.…The first Gray Notebook—as it came to be called because of the color of its cover—was published in 2001, and legislators loved it. Two years later, those legislators approved a 5-cent increase in the gas tax to fund new transportation projects.” New Jersey DOT developed project assessment summary pamphlets that tell a compelling story about how investment in a project benefitted the general public. Case H 61

Introduction • Foundation • Reporting • Insight • Cases Communicate with visualizations. Information visualization is critical in supporting successful data presentation and communication; visualization is used to help people understand and interpret the data. Human perceptual skills are remarkable—we can identify trends, clusters, gaps, and outliers extremely quickly. The presentation of data and information in a visual way should be easily processed to identify the primary message. Some of the visualizations frequently used in TPM due to their simplicity and effectiveness include heat maps, timelines, choropleths, arc diagrams, etc., but there are hundreds of visualization strategies, methods, and devices that can be leveraged, depending on the analysis or communication need. But just as visualizations can be powerful in communicating a message, they can also often be misleading. It is critically important to be consistent in presenting information to avoid introducing a bias to the user. There are many known data visualization approaches that can skew the message—truncating Y-axes, omitting data, correlating causation, and many others. In order to present information in the most objective way, it is important to provide the methodology, definitions, metadata, and any other contextual information. For more information... 1. Communicating Performance (AASHTO) http://communicatingperformance.com/ 2. The Colorado Transportation Story https://www.codot.gov/programs/colorado-transportation- matters/statewide-transportation-plans/statewide-transportation- plans 3. Visualization and Communication in Pavement Performance (Midwest Transportation Center, 2018) https://rosap.ntl.bts.gov/view/dot/36445 4. NCHRP Web-Only Document 226: Data Visualization Methods for Transportation Agencies (TRB, 2016) https://www.nap.edu/catalog/24755/data-visualization-methods- for-transportation-agencies 5. “Truth, Transparency and Transportation,” Governing Magazine, September 13, 2012 http://www.governing.com/blogs/bfc/col- washington-state-transportation-gray-notebook-transparency.html 6. Tufte, E. The Visual Display of Quantitative Information (Graphics Press, 2001) Arizona’s 2040 Long-Range Transportation Plan broke new ground in its methods for aligning Arizona stakeholders’ priorities with competing 25-year transportation needs. ADOT innovated by using a multi- objective decision analysis software platform with intuitive visual elements like slider bars, dashboards, and data visualizations that helped stakeholders see the consequences of alternate funding choices on performance outcomes. Case A 62

Introduction • Foundation • Reporting • Insight • Cases Capabilities Checklist: Present & Communicate Data Basic o Managers and analysts meet to review and interpret performance results. o Story lines for performance results are developed, reviewed, and communicated. o Training is offered to internal staff to build skills in data presentation and communication. o Staff have capabilities to present data in a variety of formats tailored to the needs of different audiences, including heat maps, thematic maps, timelines, and other infographics. o A combination of narrative and graphical presentation is used to communicate performance information. Advancing o Feedback from data consumers is sought and used to improve communication of information to different target audiences. o Individuals with expertise in data visualization and communication are available to support development of performance data products. o Social media is used to communicate key results or draw people to more detailed communication products. o Specialized visualization and analysis environments have been developed—e.g., virtual reality simulators. Do’s and Don’ts Do: Leverage visualization tools for your data analysis and for communicating TPM to the public and decision makers. Employ “best practices” in visualization that aim to communicate with users, not deceive them. Leverage 3rd-party visualization tools and/or professionals to support your analysis. Use visualization to support your narrative. Don’t:  Wait until the end of your project to begin to interpret the results.  Use all the bells and whistles in a chart or visualization tool; clean and simple graphics tell compelling stories.  Try to be too complicated with your visualizations; you’re trying to tell a story, not confuse people.  Expect visualization alone to tell your story; some supplemental explanatory text will be required. 63