Benchmarking and Comparative Measurement for Effective Performance Management by Transportation Agencies (2019) / Chapter Skim
Currently Skimming:
Pages 81-142
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 81... ...
81 Part 3. Benchmarking Pilot Results
|
From page 82... ...
Effective Performance Management by Transportation Agencies Benchmarking and Comparative Measurement Pilot Purpose • Summary • Results 82 Purpose of the Benchmarking Pilots The principles developed for this project's guidance were implemented in two pilots of the network approach to benchmarking. These pilots fine-tuned the guidance principles, generated more robust lessons learned, and provide specific examples of how to implement network benchmarking for interested practitioners and agencies.
|
From page 83... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Summary Pilot Purpose • Summary • Results 83 Pilots Summary Details of the two pilot experiences are outlined below using the benchmarking guidebook's first six steps. A high-level overview of the outcomes at each step and a summary of lessons learned are provided to help orient readers as they explore the detailed content more closely.
|
From page 84... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Summary Pilot Purpose • Summary • Results 84 Step 1. Set the Stage The project team planned a benchmarking network approach for the pilots.
|
From page 85... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Summary Pilot Purpose • Summary • Results 85 Step 2. Select Peer Agencies The project team reached out to 10 to 12 practitioners in different states who were expected to be interested in each performance area based on input by subject matter experts who attended a workshop during project development.
|
From page 86... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Summary Pilot Purpose • Summary • Results 86 Step 3. Define the Approach Defining the approach requires getting participants to agree on specific metrics, data sources, definitions, and parameters.
|
From page 87... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Summary Pilot Purpose • Summary • Results 87 Step 4. Obtain Data Effective benchmarking depends on access to quality data.
|
From page 88... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Summary Pilot Purpose • Summary • Results 88 Step 5. Analyze Data Analyzing the data often requires initial cleaning and formatting.
|
From page 89... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Summary Pilot Purpose • Summary • Results 89 Step 6. Identify Noteworthy Practices Data analysis should lead to conversations about how top performers achieve results.
|
From page 90... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Summary Pilot Purpose • Summary • Results 90 Summary of Key Implementation Lessons The project team learned several implementation lessons from conducting the pilots. Some of these lessons are not surprising, but they all serve as useful reminders and aids for anyone attempting a similar network approach to benchmarking.
|
From page 91... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Summary Pilot Purpose • Summary • Results 91 Interest in a given performance area often is not uniform from one state to another. When establishing benchmarking performance areas and identifying peers, a particularly important measure for one set of states may have little relevance to others.
|
From page 92... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Summary Pilot Purpose • Summary • Results 92 Allow for formal and informal peer interactions. Throughout the process, but especially during scheduled practice exchange calls or meetings, allow for a mix of informal, organic interactions among peer participants, along with programmed presentations and information exchanges.
|
From page 93... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results 93 Pilot Results The benchmarking pilots for environmental and nonmotorized performance implemented the first six steps of the benchmarking process outlined in this project's guidebook. Details of their activities and outcomes are presented step-by-step.
|
From page 94... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Set the Stage 94 Step 1. Set the Stage Key Actions to Set the Stage • Identify performance area(s)
|
From page 95... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Set the Stage 95 Environmental Performance The team relied on NCHRP Report 809: Environmental Performance Measures for State Departments of Transportation for its recommendations on environmental performance measures. One of the core challenges for making environmental performance comparisons is finding measures that can be applied across all state DOTs.
|
From page 96... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Set the Stage 96 Final environmental performance areas -- The research team and workshop participants chose two performance measure areas: a form of NEPA documentation timeliness and wildlife– vehicle collisions. These measures were chosen based on interest among at least a handful of states, expected data availability, and "benchmarkability" -- the capacity to share specific strategies for improvement among a small group of interested practitioners.
|
From page 97... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Set the Stage 97 Nonmotorized Performance Several categories of potential nonmotorized performance areas were proposed, including demand, network connectivity, safety, and community benefits. The research team initially recommended exploring measures related to demand and safety because data in these areas are more consistent and readily collectable than data for other categories, for which definitions and informationgathering practices vary among states.
|
From page 98... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Set the Stage 98 Table 5. Progression of Recommended Performance for Nonmotorized Benchmarking Pilot Table 6.
|
From page 99... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Set the Stage 99 Step 1 Takeaways • Very few measures relating to the environment and nonmotorized performance will have appeal across all 50 states. The smaller scale of benchmarking networks allows states with common interests to discuss performance of less widely applicable performance topics.
|
From page 100... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Select Peer Agencies 100 Step 2. Select Peer Agencies – From the Guidance Key Actions to Select Peer Agencies • Identify relevant peer selection criteria for each performance area • Collect criteria data or conduct peer research to identify appropriately similar states • Recruit participants (benchmarking network)
|
From page 101... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Select Peer Agencies 101 In the second scenario, that is, when benchmarking for a measure for which data are not likely to be uniformly available across states, the mere existence of data or a specific program may be the most important element. In these cases, which will include many benchmarking networks, research and talking with subject matter experts, rather than data collection, will be needed to find peer agencies likely to participate.
|
From page 102... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Select Peer Agencies 102 Environmental Performance Peer States for Wildlife Collisions Measure For the wildlife–vehicle collisions pilot, the team targeted states that had active wildlife collision–reduction programs, with the expectation that these states would most likely be interested and have the necessary data. The panel's subject matter expert suggested several states that had data collection programs associated with wildlife collision–reduction efforts.
|
From page 103... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Select Peer Agencies 103 Peer States for NEPA Document Timeliness Workshop attendees expressed interest in a benchmarking initiative targeted to NEPA document time frames, but no panel members had the domain-specific knowledge and connections to champion the topic. The research team therefore decided to reach out to FHWA staff for guidance on the likely availability of data, parameters that made sense for a measure, and state contacts to participate in the pilot.
|
From page 104... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Select Peer Agencies 104 via a national list serve for state bike and pedestrian coordinators. This list, which provided a targeted opportunity to reach interested practitioners, was available due to this participant's topic expertise.
|
From page 105... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Select Peer Agencies 105 Step 2 Takeaways • Selection criteria based on the characteristics of different agencies are helpful in reducing a large field of potential benchmarking peers to those most appropriate for a given performance area. However, when benchmarking on a niche topic, peer agencies may simply be those agencies that are willing to participate and have the data needed to compare performance.
|
From page 106... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Define the Approach 106 Step 3. Define the Approach Pick one or more measures related to the selected performance area that are suited to comparing performance among agencies or groups and come to agreement on relevant details and definitions.
|
From page 107... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Define the Approach 107 Environmental Performance Wildlife–Vehicle Collisions For the wildlife–vehicle collision measure, the research team held a conference call with practitioners from five of the six interested agencies. The sixth participant was unable to join the call but committed to future aspects of the project.
|
From page 108... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Define the Approach 108 corridors might not "move the needle" when looking at statewide numbers. This discussion led to a measure the project team did not originally consider: percentage reduction in wildlife strikes on a corridor after project implementation.
|
From page 109... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Define the Approach 109 Nonmotorized Performance The connectivity pilot was kicked off via a conference call with all participants except the Utah DOT (UDOT)
|
From page 110... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Define the Approach 110 Step 3 Takeaways • Have a facilitator or participant draft initial measure options to provide a starting point for a group conversation and avoid aimless discussion. • Final measure parameters may be informed by the data that are available, rather than the other way around.
|
From page 111... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Obtain Data 111 Step 4. Obtain Data Effective benchmarking depends on access to quality data.
|
From page 112... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Obtain Data 112 Environmental Performance Data Requested • Annual crash and carcass counts for the project area, including distance along the corridor just outside the project to account for "end effects" The wildlife collisions pilot group selected a measure -- percentage reduction in annual wildlife–vehicle collisions after project implementation -- for which most of the participants already had access to data. Of the six agencies that agreed to participate in the wildlife pilot, five confirmed they would have data to contribute, with only California unable to provide any.
|
From page 113... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Obtain Data 113 Nonmotorized Performance Data Requested • OSM road network • Highway Performance Monitoring System roadway ownership Bike and pedestrian data are limited at the national level, and individual state and MPO data are so varied that comparable metrics are a challenge. For this reason, the nonmotorized pilot participants decided to generate the data needed for a measure that held interest for them.
|
From page 114... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Obtain Data 114 Step 4 Takeaways • Data gathering and assembly can be time consuming. Facilitators or task leads should expect to follow up several times with participants before having all necessary data, as they did in the wildlife pilot.
|
From page 115... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Analyze Data 115 Step 5. Analyze Data Data cleaning is almost always required before the data can be analyzed.
|
From page 116... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Analyze Data 116 Environmental Performance Data Cleaning and Formatting Although the chosen performance measure required only a single number for the before-construction period and a single number for the after-construction period, breaking the results down to annual trend data provided the best means to clearly see the effect of agency interventions on wildlife collisions. The facilitator sought to obtain this annual data from all participants.
|
From page 117... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Analyze Data 117 Should collisions or carcass numbers be used? Carcasses were selected as the better option, as there are often too few crash records from which to interpret reliable trends.
|
From page 118... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Analyze Data 118 Utah Performance Measure Result: 91% Reduction Figure 4. Utah DOT wildlife carcass count results Pre-project annual average: 106.0 Post-project annual average: 9.3 Project -- I-15 Wildcat Fencing Project (2004)
|
From page 119... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Analyze Data 119 Washington Performance Measure Result: 43% Reduction Figure 5. Washington State DOT wildlife carcass count results Pre-project annual average: 12.75 Post-project annual average: 7.33 Project -- Primarily a project to correct fish barriers in waterways, but with terrestrial wildlife habitat connectivity enhancements including • Three wildlife guards • Six jumpouts (places for wildlife to "jump" from the roadway to a safer location if vehicles approach while they are trapped in the fenced area)
|
From page 120... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Analyze Data 120 Montana Performance Measure Results: 5.2% (without Control) and 71.4% (with Control)
|
From page 121... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Analyze Data 121 Colorado Performance Measure Result: 86% Reduction Figure 7. Colorado DOT wildlife carcass count results Pre-project construction annual average: 56 Post-project construction annual average: 8 Project -- State Highway 9 Colorado River South Wildlife and Safety Improvement Project • Two wildlife overpasses, five underpasses, 8-foot-high exclusion fencing, 61 wildlife escape ramps, and 29 deer guards over 10.4 miles • Phase 1: December 2015; Phase 2: December 2016 Context -- Prior to the project, wildlife–vehicle collisions were the most common accident type on this segment of highway, accounting for 60% of all accidents reported to law enforcement.
|
From page 122... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Analyze Data 122 Nevada Performance Measure Result: In Progress Figure 8. Nevada DOT horse mitigation project crash records Project -- U.S.
|
From page 123... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Analyze Data 123 Nonmotorized Performance Data Processing and Formatting For the sake of efficiency, the team scripted the analysis process to calculate the route directness index (RDI) for the nonmotorized connectivity pilot by using various freely available open source software packages along with custom code written for this project.
|
From page 124... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Analyze Data 124 The following steps were computed for each segment: 5. Defined "offset points" to either side of the segment.
|
From page 125... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Analyze Data 125 Figure 10. Example of a barrier with three 500-meter segments Results Analysis In the preliminary analysis results, state highways were defined by the OSM "motorway" tag.
|
From page 126... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Analyze Data 126 The following conclusions can be drawn from Table 13: • State highways in Kansas are the most permeable to bicyclists and pedestrians. The least out-of-direction travel occurs outside of MPO boundaries.
|
From page 127... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Analyze Data 127 Figure 11. Permeability analysis results -- Kansas
|
From page 128... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Analyze Data 128 Figure 12. Permeability analysis results -- Minnesota
|
From page 129... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Analyze Data 129 Figure 13. Permeability analysis results -- New Mexico
|
From page 130... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Analyze Data 130 Figure 14. Permeability analysis results -- Utah
|
From page 131... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Analyze Data 131 Figure 15. Permeability analysis results -- Vermont
|
From page 132... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Analyze Data 132 Figure 16. Permeability analysis results -- Washington
|
From page 133... ...
Effective Performance Management by Transportation Agencies Benchmarking and Comparative Measurement Benchmarking Pilots Results Pilot Purpose • Summary • Results: Analyze Data 133 Table 14. Summary of Roadway Segments and Crossing Availability
|
From page 134... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Analyze Data 134 Table 14 summarizes roadway segments and crossing availability. This information allows an analyst to better understand the impact of locations where no roadway crossing is available.
|
From page 135... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Analyze Data 135 considerably higher within urban areas. In this case, the decision to exclude routes where travel is possible but the RDI is greater than 10 (which happens frequently outside MPO boundaries)
|
From page 136... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results Pilot Purpose • Summary • Results: Analyze Data 136 Step 5 Takeaways • After data have been obtained, reviewing for quality and completeness is critical. If review of the data reveals deep flaws, it may be necessary to complete additional processing steps or search for alternative data sources.
|
From page 137... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results 137 Step 6. Identify Noteworthy Practices Data analysis should lead to conversations with top-performing agencies about the ways they achieve results.
|
From page 138... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results 138 Environmental Performance To ensure there was no "dead air" on the practices exchange conference call, the facilitator prepared an agenda that included a review of participants' performance data, designated topics for group discussion, and a formal presentation on a noteworthy practice by one of the participants. Performance review -- The practices sharing call began with a review of the projects and data each participant had submitted.
|
From page 139... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results 139 having enough staff to fully meet laborintensive work tasks, such as accounting for all carcasses from wildlife–vehicle collisions or identifying facilities and infrastructure that have been damaged and need maintenance. One solution, implemented in various forms at several participating agencies, involves recruiting the help of groups and individuals who are already in the areas near agency right-of-way, such as hunters, recreationalists, and other interested citizen groups.
|
From page 140... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results 140 Nonmotorized Performance Rather than identifying high-performing agencies, the practice exchange for the nonmotorized pilot focused on how to make the connectivity metric most useful. This focus was due, in part, to the newness of benchmarking bicycle and pedestrian metrics at the state DOT level and the lack of experience by the participants in comparing their state to others in a datadriven process.
|
From page 141... ...
Effective Performance Management by Transportation Agencies Benchmarking Pilots Benchmarking and Comparative Measurement Results 141 Step 6 Takeaways • Discussing the data and results as a group provided a more nuanced and robust understanding of relevant practices and performance outcomes. In particular, impromptu topics that the facilitator has not thought of but that are relevant and interesting to practitioners can arise in group discussions.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.