Benchmarking and Comparative Measurement for Effective Performance Management by Transportation Agencies (2019)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

61 Part 2. Transportation Comparative Benchmarking Platform

Effective Performance Management by Transportation Agencies Comparative Benchmarking Platform Benchmarking and Comparative Measurement Platform Overview • Purpose • Use Types • Audience • Modules • Visual Tour 62 Overview of the Guidebook’s Companion Digital Benchmarking Platform The digital Transportation Comparative Benchmarking Platform, developed in coordination with this guidance, supports independent benchmarking or benchmarking networks. In an independent benchmarking scenario, individual agencies can set peer grouping comparisons of their own choice, evaluate their performance based on measures derived from publicly available national data sources, and use the results to reach out to peer agencies and learn noteworthy practices. In a benchmarking network, a tightly connected group of peers can populate the platform with self-reported data. In this situation, the platform acts as a specialized information-sharing channel that allows for efficient and effective comparison of performance and exchange of noteworthy practices that enable learning. To support benchmarking networks, custom benchmarks may be defined by request to the site administrator in the demonstration version of the benchmarking platform. Member agencies are granted access to view and contribute network measure data. Regardless of whether the platform is used for independent benchmarking or a network, it can be used to support a range of activities. Platform Overview

Effective Performance Management by Transportation Agencies Comparative Benchmarking Platform Benchmarking and Comparative Measurement Platform Overview • Purpose • Use Types • Audience • Modules • Visual Tour 63 Benchmarking Platform Purpose Benchmarking should emphasize awareness and discussion as the primary outcome; number crunching should be viewed as a means to an end, not the primary focus. The benchmarking platform is a space to achieve this goal. Transportation Comparative Benchmarking Platform http://benchmarking.tpm-portal.com/ Both data and a means to display and compare performance are needed for benchmarking. Performance results should be easily sharable, and information about lessons learned should trigger discussions among practitioners. The Transportation Comparative Benchmarking Platform was developed to facilitate these aspects of the benchmarking process specifically for state DOTs. It is a place where data can be stored; performance can be explored, visualized, and compared; and conversations about successful practices can take place. The platform enables users to quickly compare their agency’s performance relative to peers on predetermined measures. It provides a centralized and standardized way of viewing performance on multiple transportation performance measures. The platform supports (1) peer analysis (comparing a given agency’s performance against its peers), (2) trend analysis (the same agency’s performance over time), and (3) peer trend analysis. Most importantly, given the intent of comparative benchmarking is learning and improvement, the platform can facilitate exchange of information among platform users. To achieve this exchange, the platform includes a Noteworthy Practice Narratives feature that lets user agencies offer assessments of the factors contributing to their performance. This feature was designed to allow inquiry and sharing and to foster high-quality communication of practices and experiences between agency users. Any user can register for access to try these functionalities by visiting http://benchmarking.tpm-portal.com/. Purpose

Effective Performance Management by Transportation Agencies Comparative Benchmarking Platform Benchmarking and Comparative Measurement Platform Overview • Purpose • Use Types • Audience • Modules • Visual Tour 64 Platform Use Types The digital benchmarking platform can be used to support the activities listed in Table 1. Table 1. Digital Benchmarking Platform Use Types Use User Description Define peer groups Agency executive, analyst, or planner • Identify peer agencies based on socioeconomic and travel characteristics • Identify peer agencies based on similar performance measure scores • Select set of peer agencies and save for use in future queries View and analyze performance measure data Agency executive, analyst, or planner • View state or agency performance rank relative to peers for available measures • Review state or agency performance trend information relative to peers • Analyze performance measure values relative to peer agencies • Review measure definitions • Review data set metadata • Download data in spreadsheet format for further analysis • Save graphs and charts for incorporation into reports and presentations Solicit and review noteworthy practice narratives Agency user • View existing noteworthy practice narratives • Send requests for agencies to provide noteworthy practice narratives • Receive notifications when requests are fulfilled Get support Any • Request assistance • View help documentation Use Types

Effective Performance Management by Transportation Agencies Comparative Benchmarking Platform Benchmarking and Comparative Measurement Platform Overview • Purpose • Use Types • Audience • Modules • Visual Tour 65 Intended Audience The platform is intended as a benchmarking tool for transportation agency users, particularly midlevel to senior-level managers and practitioners. The purpose of it is to foster collaborative discussion with the objective of continuous improvement among the actual practitioners who support transportation agencies in achieving identified performance goals. To allow the platform to simultaneously serve as a publicly accessible guidance companion and as a functioning platform for actual agency users, the platform protects some content and functionality behind a log-in screen. All user-generated content, including noteworthy practice narratives and measures based on data not derived from publicly available national sources, is protected behind a log-in. Only registered and authenticated users will be able to view user-generated or agency- supplied platform content, request or provide noteworthy practice narratives, or upload data. Interested users can register at the platform’s homepage. In practice, the platform would have one assigned user for each of the 52 state DOTs. As the platform gains adoption, it is expected that active DOTs will have numerous authorized users. For demonstration purposes, the platform will be accessible to the panel members and to the states participating in the environmental performance and nonmotorized performance pilot measures. Audience

Effective Performance Management by Transportation Agencies Comparative Benchmarking Platform Benchmarking and Comparative Measurement Platform Overview • Purpose • Use Types • Audience • Modules • Visual Tour 66 Core Platform Modules The platform’s functionality is provided through a suite of modules that facilitate the major steps of the benchmarking process. Peer Selection Agencies should benchmark against appropriate peers. This module allows the user to define meaningful peer comparison groups and to use these groups throughout the platform. Agencies are ranked based on their similarity across a range of relevant criteria (e.g., population, network size, performance scores). Users can select the most relevant criteria for a particular performance area or rely on prepopulated defaults. Users select final groups from a ranked list of available peer agencies. Measures and Comparison Benchmark measures are grouped under benchmark measurement groups. Performance measure groups can be defined by topic (e.g., safety) or contain a set of measurements specific to a benchmark network. Actual performance comparison is completed in the measure comparison module, which allows the user to view the agency’s performance compared to a peer group, the agency’s performance over time, or the peer comparison over time. Noteworthy Practice Narratives The intent of comparative benchmarking is to identify areas that need improvement and then learn from peers that perform well in that area. The platform’s noteworthy practice narrative functionality provides an easy, contextualized, and interactive method of sharing and exploring practices that result in performance improvements at agencies. Noteworthy practices are embedded in the measure comparison interface, allowing users to view or request noteworthy practice narratives while exploring performance outcomes. A user who wants to know more about a peer agency’s performance selects a data point and can send a request to the agency for more information about how it achieved its performance. Account Management Platform users will use the account management module to view and manage their noteworthy practice requests and narratives, view system-generated messages, upload data, and manage their preferences and profile. Account administrators can delegate access to new users having the same top-level email address domain name. Modules

Effective Performance Management by Transportation Agencies Comparative Benchmarking Platform Benchmarking and Comparative Measurement Platform Overview • Purpose • Use Types • Audience • Modules • Visual Tour 67 A Visual Tour of the Benchmarking Platform Key platform functions are highlighted in the following screenshots. Visual Tour

Effective Performance Management by Transportation Agencies Comparative Benchmarking Platform Benchmarking and Comparative Measurement Platform Overview • Purpose • Use Types • Audience • Modules • Visual Tour 68 Home Page The landing page is the publicly facing home page. It provides a high-level description of the platform and instructions for how to get started. It also provides access to the log-in page through which registered users gain access to protected modules of the platform. Visual Tour

Effective Performance Management by Transportation Agencies Comparative Benchmarking Platform Benchmarking and Comparative Measurement Platform Overview • Purpose • Use Types • Audience • Modules • Visual Tour 69 Measure Groups The Compare page presents the major performance areas on the platform, within which more specific measures can be accessed. Measures based on publicly available data are available to everyone, but measures from individual agencies or benchmarking networks, such as those from this project’s pilots, are only accessible to participating agencies. From here users can explore the individual measures and agencies’ performance in more detail, but it may be beneficial to first set peer groups appropriate to each performance area. Up Next Peer Selection Module Visual Tour

Effective Performance Management by Transportation Agencies Comparative Benchmarking Platform Benchmarking and Comparative Measurement Platform Overview • Purpose • Use Types • Audience • Modules • Visual Tour 70 Define Peer Groups The Define Peer Groups screen allows users to create and manage peer agencies against which they will compare their own performance. Default peer groups are provided, but these may be modified or deleted and new groups created by each user. To help users define the most appropriate peers, the platform provides similarity criteria to allow users to search for agencies with elements of particular interest. For example, a user may want to compare safety performance only against agencies with a similar level of urbanization, or with similar seat belt laws. Criteria are divided into agency characteristics, such as system size or climate, and performance outcomes for the platform’s defined measures. Users can select the peers with the highest similarity score or edit the selection manually as they review the list. Different groups can be created and saved for each performance area or any other peer groups a user wishes to examine. Visual Tour

Effective Performance Management by Transportation Agencies Comparative Benchmarking Platform Benchmarking and Comparative Measurement Platform Overview • Purpose • Use Types • Audience • Modules • Visual Tour 71 Define Peer Groups: Similarity Scores By default, other agencies are sorted by similarity score. The Similarity Score field is calculated by the platform based on the normalized Euclidian distance between the values of the selected similarity criteria. Peers may be sorted by any column. As mentioned, users may include performance measure scores as similarity criteria. For example, a user could quickly search for peers with similar population and similar values for “Fatalities per 100m VMT.” The similarity function allows users to view a ranked list of the performance of all states for each criterion. This is not the intended use for either the platform or benchmarking in general, but the platform design provides this opportunity. Up Next Measures Module Visual Tour

Effective Performance Management by Transportation Agencies Comparative Benchmarking Platform Benchmarking and Comparative Measurement Platform Overview • Purpose • Use Types • Audience • Modules • Visual Tour 72 Compare Measures Once a user has selected a peer group, the compare measures module provides three data views for benchmarking: • Agency trend—Only the user’s agency performance will display on a line chart for multiple years. • Peer trends—The selected peer group will display multiple years on a line chart. • Peer comparison—A single year is isolated for comparison among the selected peer group (shown). Users can change the date range and switch between their saved peer groups from the Compare page, or they can follow the link to edit peer groups. Visual Tour

Effective Performance Management by Transportation Agencies Comparative Benchmarking Platform Benchmarking and Comparative Measurement Platform Overview • Purpose • Use Types • Audience • Modules • Visual Tour 73 Compare Measures (cont.) For peer trends comparisons, the charts display the average performance of all agencies and the interquintile range (20th–80th percentile performance) for the measure in addition to peer agencies’ performance. These performance measures and visuals, as well as any agency in the peer group, can be clicked on or off the current view. For all comparison views, the measure information is defined at the bottom of the page. If a user wishes to explore the data in more detail or use them for other purposes, raw data can be downloaded directly from the Compare Performance page. Up Next Noteworthy Practice Visual Tour

Effective Performance Management by Transportation Agencies Comparative Benchmarking Platform Benchmarking and Comparative Measurement Platform Overview • Purpose • Use Types • Audience • Modules • Visual Tour 74 Noteworthy Practices Noteworthy practice narratives are the primary practice-sharing mechanism in the platform. These narratives explain the activities or initiatives an agency undertook that resulted in improved (or worsened) performance. Users can initiate their own narratives or request them from agencies whose performance they wish to learn about. Clicking interactive nodes on the chart corresponding to the particular agency and year of interest will bring up a noteworthy practices narrative request form. Nodes with a completed noteworthy practice show a larger, unfilled marker (as shown at left) compared to smaller, solid markers for regular data points. Up Next New Narrative Request Visual Tour

Effective Performance Management by Transportation Agencies Comparative Benchmarking Platform Benchmarking and Comparative Measurement Platform Overview • Purpose • Use Types • Audience • Modules • Visual Tour 75 Noteworthy Practices: New Request Clicking on a blank node within the chart opens a window to create and send a new noteworthy practice request. A notification of the request will be emailed to the assigned agency contact with a link to fulfill the request. The requesting user will be notified when the request is fulfilled. Up Next Noteworthy Practice Narrative Example Visual Tour

Effective Performance Management by Transportation Agencies Comparative Benchmarking Platform Benchmarking and Comparative Measurement Platform Overview • Purpose • Use Types • Audience • Modules • Visual Tour 76 Noteworthy Practices: Example Clicking on a node with a noteworthy practice will bring the user to the Peer Noteworthy Practices section located below the chart. Here, all noteworthy practices will be listed for that measure–peer group combination. If users want to be informed when new noteworthy practices are added for the performance measure, they can “subscribe.” An email will be sent when any user submits a noteworthy practice for that measure. Note: The noteworthy practice shown here was created for demonstration purposes only. Up Next Account Management Visual Tour

Effective Performance Management by Transportation Agencies Comparative Benchmarking Platform Benchmarking and Comparative Measurement Platform Overview • Purpose • Use Types • Audience • Modules • Visual Tour 77 Account Management Account Management is accessed by clicking on the bell icon at the top of the web page menu. The Account Management page has three tabs. From the Noteworthy Practice Statements tab, users can view their previously submitted noteworthy practice narratives and view and fulfill requests. The Profile tab lets users update their names, department, title, and contact information. Up Next Data Submission Visual Tour

Effective Performance Management by Transportation Agencies Comparative Benchmarking Platform Benchmarking and Comparative Measurement Platform Overview • Purpose • Use Types • Audience • Modules • Visual Tour 78 Account Management: Data Submission Data submission for user-supplied performance measures is handled through the Data Submission tab on the Account Management page. Under this tab, users can manage their data that is displayed through the platform. On this page, an agency can select any performance measures to which it has access. Upon selecting them, users will be taken to a series of cells in which they can enter annual performance measure results. New data requirements (e.g., a new calendar year) will generate a notification email to platform users and will display as an unfulfilled data request until completed. Visual Tour

Effective Performance Management by Transportation Agencies Comparative Benchmarking Platform Benchmarking and Comparative Measurement Platform Overview • Purpose • Use Types • Audience • Modules • Visual Tour 79 New Benchmarking Initiative One major functionality on the Account Management page is the ability to create a new benchmarking initiative. The request form asks the user for all relevant information about the desired performance measure and sends this information to the site administrator, who can then add the measure to the site for participating agencies. Visual Tour

Additional Content In addition to the core elements of the platform functionality shown in the screenshots above, the following functionalities are built in as part of the platform: • Measure Definitions and Sources • Account: Delegate Access to Other Agency Users • Administration: Assign Agencies to Measures and Measure Groups • Administration: Define Measures, Measure Years, Manage Users • Help and Support: View help and support document, contact the site administrator (such as to request that a new benchmarking network measure group be created) Effective Performance Management by Transportation Agencies Comparative Benchmarking Platform Benchmarking and Comparative Measurement Visual Tour Platform Overview • Purpose • Use Types • Audience • Modules • Visual Tour 80