A Methodology for Performance Measurement and Peer Comparison in the Public Transportation Industry (2010)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

S U M M A R Y Performance measurement is a valuable management tool that most organizations con- duct to one degree or another. Many examples exist of report cards, dashboards, key perfor- mance indicators, and similar techniques for presenting performance results, and these are important first steps in efforts to improve an organization’s performance. Taken in isolation, however, performance measures are capable of providing tremendous quantities of data but little in the way of context. To begin to provide real value, measures need to be compared to something else—for example, one’s past performance, one’s targeted performance, or com- parable organizations’ performance—to provide the context of “performance is good,” “per- formance needs improvement,” “performance is getting better,” and so on. Once a need to improve performance has been identified, an important follow-up step is to identify and con- tact top-performing peers to learn from them and thereby improve one’s own performance. Performance measurement involves the collection, evaluation, and reporting of data that relate to how well an organization is performing its functions and meeting its goals and objectives. The measures used in the process ideally relate to the outcomes achieved by the organization; however, descriptive measures can also be used to provide context and help identify underlying reasons for changes in performance. Peer comparison is an activity where an organization compares its performance to that of similar (“peer”) organizations using a pre-determined set of performance measures. To provide meaningful results, the measures used in the comparison need to be consistently defined and reported among the different organizations included in the peer comparison. Benchmarking is the process of systematically seeking out best practices to emulate. A peer comparison provides an informative, but passive, starting point to a performance analysis, but is unlikely to explain why particular organizations are successful in particular areas. Benchmarking involves direct contact with other organizations, delves into the reasons for their success, and seeks to uncover transferable practices applicable to the organization per- forming the analysis. A performance report is not the desired end product of a benchmark- ing effort; rather, performance measurement is a tool used to provide insights, raise ques- tions, and identify other organizations from which one may be able to learn and improve. Benchmarking was first used in the private sector in 1979 and has subsequently been embraced by business leaders and become the basis for many of the Malcolm Baldrige National Quality Award’s performance criteria. It has been used in the U.S. public sector since the mid-1990s, particularly in municipal applications. Benchmarking in the public transit industry was the focus of several European research efforts in the early 2000s, and at least four international and one U.S. public transit benchmarking networks (voluntary asso- ciations of transit agencies that share data and practices with each other) now exist. Despite this track record of success, benchmarking has yet to catch on to any significant degree within the U.S. public transportation industry. Past transit agency peer-comparison A Methodology for Performance Measurement and Peer Comparison in the Public Transportation Industry 1

2efforts uncovered in this project’s literature review and agency outreach rarely extended into the realm of true benchmarking. Commonly, transit agencies have conducted peer reviews as part of transit agency or regional planning efforts, although some reviews have also been generated as part of a management initiative to improve performance. In most cases, the peer-comparison efforts were one-time or infrequent events, rather than part of an ongoing performance measurement and improvement process. One reason that benchmarking is not widely used in the U.S. public transportation indus- try is that many believe that no two transit agencies are alike and that the data available to measure transit agencies are not comparable or even reliable. Transit agencies that seem sim- ilar may have very different policy objectives or may operate in environments where public transportation has a vastly different competitive position relative to transportation alterna- tives. Such differences impact performance. However, if these and other issues could be overcome, the experience of other industries shows that benchmarking would be a valuable tool for the U.S. public transportation industry. The performance-measurement and peer-comparison methodology described in this report addresses the issues described above by incorporating a variety of nationally avail- able, standardized factors into the peer-selection process, and describing ways for also incor- porating policy objectives and other factors into the process. The methodology has been incorporated into a freely available, online software tool (the Florida Transit Information System, FTIS) that provides access to the full National Transit Database (NTD), allowing users to quickly identify a group of potential peer transit agencies, retrieve standardized per- formance data for them, and perform a variety of comparisons. During real-world applica- tions by transit agencies, users were typically able to learn how to use the software, create a peer group, and perform an analysis with 16 person-hours of effort or less. This project’s testing efforts found that, for the most part, the NTD data used in analyses were reliable and that what errors did exist were readily spotted. The range of performance questions that benchmarking and peer comparison can be applied to spans many aspects of a transit agency’s functions. Applications can be divided into the following four general categories that describe the focus of a particular comparison effort, recognizing that there is room for overlap between the various categories: 1. Administration – questions related to the day-to-day administration of a transit agency, includ- ing (but not limited to) financial-performance questions asked by agency management, agency board members, and transit funding organizations. 2. Operations – questions related to a transit agency’s daily operations. 3. Planning – long-term policy and service questions of interest to transit operators, metropol- itan planning organizations, and state departments of transportation. 4. Public and market focus – questions that consider the viewpoint of the broad range of cus- tomers, including riders, non-riders, local jurisdictions, and policy-makers. This report’s methodology does not recommend one set of performance measures as being appropriate for the entire range of performance applications that exist. During test- ing of the methodology, even when transit agencies picked identical topics to study (e.g., rel- ative subsidy levels), they selected different sets of performance measures that related to the outcomes of particular interest to them (in one case in this example, measures relating to a funding perspective and in the other, measures relating to an operating perspective). There- fore, this report provides guidance on selecting performance measures appropriate to a particular performance question and provides case study examples that can be used for inspiration, but does not prescribe a particular set of measures. This approach requires some thoughtfulness on the part of transit agencies that apply the methodology, but also provides

3much-needed flexibility that allows the methodology to be applied to a wide variety of tran- sit modes, transit agency sizes, and performance questions. The methodology was not designed as a means of ranking transit agencies to determine the “best” agencies overall on a national basis or the best at a particular aspect of service. Rather, this report’s approach is that peer-grouping and performance measurement should serve as a starting point for a transit agency to ask questions about performance, identify areas of possible improvement, and contact top-performing peers. That course—a true benchmarking process—holds the greatest potential for producing long-term performance improvement. This report describes eight steps for conducting a benchmarking effort, not all of which may be needed for a given analysis, depending on the performance question being asked and the time and resources available to conduct the effort. These steps are: 1. Understand the context of the benchmarking exercise, 2. Identify standardized performance measures appropriate to the performance question being asked, 3. Establish a peer group, 4. Compare performance within the peer group, 5. Contact best-practices peers in areas where one’s performance can be improved, 6. Develop a strategy for improving performance based on what one learns from the best-practices peers, 7. Implement the strategy, and 8. Monitor changes in performance over time, repeating the process if the desired results are not achieved within the desired timeframe. This report complements TCRP Report 88: A Guidebook for Developing a Transit Performance- Measurement System, which describes how to implement and use performance measure- ment on an ongoing basis at a transit agency.