A Methodology for Performance Measurement and Peer Comparison in the Public Transportation Industry (2010)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

69 Value of Peer Comparison and Benchmarking The literature review summarized in Chapter 2 showed that peer comparison and benchmarking are commonly used man- agement tools in other industries, including portions of the public sector. The integration of benchmarking into many of the Malcolm Baldrige National Quality Award’s performance criteria speaks to its long-term value as a core business practice. During the last half of the 1990s and the first decade of the 21st century, benchmarking has been a focus of European efforts to improve the quality of transit service delivered to customers. At least four international benchmarking net- works, funded by member contributions, have been in exis- tence for extended periods of time; their longevity also speaks to those agencies’ belief in the value of benchmarking. Despite having a significant advantage not available to tran- sit agencies in most of the rest of the world—the existence of a database of relatively standardized transit data in the form of the NTD—the U.S. transit industry has been slow to adopt benchmarking as a business practice. The development of this report and the incorporation of its peer-grouping methodol- ogy into the FTIS software tool will hopefully remove a barrier to its adoption. However, additional work will be required for the U.S. public transit industry to fully realize benchmarking’s potential. This issue is discussed next. Key Findings and Conclusions Transit Agencies • NTD data quality. Each transit agency has a responsibility to make sure that its NTD data are reported consistent with the NTD definitions. Some transit agencies view NTD reporting as a burden, and this project’s outreach efforts identified that industry confidence in NTD data quality was still lacking. However, as the Chapter 5 case studies show, agencies can quickly obtain a number of useful insights about their own performance by using FTIS to work with NTD data. However, consistent data collection is essential for a transit agency’s reporting effort to generate any value. • Performance improvement. Benchmarking can validate an agency’s strengths and may reveal opportunities for improvement; in either case, the approach from the start should be that the agency is committed to looking for (and implementing) ways to improve its performance. Many suc- cessful benchmarking applications start with the premise that everyone has room for improvement. A performance comparison should be a starting point for asking ques- tions, and looking for ways to improve one’s performance is preferable to an exercise where a performance “report card” is the final outcome. • First steps. The FTIS tool makes it possible to quickly form a peer group and retrieve NTD-based performance data. A small-scale peer-grouping exercise focusing on key cost and productivity outcomes can demonstrate the value of performing peer comparisons and can help build internal agency support for a larger-scale, permanent effort. • Permanent internal performance-measurement pro- gram. As demonstrated by some of the Chapter 5 case studies, an initial NTD-based performance comparison can lead to insights and questions that require more- detailed internal data. Having a structure set up to routinely collect, analyze, report, and store this information will both support a transit agency’s day-to-day activities as well as support less-frequent benchmarking efforts. TCRP Report 88 (1) provides guidance on developing such a program. The more transit agencies there are that have good perfor- mance-measurement programs, the easier it will be for all to share and obtain non-NTD performance data. • Success stories. Early adopters of benchmarking, such as members of the TFLEx benchmarking network in the United States and individual transit agency general man- agers who have incorporated peer comparisons into their agency activities, can spread the word about the tangible C H A P T E R 6 Concluding Remarks

(e.g., performance improvements) and less-tangible (e.g., staff professional growth) benefits of their work. Forums include professional conferences, committees, and work- shops; the APTA General Manager’s Workshop; and work- ing groups of transit agencies that have decided to form a benchmarking network. Transit agencies that have implemented and use benchmarking and performance- measurement programs could also consider applying for the Malcolm Baldrige National Quality Award. • Benchmarking and performance-measurement cham- pions. Management support is necessary to provide the resources to begin a program, to protect those resources during tight financial times, and to implement programs designed to improve performance. Sharing past success sto- ries, internal and external, with transit agency stakeholders (e.g., the agency board, lower-level management, and front- line employees) can help build long-term support for a pro- gram and actions that are developed as a result of having the program. State and Regional Transportation and Funding Agencies • Local transit agency NTD data. The public transit func- tion within these agencies should be familiar with local transit agencies and should know whether a change in a performance trend is due to something that has changed locally or whether it is a sign of a possible data problem. Some states, such as Texas and Florida, contract with univer- sities to check NTD data and provide training in areas where data problems occur. In addition, for those state DOTs that incorporate performance results into grant-allocation for- mulas, having a data-checking process will help in obtain- ing transit agency acceptance that the data used by the distribution process are reliable. • Training efforts. If the state DOT’s review of its transit agencies finds that many are lagging their out-of-state peers in particular areas, the state can use this informa- tion to develop training activities in those areas that will benefit a large number of agencies. • Transit agency benchmarking programs. The North Carolina DOT, for example, has developed a benchmark- ing guidebook (34) for use by its state’s transit agencies. This activity helps support the regional or state funder’s goal of having its transit agencies serve riders efficiently and effectively and helps ensure that public money directly provided by the state is used responsibly. Funding agencies could consider providing incentives each year to local tran- sit agencies that have developed and use such programs. • Annual reports on transit performance. These reports can highlight performance-improvement success stories and the need for action in certain areas (such as dealing with aging infrastructure). These reports can also incorporate non-NTD measures that are of interest at a regional or state level, providing an additional information source that benefits all. The Washington State DOT’s annual public transportation report (35) is a good model. • Service area population and size values. This research has shown the value of using per-capita performance measures and the desire of practitioners for reliable service area data. However, tracking regional population is not a normal tran- sit agency function, and as a result the service area popu- lation and size values are not reported consistently to the NTD. MPOs, on the other hand, have the data and tools to readily perform these calculations. Standards Development • Standard definitions for important non-NTD measures. APTA serves as a standards development organization for the U.S. transit industry and is a logical organization for developing standard definitions of measures (such as those relating to transit reliability) that relate to customer quality of service and that would be useful to benchmark but are not available through the NTD. Such standards are more likely to be accepted when they are a result of a transit industry consensus. • Performance measurement and benchmarking as stan- dard practices. Defining standard practices for perform- ance measurement and benchmarking that transit agencies can routinely undertake would elevate their prominence within the industry. TCRP research provides the tools for implementing such programs. National Transit Database • Transit industry outreach. Industry confidence in the qual- ity of NTD data is crucial for obtaining support for conduct- ing benchmarking efforts and taking actions based on their results—if transit agency management is not convinced of the NTD’s data quality, they will not devote resources to an effort that relies on that data. This research’s outreach efforts found that there is still considerable skepticism in the transit industry about the reliability of NTD data, while this research’s testing efforts found that, for the most part, the data needed for an analysis were reliable and that what errors did exist were readily spotted. • NTD data quality. NTD measures that most frequently had errors during the course of this research’s testing efforts were service area population, service area size, vehicle system fail- ures, and route miles vs. track miles. Each would be valuable for developing outcome or descriptive measures as part of a benchmarking effort if the data quality could be improved. Future Steps • Pilot benchmarking network projects. International experience shows that benchmarking networks can pro- vide greater knowledge benefits and cost-sharing oppor- 70

tunities than when individual agencies conduct their own peer-comparison activities. However, that experience also shows that having an external organization to manage the data collection and analysis process and to facilitate the exchange of information within the network is an impor- tant long-term success factor. Pilot projects could: – Recruit agencies of various sizes and/or modes operated that already have established performance-measurement programs to participate in a benchmarking network. Hav- ing a performance-measurement program already in place (a) demonstrates a transit agency’s commitment to performance measurement and (b) reduces the time required to begin benchmarking and demonstrate results. – Fund a facilitator for each network for the first few years. Since several pilot European networks dissolved after the pilot funding ended, it would be important for pilot projects to seek ways to minimize costs, while still pro- viding real benefits, to maximize the potential for the benchmarking network continuing on its own after the pilot period. – Document success stories (i.e., tangible and intangible benefits realized) from the pilot networks to encourage greater use of benchmarking by others. • Confidential clearinghouse for performance data. The lit- erature review found that many organizations, both inside and outside the transit industry, are more willing to share data and practices when they are assured that the data will be kept confidential. The clearinghouse could contain stan- dardized non-NTD measures of value to benchmarking efforts as well as more-detailed versions of NTD measures (e.g., summarized by service type—commuter bus, suburban bus, BRT—rather than by mode). Transit agency participa- tion would be voluntary, but only contributors to the clear- inghouse would be able to access data. Accomplishment of Research Objectives Table 27 lists the original objectives of the research and how they have been accomplished. 71 Research Objective How the Objective Was Accomplished The methodology should include performance measures composed of uniformly reported data that are as transparent as possible, credible, and relevant to the concerns of public transportation systems. The peer-grouping methodology incorporates uniformly reported data from the NTD, the U.S. Census Bureau, and the Urban Mobility Report (45), as well as data developed by this research. The performance-measurement component of the methodology incorporates uniformly reported inflation and labor-cost data from the U.S. Bureau of Labor Statistics and transit data from the NTD. Guidance is provided on obtaining uniform non-NTD data and checking for data consistency. For transparency, the inputs to the peer-grouping process are provided with the peer-grouping results in FTIS; the entire methodology is described in Appendix B. The methodology underwent two rounds of review with the project panel and industry stakeholders, followed by two rounds of real-world transit agency testing. The peer comparison approach should enable performance assessments of public transportation systems of different sizes, operating environments, and modes. The methodology accommodates assessments of any non-rural transit system operating any mode reported to the NTD. Performance can be compared for the agency as a whole or for an individual mode. Operating environment peer-grouping process. Guidance is provided on screening (service area type) is a factor used directly in the for other factors, such as the agency’s operating philosophy (e.g., coverage vs. efficiency). The research should consider lessons learned from other industries and from international transit peer- comparison experience. These lessons learned are summarized in Chapter 2 of this report. The research should identify potential applications for the methodology and develop potential strategies for industry adoption of the methodology. Applications are summarized in Chapter 3, while recommended strategies are provided in Chapter 6. The methodology should be able to be applied not only by individual public transit agencies but also by state departments of transportation and other transit funding agencies. Five state DOTs and the Chicago Regional Transportation Authority were included in the methodology testing. Two of the case studies in Chapter 5 demonstrate DOT applications. Table 27. Accomplishment of research objectives.