Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
97 Introduction The purpose of Task 10, Interpret Results/Recommenda- tions, was fourfold: 1. Present and interpret the results from the Task 8 and 9 test applications of the projectâs peer-grouping and performance-measurement methodology; 2. Revise and expand, if necessary, the list of potential applica- tions for the methodology presented in Working Paper #2; 3. Provide recommendations for new standard performance measures (or modifications to existing measures) that would help support the methodology; and 4. Develop strategies for the adoption of the methodology by the transit industry. The aspects of item #1 related to results presentation were provided in the three working papers developed for Tasks 8 and 9. In addition, some initial interpretation of the Task 8 results was provided in the appendix to Working Paper #4. Also, six of the case studies from Tasks 8 and 9 that illustrate different applications of the methodology have been re-worked using the final version of the methodology and are provided in Chapter 5 of the final report. The Chapter 5 case studies provide interpretations of the case study results and guidance on how the questions that were raised by the results could be explored further. The results of item #2 are incorporated into Chapter 3 of the final report. The Task 8 and 9 testing uncovered new planning application examples related to (a) thinking to the future and what might happen when a region reached 200,000 population and its funding sources changed, and (b) comparing the performance of an agency without a dedicated local funding source to peer agencies with one. These examples have been added to the list of applications in Chapter 3. In addition, the lists of applications have been revised and reworded in response to panel comments over the course of the project. Finally, the lists of standard per- formance measures that are applicable to benchmarking applications have been reorganized into descriptive and outcome categories, with multiple subcategories for each (e.g., cost-effectiveness, cost-efficiency, perceived service quality, delivered service quality). The results of item #4 are provided in Chapter 6 of the final report. Recommendations are provided for transit agencies, state and regional transportation and funding agencies, stan- dards development, the NTD, and future steps. This working paper focuses on the remaining aspects of Task 10 not already documented in the final report. This working paper presents key examples from the Task 8 and 9 peer comparisons that serve to highlight lessons learned and methods for dealing with common challenges. It also provides recommendations on modifications to existing performance measures that would help support the peer-grouping and performance-measurement methodology. The paper is organized around the eight-step benchmark- ing methodology described in detail in the final report. Typ- ical questions and challenges associated with each step of the process are covered here, using specific examples from the Task 8 and 9 testing. Because of the projectâs time and resource limitations, the Task 8 and 9 testing covered only Steps 1â4 of the process. Guidance on conducting Steps 5â8 is provided in the final report. Table C1 provides a summary of the questions and topics addressed in this paper, organized by the methodological step to which each question applies. Summary of Case Study Results Step 1: Understand Context While relatively straightforward, understanding the con- text of the peer comparison is a key component of success. Specific lessons from the case studies included the need to A P P E N D I X C Task 10 Working Paper
carefully formulate the peer comparison question and the need to understand the analysis timeframe, particularly if non-NTD data are being considered. Determining the Topic Determining the topic for the peer comparison effort requires careful consideration if the peer comparison is to prove ben- eficial to decision-makers. Without a well-defined topic it will be difficult to select an appropriate set of performance measures, select relevant descriptive measures, and determine if secondary screening of peers is required. Similarly, topics that are too broad may yield results that do not directly address the intent of decision-makers. Many case study participants had a tendency to select broad or ill-defined performance comparison topics. For instance, several agencies simply stated a desire to understand their âefficiency.â The resulting peer comparisons were often focused on high-level efficiency measures (e.g., cost per ride) across a range of areas, but did not provide depth in any particular area. That would not necessarily be a problem except that once the high-level results were identified, no additional work was performed to dig into the reasons for the results. Thus, the underlying reasons why a particular agencyâs performance 98 Methodological Step Topic Example Agencies Carefully tailor the topic to avoid an ill- defined performance topic. Utah Transit Authority (UTA), Washington DOT Step 1: Understand Context Understand the analysis timeframe. Greater Bridgeport Transit Authority (GBTA) Carefully review relevant NTD forms to understand content. Santa Clara Valley Transit Authority (VTA), UTA Use descriptive measures to provide context to the analysis. VTA, Hillsborough Area Regional Transit (HART) Step 2a: Performance Measure Selection Use multiple measures to get a rounded perspective of a particular issue. Denver Regional Transit District (RTD) Step 2b: Identify Secondary Screening Measures What are appropriate variables for secondary screening? Setting thresholds to ensure that peers are relevant to performance question. Laredo Transit, Knoxville Area Transit Step 2c: Identify Thresholds Setting thresholds to lend additional credibility to peers. King County Metro Step 3a: Register for FTIS This step is self-explanatory. No additional detail is provided. Refine grouping methodology to remove specific variables. GBTA, King County Metro, Oahu Transit Step 3b: Form an Initial Peer Group Interpreting likeness scores and choosing a peer group. North County Transit District Step 3c: Perform Secondary Screening Applying the thresholds identified in Step 2c. King County Metro Exporting data to Excel for calculating non-standard performance measures. Step 4a: Gather Performance Data Gathering non-NTD data. GBTA, Rochester-Genesee RTA Using Excel to create graphs and charts to display result. Normalizing results to account for cost of RTD living and inflation. Step 4b: Analyze Performance Gather additional descriptive measures as needed. Table C1. Summary of working paper topics by methodological step.
was superior or inferior to its peers were not identified. To address this issue, the Altoona case study in the final report provides examples of digging deeper into the data to identify potential explanations for the high-level results. In contrast, several agencies provided detailed performance questions that allowed the peer comparison to be more focused from the start. For instance, Utah Transit Authority, like many other agencies, was interested in understanding operating efficiency. Rather than attempting to cover all aspects of efficiency within a single peer comparison, however, UTA chose a more focused question to consider: âHow efficient are my bus and rail operator schedules?â The methodology provided a well-rounded set of performance measures that addressed the topic to the extent possible through the NTD. (This aspect of the methodology is discussed further under Step 2a.) Other aspects of agency efficiency could be considered by follow-up peer comparisons in a similar manner. Understand the Analysis Timeframe At the outset of the peer comparison, it is important to understand the timeframe for which results will need to be available. This is particularly important if the target agency is considering the possibility of using non-NTD data for the peer comparison. The Florida Transit Information System soft- ware tool allows users to complete basic peer comparisons in a matter of hours when only NTD data and other standardized data provided by FTIS are required. However, the process for collecting non-NTD data is much more time-consuming. The Greater Bridgeport Transit Authority (GBTA) case study provides an illustrative example. GBTA wished to analyze employee absenteeism among peer agencies, a topic requiring non-NTD data. Eight peer agencies were contacted to obtain information about absenteeism rates by job category. After 2 months, replies were received from five of the eight peers, only two of which were able to provide specific absenteeism data/rates by worker category. As a result, the peer compari- son was not completed within the timeframe required by the research. With additional time and resources to dedicate to the effort, it is likely that sufficient data for a peer comparison could have been obtained by requesting data from more than the top eight peers. However, the overall effort would certainly take months, compared to days for a peer comparison using NTD data. Thus, understanding up-front whether such a timeframe is viable is necessary to avoid wasted effort. At the same time, once the initial effort has been made, it may be easier in the future to obtain the same data from the same set of peers as part of peer-comparison activities in future years. The state of the economy at the time of the research also influenced other agenciesâ ability to respond to data requests. Due to the financial crisis and its impacts on tax revenue, most transit agencies were facing funding shortfalls and agency staff did not have time to respond to outside data requests. Here again, having previously established relationships with peer agencies might make it easier to obtain data during such times. However, even an established benchmarking network such as TFLEx reported difficulty in getting its members to con- tribute data since the crisis started. Nevertheless, as pointed out in the final report, when funding is tight, it is more critical than ever for agencies to identify where they can improve their performance. Sharing data and practices with other agencies, while not necessarily providing an immediate benefit to an agency, helps establish relationships that can provide longer- term benefits. Step 2a: Performance Measure Selection The selection of an appropriate set of performance measures is one of the most important components of a successful peer comparison. There are hundreds of potential measures avail- able through the NTD, and choosing a reasonably-sized set of measures (ideally fewer than ten) that provides the desired detail can be difficult. While definitive âcookbook-styleâ guid- ance, such as that provided for selecting a peer group, may be ideal from the guidebook userâs perspective, the case study experience clearly shows that there is no single correct set of measures for a particular question. Rather, each agency must custom-select performance measures that address their specific performance question and operating environment. Case study participants that identified identical performance topics often ultimately chose considerably different perfor- mance measures. For instance, both the Greater Cleveland Regional Transit Authority (GCRTA) and the Lane Transit District (LTD) chose âWhat is a reasonable level of subsidy?â as their performance question. Table C2 shows the performance measures that each agency ended up selecting. Table C2 shows that only one performance measure (farebox recovery ratio) was chosen by both agencies. In general, LTD chose to focus on per-capita measures of revenue sources (i.e., a funding perspective), whereas GCRTA focused on the proportion of overall operating expense from various sources (i.e., an operating perspective). In addition, LTD chose to include several descriptive measures (i.e., measures selected purely to provide context), such as passenger trips per capita, to understand how various funding strategies may impact other performance measures of interest. While each agency chose to approach the question from a different perspective, both found the results of their respec- tive peer comparisons beneficial. This highlights the need to approach the selection of performance measures for every peer comparison as a unique exercise. The following section summarizes some key issues to consider while selecting per- formance measures. 99
Review Relevant NTD Forms As described in the final report, most peer comparisons will rely on NTD data due to the lack of viable alternatives. Consequently, fully understanding the contents of the NTD is critical to understanding performance measure options. FTIS reports hundreds of NTD measures across 18 different NTD formatting forms, making it difficult for any individual to be familiar with everything that the NTD reports. VTA and UTA peer comparisons (modified versions of which are included as case studies in the final report) provide good examples of the potential to mine NTD forms to develop nonstandard performance measures to answer specific peer- comparison questions. Research team staff worked closely with agency staff to review relevant NTD forms and develop a tai- lored set of performance measures. In both cases, reviewing the NTD forms allowed the groups to select measures that they were initially not aware of. Most of the selected performance measures are ratios, which provide greater comparability of results across agencies. Table C3 summarizes the peer comparison questions and selected outcome measures for each agency. The variables listed in Table C3 show clearly how a variety of focused performance measures can be derived from NTD data by considering more than the most commonly used measures. Use Descriptive Measures to Provide Context In addition to the outcome measures that form the core of a peer comparison, it is often beneficial to collect data for descriptive measures as well. Descriptive measures are mea- sures that do not directly address the performance question at hand, but provide context as to why a particular result occurs. Many useful descriptive measures are included among the methodologyâs peer-grouping variables (e.g., total operating budget, congestion per capita), but other descriptive measures may also be valuable depending on the application. Chapter 4 of the final report provides lists of descriptive measures, arranged by topic. For instance, as part of the VTA peer comparison for light- rail maintenance shown in Table C-3, VTA also used miles of track, number of elevators, and number of escalators as descrip- tive measures to provide information on factors that may drive non-vehicle maintenance costs. In general, identifying useful descriptive measures during Step 2 of the benchmarking process will reduce the need to collect supplemental data later in the process. Similarly, Hillsborough Area Regional Transit used such descriptive variables as the total number of vehicles operated in peak service to supplement their peer comparison on system-wide efficiency. Use Multiple Overlapping Measures to Provide Perspective Some agencies may rely primarily on a single performance measure to address a specific question (e.g., cost per trip as the measure of cost-effectiveness). While such an approach may make sense for a benchmarking exercise across a wide range of topics, the ease with which FTIS allows users to summarize NTD data means that data for multiple measures can be gathered with little additional effort. During the Task 8 case studies, the Denver Regional Transit District (RTD) examined cost-effectiveness as its performance 100 LTD Measures GCRTA Measures Overlap? Farebox recovery ratio Farebox recovery ratio Local operating funds per capita Local revenue as percent of operating expense State operating funds per capita State revenue as percent of operating expense Federal operating funds per capita Federal revenue as a percent of operating expense Operating cost per revenue hour Other directly generated revenue as percent of operating expense Operating subsidy per revenue hour Operating subsidy per capita Passenger trips per capita Revenue hours per capita Operating costs per capita Table C2. Performance measures selected by GCRTA and LTD relating to subsidy level.
topic. Rather than using a single measure of cost-effectiveness, several measures were used, including cost per trip, cost per revenue hour, revenue per trip, trips per revenue mile, and trips per revenue hour. By examining the same issue from multiple angles, the peer comparison was able to provide more insight into the differences between RTD and its peers. For instance, RTD rated at the peer group average for trips per revenue hour, but lower than average for trips per revenue mile, as a result of RTD providing more long-distance, higher-speed service than the peer group as a whole. Step 2b: Identify Secondary Screening Measures As described in the final report, the selection of a peer group is a vital part of the benchmarking process to produce relevant results and establish credibility with stakeholders. While the peer-grouping methodology developed by this research and incorporated into FTIS is designed to produce a reasonable peer group for most situations, secondary screening may be needed in some circumstances, either to answer a perfor- mance question that requires a specific type of agency or to eliminate agencies deemed âtoo differentâ from the target agency. Table C4 lists each of the peer-grouping variables and provides general guidance on their appropriateness as secondary screening measures. Note that secondary screening measures should be deter- mined prior to identifying a preliminary peer group through FTIS in order to avoid the appearance of subjectivity (i.e., choosing a secondary screening measure to exclude a specific agency). Other potential secondary screening measures are also described in the final report. Step 2c: Identify Thresholds As described in Step 2b, agencies may identify secondary screening measures and thresholds to ensure the relevancy of the peer group to the question at hand and to ensure peer group credibility. Setting Thresholds to Ensure Relevancy The most common reason to conduct secondary screening is to ensure that peer agencies are relevant to the question at hand. For instance, the Texas DOT conducted a peer comparison for Laredo Transit to better understand the transit agencyâs funding options after Laredoâs population reaches 200,000, at which time it will no longer qualify for state funding. The peer comparison focused on the mix of funding sources for peer agencies that operated in urban areas larger than 200,000, and thus it required a minimum population of 200,000 for any peer agency. Similarly, Knoxville Area Transit was specifically 101 VTA UTA How cost-effective are VTAâs vehicle and non- vehicle light-rail maintenance program? How efficient are UTAâs bus and rail operator schedules? Maintenance expenditures as percent of operating expense Operating cost per passenger mile Actual car miles per malfunction Operating cost per passenger hour Maintenance labor as percent of total maintenance cost Revenue hours as percent of vehicle hours Maintenance costs per actual car mile Salaries/wages/benefits as percent of operating expenses Vehicle materials and supplies cost per actual car mile Operating wages as percent of operating expenses Vehicle maintenance labor cost per actual car mile Vehicle revenue hours per operating FTE Non-vehicle maintenance costs per station Passenger trips per operating FTE Non-vehicle materials and supplies cost per station Non-operating time as percent of total operatingtime Non-vehicle maintenance labor per station Breaks and allowances as percent of total operatingtime Premium hours as percent of operating hours Table C3. Outcome measures used in the VTA and UTA case studies.
102 Peer-Grouping Variables Potential Secondary Screening Applications Urban area population Commonly used for secondary screening, either because peers must fall into a specific population category to be relevant (e.g., same FTA funding eligibility) or because vastly different urban area populations may hinder the credibility of a given peer. Larger population tolerances are acceptable for larger urban areas because they will naturally have fewer peers. Typical population tolerances may range from 25% to 50%. Note that in most cases, the methodology will naturally select peers within this range. Total annual vehicle miles operated May be appropriate for secondary screening for operations- and finance- related applications, where the scale of a peer agencyâs operations is particularly important. Annual operating budget May be appropriate for secondary screening for operations- and finance- related applications, where the scale of a peer agencyâs operations is particularly important. Population density Used when typical regional land-use patterns are important to the comparison. Service area type May be appropriate for secondary screening where all peer agencies must operate comparable service. For instance, an agency that runs all service in a region may wish to only compare itself to agencies that do the same. In most cases, the methodology will select peers with identical service types, but will also include some agencies with similar service types (e.g., service that extends outside the urbanized area). State capital (yes/no) Not typically used for secondary screening but can be used when evaluating a marginal candidate peer (i.e., one with a total likeness score >0.74). Percent college students Not typically used for secondary screening, although an agency operating in an area with a high student population may set a minimum percentage for peer agencies. This may be particularly applicable for funding-related questions since systems serving large universities may receive funding from the university and/or be more likely to have free or reduced fares on at least some routes. For smaller college towns dominated by the presence of a university, the methodology will tend to select other college towns. Population growth rate Used when regional growth (or shrinkage) and an agencyâs response to the growth is important to the comparison. Percent low-income population Not typically used for secondary screening but can be used when evaluating a marginal candidate peer (i.e., one with a total likeness score >0.74). Annual roadway delay (hours) per traveler Not typically used for secondary screening but can be used when evaluating a marginal candidate peer (i.e., one with a total likeness score >0.74). Freeway lane miles (thousands) per capita Not typically used for secondary screening but can be used when evaluating a marginal candidate peer (i.e., one with a total likeness score >0.74). Percent service demand- responsive May be used for secondary screening, particularly if an agency dedicates an unusually large portion of its budget to demand-responsive service and wishes to have a peer group that does the same. Percent service purchased May be used for secondary screening, particularly for finance and operations-related comparisons. Not only may the amount of purchased service have a significant impact on operations, but some NTD measures (e.g., operating employee FTEs) are not reported for purchased service, limiting the usefulness of agencies with purchased service for certain comparisons. Distance May be used for secondary screening when having relatively nearby peers will aid in stakeholder acceptance of the process due to being familiar with the peers. Table C4. Potential applications of peer-grouping variables for secondary screening.
interested in various types of dedicated local funding sources and therefore only selected agencies with dedicated local fund- ing sources as peers. Setting Thresholds to Ensure Peer Group Credibility The peer grouping methodology developed by the research team is specifically designed to produce a reasonable peer group with no secondary screening, and the results of the case studies indicate that this is typically the case. However, there may be some instances when an agency feels that a threshold should be set for a particular variable to ensure the credibility of the resulting peer group. This is most likely to be necessary when an agencyâs uniqueness limits the number of close-fitting peers. For instance, King County Metro is the largest bus-only operator in the country, limiting the number of good-fitting peers available. Because of this, the TCRP Project G-11 method- ology returned several much smaller transit agencies within the same urban area that King County Metro would not consider as peers. Although the final methodology was ad- justed to address this situation, agencies that are among the largest in their class may still find it appropriate to set a min- imum threshold for peers based on vehicle miles operated. Step 3a: Register for FTIS This step is self-explanatory, and no additional detail is provided. Step 3b: Form an Initial Peer Group In most cases, forming a peer group is simple, requiring only a straightforward application of the peer-grouping tool provided by FTIS. However, in some instances, agencies will wish to refine the peer grouping methodology to better fit their needs and/or may need to select a peer group from agencies whose likeness scores do not indicate close fits. Refine Grouping Methodology By exporting the FTIS peer grouping results table into an Excel spreadsheet, agencies are able to refine the methodology to provide a peer group that meets their individual needs. Most commonly, this will involve removing a specific peer grouping variable for one of three reasons: 1. The agency does not feel that the variable is relevant for establishing its peer group. For instance, the peer group- ing methodology assigns a high weight to agencies that operate rail service in order to avoid selecting a bus-only agency as a peer for a rail-operating agency and vice versa. Two large bus-only agencies, King County Metro and Orange County Transit Authority, however, expressed no problem with including rail agencies in their peer group, while a third (PACE in suburban Chicago) preferred a bus- only group. Distance is another peer-grouping factor that may make sense for agencies in isolated locations such as Hawaii or Alaska to eliminate. 2. The agency does not wish to exclude agencies with missing data for a particular measure from being in its peer group because the factor is not essential for the performance question being asked. For instance, GBTA indicated in the case studies that they would prefer not to exclude agencies with missing congestion data from the peer grouping. 3. A potential peer is operated by multiple NTD reporters (e.g., the Trinity Railway Express commuter rail line, which is jointly operated by the transit agencies in Dallas and Ft. Worth) and the operatorsâ data need to be combined. Table C5 shows an example peer grouping refinement for Oahu Transit Service to eliminate the distance factor. Through the screening, several potential peers are available for Oahu with total likeness scores less than 1.0, whereas none were available previously. Overall, the screening had the effect of replacing several west coast agencies with agencies located elsewhere in the country. Note, however, that five of the top six peers identified through the screened method are still located in warmer areas of the country (California, Texas, Nevada, and Florida) due to sharing other demographic characteristics with Honolulu. Interpreting Likeness Scores Agencies with unique characteristics will often have few potential peer agencies with total likeness scores that meet the ideal thresholds described in the final report. Many agencies may have difficulty identifying a full peer group with likeness scores less than 0.75, and some may even have difficulty find- ing peers with likeness scores less than 1.0. The final report provides several potential reasons why agencies may be unable to find a large peer group, several of which were encountered in the case studies. For instance, North County Transit District (NCTD) in Oceanside, California, is unique among suburban bus operators in that it also operates commuter rail service (the âCoasterâ train) and a diesel light-rail line. As a result, only three agencies received likeness scores less than 0.75, as shown in Table C6. Moreover, the top-ranked agency (Caltrain) runs primarily commuter rail service with only minimal bus service, making it a poor choice for an agency-wide comparison. Despite this, NCTD was still able to form a peer group with which they felt comfortable and which provided useful results. However, moving ahead with a peer group with higher like- ness scores requires that analysts pay greater attention to the performance results to understand if performance differences are likely caused primarily by fundamental differences between 103
agencies. An alternative approach that could have been taken for the NCTD case study would have been to compare each of its modes separately, instead of doing an agency-wide comparison. Step 3c: Perform Secondary Screening As described above, secondary screening may beneficial under several circumstances. Typically, secondary screening is relatively straightforward using the FTIS software once the thresholds have been identified in Step 2c. Continuing with the King County Metro example, Table C7 shows an example of how a secondary screening could have been conducted for the top ten King County Metro peers. This screening eliminates transit agencies with operating budgets of less than one-third of King County Metroâs budget, as well as agencies with total likeness scores exceeding 1.0. Although hav- ing a budget only one-third of King County Metroâs might seem like a low threshold to meet, there are very few bus-only opera- tors in the country that meet that criterion, and AC Transit was an agency that King County thought was an appropriate peer. Table C7 shows that two agencies (Pierce County and Snohomish County) would be eliminated based on oper- ating budget, leaving four potential bus-only peers with likeness scores less than 1.0. While four peers is at the low end of the preferred size for a peer group, the secondary screening served to make the peer group more credible and relevant to King County. (Note that under the final version 104 Rank Agency Likeness Score Used as Peer? 1 Caltrain (San Francisco, CA) 0.58 No 2 Sound Transit (Seattle, WA) 0.65 Yes 3 Fort Worth Transportation Authority 0.72 Yes 4 Sacramento RTD 0.86 Yes 5 Santa Clara VTA (San Jose, CA) 0.99 Yes 6 Utah Transit Authority (Salt Lake City, UT) 1.04 Yes 7 Bi-State Development Agency (St. Louis, MO) 1.05 No 8 Metro Transit (Minneapolis, MN) 1.06 No 9 Memphis Area Transit Authority 1.07 No 10 TriMet (Portland, OR) 1.15 No Table C6. North County Transit District peer group. Peer Group Using Distance Factor Peer Group Without Distance Factor Rank Agency Likeness Score Rank Agency Likeness Score 1 Las Vegas RTC 1.24 1 Capital Metro (Austin, TX) 0.75 2 Capital Metro (Austin, TX) 1.31 2 Central Florida RTA (Orlando, FL) 0.75 3 City of Phoenix Transit 1.33 3 Las Vegas RTC 0.85 4 Kansas City Area Transportation Authority 1.44 4 Kansas City Area Transportation Authority 0.86 5 Omnitrans (San Bernandino, CA) 1. 45 5 City of Phoenix Transit 0.92 6 Central Florida RTA (Orlando, FL) 1.48 6 VIA (San Antonio, TX) 0.96 7 VIA (San Antonio, TX) 1.50 7 Capital District Transit (Albany, NY) 0.97 8 Riverside Transit (Riverside, CA) 1.52 8 Rhode Island Transit Authority 1.01 9 King County Metro (Seattle, WA) 1.62 9 Milwaukee County Transit 1.03 10 San Mateo County Transit 1.62 10 Omnitrans (San Bernandino, CA) 1.11 Table C5. Example peer-grouping refinement for Oahu Transit Service.
of the methodology, Snohomish County and Pierce County would receive scores of 1.26 and 1.60, respectively, eliminat- ing them from consideration prior to the secondary screen- ing process.) Step 4a: Gather Performance Data FTIS makes the collection of performance data a straight- forward task for peer comparisons that rely on NTD data. However, gathering data may be the most challenging portion of a peer comparison where non-NTD data are used. Export Data to Excel to Calculate Non-Standard Variables As described earlier, most outcome measures take the form of ratios (e.g., cost per boarding). While FTIS provides many of these variables directly through its Florida Standard Variables list, users will often need to calculate some of these measures manually. Because FTIS allows users to export performance data directly into an Excel spreadsheet, these calculations are fairly straightforward. This method was used to calculate performance measures associated with numerous case studies and to develop the updated performance measure results used in the Chapter 5 case studies. Collecting Non-NTD Data Most peer comparisons will rely on NTD data because of the difficulty with collecting non-NTD data. However, agencies frequently encounter situations where NTD data do not address a particular performance question. Through the case studies, two agencies chose to perform a peer comparison using non- NTD data: GBTA and RGRTA. For the GBTA case study, eight peer agency comparisons were conducted to obtain information about absenteeism rates by job category. However, after 2 months only two had provided usable data, and the effort was aborted. For the RGRTA case study, seven peer comparisons were conducted to obtain information on customer service and satisfaction using eight non-NTD performance measures. Of the seven peer agencies, four were able to provide data, and the peer comparison was able to proceed. In general, these experiences suggest several recommenda- tions for using non-NTD data: ⢠Agencies wishing to use non-NTD data for peer compar- isons will need to invest significant time and resources to be successful, ⢠Agencies may wish to select unusually large peer groups, knowing that a large percentage of peers will be unable to provide information, and ⢠Forming benchmarking networks may be the most effec- tive means to gather non-NTD data over the long term. Step 4b: Analyze Performance Use Charts and Graphs to Visually Enhance Results The case studies showed repeatedly the value of using charts and graphs (primarily generated in Excel) to display the peer comparison results. Such graphics serve to enhance 105 Agency Total Likeness Score Annual Vehicle Miles Operated (000,000s) % King County Operating Budget Retained as Peer? King County Metro 0.00 53.4 100.0% N/A VIA Metropolitan Transit 0.76 26.8 50.2% Yes Alameda-Contra Costa Transit 0.76 21.2 39.7% Yes Snohomish County Transit 0.83 12.0 22.5% Pierce County Transit 0.89 14.2 26.6% Orange County Transit 0.92 32.2 60.3% Yes Pace 0.96 46.6 87.3% Yes City of Detroit 1.05 16.5 30.9% Capital Metro (Austin) 1.05 17.6 33.0% Milwaukee County Transit 1.06 22.0 41.2% San Mateo County 1.06 10.0 18.7% Table C7. Example secondary screening process.
understandingof theresultsandincrease the likely impact that a peercomparisonmayhaveon stakeholders or decision-makers. There is no one âbestâ way to display the peer comparison results. The following graphics display several examples from the case studies that participants found useful (see Figure C1). Normalize Results to Account for Inflation and Cost of Living Normalizing performance results to account for inflation and/or cost of living can be an important component of increasing the usability of results. The process for doing so is described in detail in the final report under the case study for the Denver RTD. Gather Additional Descriptive Measures as Needed The performance analysis will undoubtedly raise ques- tions of why a particular agency does better or worse on a given measure. In many cases, direct agency follow-up is required to understand the precise reasons for any result. However, additional descriptive measures may also help the analyst better understand the factors underlying the analysis results. For instance, an analysis that identifies a disparity in the number of vehicle malfunctions or vehicle maintenance costs between agencies may wish to also col- lect data on fleet age (if it has not already been collected) to assess whether this may contribute to the observed differ- ence. The Altoona case study in the final report illustrates this process. Performance Measure Recommendations The case studies found that the NTD can be used to derive a variety of performance measures that are useful for peer- comparison applications. However, the case studies also found that many in the industry still have doubts about the accuracy of NTD data. Although data quality was generally not an issue 106 Revenue Hours Per Capita* Motorbus (MB) 0.00 0.20 0.40 0.60 0.80 1.00 1.20 1.40 20 06 P ee r A ve ra ge 2002 1.11 1.04 0.75 0.79 0.75 0.77 0.34 0.64 0.32 2003 1.02 1.04 0.80 0.86 0.82 0.76 0.31 0.71 0.31 2004 1.19 1.05 0.90 0.86 0.78 0.29 0.75 0.31 2005 1.15 1.09 0.85 0.98 0.80 0.79 0.64 0.69 0.31 2006 1.14 1.18 0.85 1.09 0.75 0.77 0.44 0.67 0.32 Rank 2006 2 1 4 3 6 5 8 7 9 Lane Transit District Spokane Transit Authority Ben Franklin Transit (Kennewick) Intercity Transit (Olympia) Kitsap Transit (Bremerton) Salem Area Mass Transit Mountain Metro Transit (Colorado Springs) San Joaquin RTD (Stockton) Sonoma County (Santa Rosa) *Higher values are desired. 2006 peer average does not include LTD. (a) Lane Transit District (Eugene, Oregon) Figure C1. Examples of performance comparison graphics.
107 DIRECTLY OPERATED FAREBOX RECOVERY: UTA VS. PEER GROUP 0 5 10 15 20 25 30 35 2002 2003 2004 2005 2006 2007 YEAR FA R EB O X RE CO VE RY % Central Puget Sound Regional Transit Authority Charlotte Area Transit System Denver Regional Transportation District Jacksonville Transportation Authority Metro Transit Niagara Frontier Transportation Authority North County Transit District North San Diego County Transit District San Francisco Municipal Railway Santa Clara Valley Transportation Authority Tri-County Metropolitan Transportation District of Oregon Utah Transit Authority (c) Utah Transit Authority (Salt Lake City, Utah) (b) Hillsborough Area Regional Transit (Tampa, Florida) Figure C1. (Continued).
108 (e) Star Metro (Tallahassee, FL) Peer Group Analysis 0 1 2 3 4 5 6 7 8 Providence, RI Fairfax, VA Dover, DE Norfolk, VA Clearwater, FL Pompano Beach, FL West Palm Beach, FL Concord, CA A ve ra ge T rip L en gt h (m ile s) 0 5 10 15 20 25 30 35 40 Pa ss en ge r T rip s (m illi on s) (d) Pinellas Suncoast Transit Authority (St. Petersburg, Florida) Figure C1. (Continued). that came up during the case studies, there were occasions when isolated bad data were spotted, and there were particular meas- ures observed where it appears that NTD reporters are not yet following the FTAâs guidance on how to calculate certain measures. Because of the volume of data already reported to the NTD and because there is not yet widespread acceptance of the quality of NTD data or the value of NTD reporting, no new NTD measures are recommended. Rather, the NTD-related recommendations focus on better reporting of certain existing NTD measures that would be valuable for peer comparisons. As NTD data quality improves, and as benchmarking becomes more of a standard transit industry practice, the time may come in the future when additional standard measures can be added. These are addressed in the second half of the recommendations.
109 NTD Recommendations Service Area Size and Population There are two significant issues with these NTD measures. First, the NTD defines service area as the area within three- quarters of a mile of bus routes and rail stations. However, many agencies report the entire urban area or transit district population and area. Second, only a single value is reported per agency, while the service areas of different modes oper- ated by an agency may vary considerably. Outcome measures based on per-capita ratios can be valu- able comparison tools for comparing relative transit invest- ments and productivity but require good service-population data to be able to make the comparison. This projectâs bench- marking methodology substitutes the combination of urban area population and agency service type as proxies for the number of people served. While these factors are useful in combination as a first-cut tool for identifying potential is- sues, it is readily acknowledged that they are not ideal and that service-area specific data would be preferable if the data could be relied upon. Tracking regional population is not a normal transit agency function. MPOs, on the other hand, have the data and tools to readily perform these calculations. MPOs might be used in the future as the source of reliable service area and popu- lation data, rather than relying on transit agencies to supply these data. Vehicle System Failures There are few NTD-derivable measures that directly address the reliability of the service experienced by passengers. Miles between vehicle system failures is one such measure. However, the case studies found considerable inconsistency in how the NTDâs number of vehicle system failures variable was reported, resulting in a lack of confidence in any measure based on that variable. Track Miles versus Directional Route Miles For commuter rail systems, the amount of single tracking impacts the amount of service that can be provided and, potentially, the reliability of that service. Comparing track miles and directional route miles should provide this infor- mation (1 mile of single track = 1 track mile and 2 directional route miles, while 1 mile of double track = 2 track miles and 2 directional route miles). However, all peer agencies in the Tri-Rail case study reported the same number of track miles as directional route miles, after rounding, indicating that track miles are not being reported correctly. (Some of the agencies were known to be mostly single-track operations and track miles should also include sidings and non-revenue track, such as tracks in rail yards.) Transit Industry Recommendations The benchmarking literature indicates that a customer focus is important; however, very few NTD measures address service quality or customer satisfaction outcomes. At the same time, many agencies collect some form of service-quality data, for example by tracking complaints, conducting customer- satisfaction surveys, and measuring reliability and passenger loads. These data could be of great use in conducting peer comparisons related to service quality. The difficulty is that agency definitions for these measures are inconsistent. Rather than trying to force agencies to change their existing measures, it may be easier to encourage the development and storage of lowest-common-denominator measures that can be used to calculate related measures given a particular definition. For example: ⢠Measuring and storing minutes early/late at time points (something that is possible with automatic vehicle systems technology) can be used to calculate such reliability-related measures as on-time performance (using any desired defini- tion of âon-timeâ), excess wait time (extra minutes passengers had to wait past the scheduled time, also accounting for early departures if desired), and headway adherence. These measures could be calculated for any desired time point along a line (e.g., start, middle, close to the end, end) and could be aggregated to a system average for a chosen location (e.g., the end of a line). ⢠For measures relating to passenger load, data on number of passengers at the maximum load point that was linked to spe- cific fleet data (number of seats, already collected for the NTD, and standing area, not currently collected) would allow calculation of load factor and area per standing passenger measures. Data on passenger loads by route segment is al- ready collected as part of the process for estimating passen- ger miles. ⢠Number of complaints is easy to track but for comparison purposes needs to be normalized by both the number of comments received and total passenger boardings since some agencies make it easier than others to submit comments and the number of people using transit service will obviously vary from agency to agency. ⢠TCRP Report 47 (C-1) provides recommended standard- ized questions for customer-satisfaction surveys. While every survey needs to be customized to the needs of the agency performing it, having industry agreement on at least a few core questions that could be asked using the same language and reporting scale would facilitate comparing customer satisfaction results between agencies.
Other case studies found desires for employee absenteeism data and more-detailed maintenance data, and there are any number of performance questions that could be conceived that would require specialized data. Rather than trying to encourage more NTD reporting, it may be more fruitful to encourage more widespread industry adoption of practices and definitions that will lead to greater collection and avail- ability of standardized transit data. The industry-related recommendations in the final report encourage (a) developing standard definitions for key non- NTD performance measures, (b) establishing performance measurement and benchmarking as standard practices through their standards program, and (c) establishing a confidential clearinghouse for non-NTD data. The first recommendation addresses data-definition needs, the second would lead to more widespread internal agency data collection for a variety of needs, and the third would allow agencies to voluntarily and confiden- tially share their standardized non-NTD data with other agen- cies, for the betterment of the public transportation industry. Reference C-1. MORPACE International, Inc., and Cambridge Systematics, Inc., TCRP Report 47: A Handbook for Measuring Customer Satisfaction and Service Quality. TRB, National Research Council, Washington, D.C., 1999. 110
