National Academies Press: OpenBook

Performance Measures for Freight Transportation (2011)

Chapter: Chapter 3 - Performance Measurement Experience in the Public Sector

« Previous: Chapter 2 - Performance Measurement Lessons from the Private Sector
Page 32
Suggested Citation:"Chapter 3 - Performance Measurement Experience in the Public Sector." National Academies of Sciences, Engineering, and Medicine. 2011. Performance Measures for Freight Transportation. Washington, DC: The National Academies Press. doi: 10.17226/14520.
×
Page 32
Page 33
Suggested Citation:"Chapter 3 - Performance Measurement Experience in the Public Sector." National Academies of Sciences, Engineering, and Medicine. 2011. Performance Measures for Freight Transportation. Washington, DC: The National Academies Press. doi: 10.17226/14520.
×
Page 33
Page 34
Suggested Citation:"Chapter 3 - Performance Measurement Experience in the Public Sector." National Academies of Sciences, Engineering, and Medicine. 2011. Performance Measures for Freight Transportation. Washington, DC: The National Academies Press. doi: 10.17226/14520.
×
Page 34
Page 35
Suggested Citation:"Chapter 3 - Performance Measurement Experience in the Public Sector." National Academies of Sciences, Engineering, and Medicine. 2011. Performance Measures for Freight Transportation. Washington, DC: The National Academies Press. doi: 10.17226/14520.
×
Page 35
Page 36
Suggested Citation:"Chapter 3 - Performance Measurement Experience in the Public Sector." National Academies of Sciences, Engineering, and Medicine. 2011. Performance Measures for Freight Transportation. Washington, DC: The National Academies Press. doi: 10.17226/14520.
×
Page 36

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

30 c h a p t e r 3 Performance Measurement Experience in the Public sector The research for this project included an extensive review of public-sector use of performance measures. Appen- dices A–D describe: (1) freight-specific performance mea- sures in use by state transportation departments, (2) the measures published by federal agencies, (3) the availability of performance metrics by individual modes, and (4) a sum- mary of the performance measures relating to environmental and safety issues. Several summary points can be made based upon the review described in the appendices. First, although public- sector performance measurement has matured and expanded significantly, the number of freight-specific performance measures remains limited. The few states that included freight performance measures in their performance report- ing suites typically had fewer than four freight measures. The measures tended to be captured from existing data sources. Second, no consensus as to which freight measures were most important to states was evident. No two states had selected the same measures. It was not possible from an examination of the state freight measures to identify a common cohort of measures that were generally agreed on. Third, consid- erable ambivalence exists among states about performance measurement. Although many embrace it, some expressed concern that it will be difficult to capture accurate, consis- tent, and meaningful measures across such a diverse set of states, modes, and issues. Several state officials expressed concern that, if the measures were not accurate, consistent, and meaningful, the measures would not lead to improved decision making. Fourth, performance measures related to freight system condition are more available than measures of freight system performance. Fifth, performance data related to externalities, such as emissions and crashes, are among the most complete performance data available. Migration of Performance Measures from the Private to Public Sectors A major turning point in the migration of performance measures from the private to the public sectors occurred with the 1992 publication of Osborne and Gaebler’s Reinventing Government: How the Entrepreneurial Spirit Is Transforming the Public Sector.1 Osborne and Gaebler posited several pri- mary points that have become widely accepted now in the public sector: • What gets measured gets done. • If you can’t measure results, you can’t tell success from failure. • If you can’t see success, you can’t reward it. • If you can’t reward success, you’re probably rewarding failure. • If you can’t see success, you can’t learn from it. • If you can’t recognize failure, you can’t correct it. • If you can demonstrate results, you can win public support. Osborne and Gaebler identified the parameters of public- sector performance measures. They accordingly offered the following recommendations: • Use both quantitative as well as qualitative measures. Some important results are impossible to quantify. • Watch out for creaming, or the tendency to select the easily accomplished while avoiding the difficult. • Anticipate powerful resistance to accountability. • Involve stakeholders in developing measures. • Subject measures to periodic review and evaluation. • Don’t use too many or too few measures.

31 • Watch out for perverse incentives. • Keep measurement functions independent. • Focus on maximizing the use of performance data. The Government Performance Results Act of 1993 codi- fied many of the private-sector lessons into a framework for federal performance management. Inherent in the Act are the key findings of earlier performance measurement research: • A strategic plan and strategic mission statement are the foundation for performance measurement. • General goals and objectives are to be established and are to be described in terms of outcomes, not inputs or outputs. • Strategies for achieving the objectives are to be identified. • Performance measures should be defined that measure the effectiveness of the strategies in achieving the outcomes. • Key factors beyond the agency’s control that can influence the achievement of the goals need to be identified. • Processes will be established to evaluate the effectiveness of the measures and to update them as needed. The Maturation of Public-Sector Performance Measurement By the early 2000s, the use of performance measures in public-sector transportation agencies was widespread and exhibited evolution similar to that reported earlier in the pri- vate sector. A 2004 report, Performance Measurement in Trans- portation: State of the Practice,2 notes that use of performance measures among transportation agencies has expanded sig- nificantly and that agencies have become increasing informed and insightful in using them. In reviewing more than a decade of performance measurement by departments of transpor- tation (DOTs) and local transportation agencies, the report notes the following trends: • Although more states are using measures, the leading states are involved in second- and third-generation measures, which are increasingly sophisticated. • States are relying more on measures that emphasize strate- gic outcomes and customer-focused measures. • States that are early in the use of measures tend to prolifer- ate them, whereas more mature states tend to focus upon a “vital few.” • The use of measures to support broader planning, man- agement, and decision-making processes is becoming common. • There is increased reporting directly to the public and policy makers. • Elements of the Balanced Scorecard approach are evident in several states that rely on customer satisfaction as a balancing measure to augment engineering and financial performance data. Likewise, measures of environmental quality are increasingly prominent. States are increasingly careful about how they specify per- formance measures, because they realize that they can be critically important in driving decisions and actions. Mea- suring an aspect of performance encourages the agency to focus upon that aspect, sometimes to the neglect of other important functions. Skewing organizational behavior can be an unintended consequence of performance measurement, particularly if measures are narrow or are not tempered by qualitative considerations. State of the Practice, cited above, notes significant variation among the states’ use of measures. A few agencies, though, have mature systems, all of which could be characterized as possessing: • a range of sophisticated measurement systems in place; • the alignment of measures with performance-oriented goals, objectives, standards, and targets; • useful performance-reporting processes tailored for vari- ous audiences and management needs; and • systematic procedures for reviewing performance data and using the information to strengthen performance and decision making. The report also cited a set of continuing challenges that include: • Agreeing on common terminology; • Developing measures for cross-modal comparisons of performance; • Developing freight measures; • Getting broader public and constituency feedback and bal- ancing that with engineering and planning criteria; • Implementing useful benchmarking criteria for compara- tive analysis; and • Institutionalizing performance measurement and strate- gic planning to prevent their being derailed by changes in administration. By the end of the decade, the use of performance measures was common in the majority of states. AASHTO’s Standing Committee on Performance Management supports a web- site linking performance reports or annual reports to 41 state transportation agencies. Although not definitive, the links indicated that more than 50 percent of the states produced some kind of performance reports.

32 Another trend is that the states with the most mature and expansive performance reporting produced interpretive reports to accompany the published metrics. The Washing- ton State DOT’s Grey Book includes more than 100 pages of metrics quarterly. In addition, the DOT accountability website includes links to other department reports of their performance. The Missouri DOT’s Tracker includes more than 100 mea- sures, and its monthly report includes more than 220 pages of data and interpretation. The Minnesota DOT’s Annual Transportation Performance Report tracks 16 basic areas of performance and includes 38 pages of explanatory material. Commonly used metrics address infrastructure condition, highway safety, project delivery, budgeting, personnel goals, and progress toward specific programs. Less common are mobility measures and freight measures. NCHRP Synthesis 311 reviewed the use of performance measures for the moni- toring and operational management of highway segments and systems.3 An assessment of the relative strengths and weaknesses of over 70 performance measures in use then was performed. The survey of DOTs and metropolitan planning organizations (MPOs) yielded the list of performance mea- sures shown on Table 3.1. The Responses column shows how many responding agencies use the measure. Note that freight performance measure use is low, as is the use of measures that capture real-time highway operations. Measures of the number of persons or vehicles served were most commonly reported as the most important mea- sures, including volume, vehicle-miles traveled (VMT), persons served expressed in person-miles traveled, and freight- volume served expressed in truck-miles traveled. With respect to freight, the synthesis suggests that measures could be developed to reflect the freight system and could use data such as vehicle length, height, and weight, the number of axles, safety inspection statistics, truck crashes, commer- cial vehicle enforcement/inspection times and costs, railroad crossing protection, and weigh-in-motion information. Public-Sector Logic for Selecting Measures NCHRP Report 5514 recommends a step-by-step process specifically for developing asset management measures, but the process can be applied to most categories of measures: Identify Performance Measures • Inventory existing performance measures and identify how they are being used, • Identify gaps to be addressed based on coverage of critical outcome areas for agency goals and objectives and support for the asset-management best practices, • Define criteria for selecting new measures, • Identify additional candidate measures, and • Select a set of measures from the list of candidates for fur- ther design and implementation. Integrate Performance Measures into the Organization • Engage internal and external stakeholders to achieve buy-in, • Identify the different decision contexts in which perfor- mance measures are to be used (project, corridor, and net- work levels and for short- or long-range decisions) and refine measures so that they are at the appropriate level of sensitivity, • Identify opportunities for using measures that are consis- tent across different organizational units responsible for various asset classes, modes, or work types, • Identify needs for additional data collection, data manage- ment, and analytic tools to support the selected measures, • Design communication devices with formats appropriate to the target audiences, and • Document measure definitions and procedures. Establish Performance Targets • Define the context for target setting and establish time horizon(s), • Determine which measures should have targets, • Develop long-term goals based on consideration of techni- cal and economic factors, • Consider current and future funding availability, • Analyze resource allocation scenarios and tradeoffs, • Consider policy and public-input implications for target setting, and • Establish targets and track progress. States Use Only a Handful of Freight Measures Although the research literature identified hundreds of potential freight performance measures, in practice the minority of states that have freight performance measures use only a handful. Mature performance measurement states such as Washington, Missouri, and Minnesota use between 5 and 10 measures. It was noticeable that no two states had the same measures, and in most cases there were wide differ- ences in the metrics. Although states reported freight perfor- mance metrics, most of the metrics were not used to calibrate performance of specific state programs. Exceptions were for Missouri’s customer satisfaction with its motor carrier office. Generic measures such as travel time in freight-significant

33 1 Table 3.1. Examined performance measures. Performance Measure Typical Definition Responses Level of Service (LOS) Qualitative assessment of highway point, segment, or system using A (best) to F (worst) based on measures of effectiveness 11 Traffic Volume Annual average daily traffic, peak-hour traffic, or peak-period traffic 11 Vehicle-Miles Traveled Volume times length 10 Travel Time Distance divided by speed 8 Speed Distance divided by travel time 7 Incidents Traffic interruption caused by crash or other unscheduled event 6 Duration of Congestion Period of congestion 5 Percent of System Congested Percent of miles congested (usually defined based on LOS E or F) 5 Vehicle Occupancy Persons per vehicle 5 Percent of Travel Congested Percent of vehicle-miles or person-miles traveled 4 Delay Caused by Incidents Increase in travel time caused by an incident 3 Density Vehicles per lane, per period 3 Rail Crossing Incidents Traffic crashes that occur at highway-rail grade crossings 3 Recurring Delay Travel time increases from congestion; this measure does not consider incidents 3 Travel Costs Value of driver’s time during a trip and any expenses incurred during the trip (vehicle ownership and operating expenses or tolls or traffic) 3 Weather-related Incidents Traffic interruption caused by inclement weather 3 Response Times to Incidents Period required for an incident to be identified, to be verified, and for an appropriate action to alleviate the interruption to traffic to arrive at the scene 2 Commercial Vehicle Safety Violations Number of violations issued by law enforcement based on vehicle weight, size, or safety 1 Evacuation Clearance Time Reaction and travel time for evacuees to leave an area at risk 1 Response Time to Weather-related Incidents Period required for an incident to be identified, to be verified, and for an appropriate action to alleviate the interruption to traffic to arrive at the scene 1 Security for Highway and Transit Number of violations issued by law enforcement for acts of violence against travelers 1 Toll Revenue Dollars generated from tolls 1 Travel Time Reliability Several definitions are used 1 table 3.1. examined performance measures. Source: PBS&J. NCHRP Synthesis 311: Operational Effectiveness for Highway Segments and Systems. Transportation Research Board of the National Academies, Washington, D.C., 2003, Table 6, p. 22.

34 corridors were likely a contributing factor to state efforts to improve overall travel times. However, it appeared to be rare that state DOT freight performance measurement was used to make frequent decisions. Most of the measures appear to be indicators of broad trends of overall transportation system performance. The Iowa DOT freight-related performance measures include highway crash rates per million vehicle miles for large trucks, total crashes at rail–highway crossings, and railroad derailments per million ton-miles.5 Freight efficiency mea- sures include the percentage of Iowa rail carriers earning a reasonable return on investment and average rail revenue per ton-mile. The quality-of-life performance measures iden- tify approximate travel times to major external markets in the Midwest Region, percentage of railroad track-miles able to handle 286,000-pound cars, percentage of railroad track- miles able to operate at 30 miles per hour or more, and rail fuel use per ton-mile. Washington State DOT’s Grey Book includes a handful of freight performance measures. It reports upon truck vol- umes on state highways, the number of truck border cross- ings, rail freight tonnage, and container shipments through state ports. The Missouri DOT Tracker includes 111 measures, of which five relate to freight: freight tonnage by mode; percent- age of trucks using advanced technology at weigh stations; interstate motor carrier mileage; percentage of satisfied motor carriers; and customer satisfaction with timeliness of Motor Carrier Services response. The customer satisfaction ratings focus on users’ satisfaction with service received by the Motor Carrier Services office. Relevance of the Public- Sector Experience A significantly more detailed discussion of the public- sector use of freight performance measures is included in Appendices C and D. The summary of that review indicates that widespread use of freight performance measures is not common. A consensus as to which freight measures should be adopted was not apparent from a review of the measures states were using. An AASHTO task force on examining freight performance measures recommended three mea- sures: travel speeds on the freight-significant routes; reliabil- ity on freight-significant routes; and border crossing delay. Although those three are important, they fall far short of the expansive objective of this research, that is, to have a com- prehensive set of measures that examines multiple aspects of freight performance for all modes. Chapter 4 summarizes the types of freight performance data available to support a performance measurement framework. Endnotes 1 Osborne, David, and Ted Gaebler. Reinventing Government: How the Entre- preneurial Spirit Is Transforming the Public Sector, Addison-Wesley, Reading, Mass., 1992, pp. 147–154. 2 Poister, Theodore H. “Performance Measurement in Transportation: State of the Practice.” Conference Proceedings 36: Performance Measures to Improve Transportation Systems—Summary of the Second National Conference, 2004, pp. 7–9, 85–86. 3 PBS&J. NCHRP Synthesis 311: Performance Measures of Operational Effec- tiveness for Highway Segments and Systems. Transportation Research Board of the National Academies, Washington, D.C., 2003. 4 Cambridge Systematics, Inc., PB Consult, and Texas Transportation Insti- tute. NCHRP Report 551: Performance Measures and Targets for Transpor- tation Asset Management. Transportation Research Board of the National Academies, Washington, D.C., 2006. 5 Iowa DOT, Performance Measures for Iowa Transportation Systems, 2006. Prepared by Iowa State University, Center for Transportation Research and Education.

Next: Chapter 4 - Freight Performance Measures »
Performance Measures for Freight Transportation Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

TRB’s National Cooperative Freight Research Program (NCFRP) Report 10: Performance Measures for Freight Transportation explores a set of measures to gauge the performance of the freight transportation system.

The measures are presented in the form of a freight system report card, which reports information in three formats, each increasingly detailed, to serve the needs of a wide variety of users from decision makers at all levels to anyone interested in assessing the performance of the nation’s freight transportation system.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!