Click for next page ( 12


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 11
Volume II: Guide for Performance Measure Identification and Target Setting 3.0 Framework for Performance Measure Selection and Use 3.1 Overview This framework for selecting and applying performance measures recognizes that each agency will have a different set of circumstances and needs. Some agencies may already have a solid set of per- formance measures in place and are looking to refine measures in a few specific areas. Other agen- cies may have a large set of measures in place across different units that are not well coordinated or integrated and are looking to rationalize or streamline the measures in use. Still others may want to start from scratch and identify a complete new set of performance measures for use in their agency. The framework also acknowledges that an agency's performance measurement efforts will likely evolve over time in response to lessons learned. In some cases, targets will be adjusted to account for new circumstances; in others, the measures themselves may be modified or completely replaced as new data, system capabilities, and business processes evolve. The framework is organized into three sections, as shown in Figure 3, which can be used indepen- dently. Section 3.2 provides guidance for identifying performance measures that are in place, identi- fying gaps, and considering new measures. Section 3.3 provides guidance for integrating performance measures into the organization, including how to design related measures that can be used at differ- ent levels and for different purposes. Section 3.4 provides guidance for establishing performance targets within a resource allocation process. 3.2 Identifying Performance Measures Hundreds of possible performance measures can be useful for asset management, and no single set of measures will work well in every situation. The challenge is to identify a manageable set that can be effectively implemented and used within a given organization. Step 1: Inventory Existing Measures Most agencies have a set of existing performance measures that provide a base from which to begin. A recommended first step is to take an inventory of these measures and create a table, such as the one shown in Table 1. This approach can identify existing measures that are currently not used or that have issues that impede full implementation. Eliminating these measures can free up resources that can be used for implementation of more useful measures. 11

OCR for page 11
Figure 3. Guidance for Performance Measures and Targets Identify Performance Identify Performance Measures Measures Inventory Define Identify Candidate Assess and Select Measures Assess Existing Selection Measures/Adjustments for Further Design and Gaps Measures Criteria to Existing Measures Implementation Integrate Performance Integrate Performance Measures Measures into into the the Organization Organization Tailor Design consistent Identify Design Document Engage Measures to Measures Across Improvements to Communication Definitions and Stakeholders Decisions Program Areas Data and Tools Devices Procedures Establish Performace Targets Analyze Define Establish Select Scope Develop Consider Resource Consider Contexts Targets and of Measures Long-Term Funding Allocation Policy and and Time Track for Targets Goals Availability Scenarios Public Input Horizons Progress and Tradeoffs

OCR for page 11
Volume II: Guide for Performance Measure Identification and Target Setting Table 1. Performance Measure Inventory Example Format Measure Owner How Used Comments/Issues Percent good Materials section External reporting Stable and well-accepted pavement District performance targets Crash rate Safety office External reporting Only includes serious crashes Project prioritization Travel rate index Traffic management Internal reporting Urban freeways only center Congested VMT Traffic management Internal reporting Urban freeways only center Asset value Finance office GASB 34 reporting Need to improve basis for calculations External performance reporting Step 2: Assess Gaps The next step is to decide on the scope of your efforts by conducting a gap analysis of your agency's existing measures. Gaps in performance measurement to be addressed include (1) gaps in coverage of key policy goals or result areas, (2) gaps in use of performance measures to guide resource allo- cation, and (3) gaps in alignment of performance measures both vertically and horizontally within the agency. Even though measures in some areas may be defined and used to some extent for report- ing, it is important to see where there may be disconnects between (1) information gathered and reported for a particular policy objective and (2) the resource allocation decisions that are made that impact achievement of that objective. For example, in Table 1, one might ask if congested vehicle- miles traveled (VMT) is used to help determine priorities for corridor improvements or to support decisions on allocation of funds across different operational program areas. It is also important to assess the need for greater horizontal and vertical alignment across measures. Improved horizontal alignment may be needed when existing measures do not adequately support tradeoffs across asset classes, geographic areas, and/or investment types. Improved vertical alignment may be needed when there are inconsistencies in measures used to make decisions related to a given policy objec- tive. For example, if an agency uses pavement smoothness targets as the basis for setting the pave- ment preservation budget level, but then uses structural condition as the basis for identification and prioritization of projects, this reflects a lack of vertical alignment. A logical way to assess gaps in coverage of performance measures is to review your performance measure inventory against your agency's stated goals and objectives to see where there may be gaps or areas where existing measures need to be improved. Figure 2 may also be helpful in determining areas where additional measures are needed. The following list of questions can be used as a guide in assessing gaps in performance measures: Given your agency's goals and objectives, what are the most important outcomes to be achieved? Do your existing performance measures adequately cover these outcome areas? 13

OCR for page 11
Volume II: Guide for Performance Measure Identification and Target Setting Do you currently track the condition of all of your major assets? Do you have measures related to the level of service or function provided by these assets? Do you have measures that reflect customer perspectives? Do your performance measures align with national guidance, statewide priorities, governor's initiatives, and so forth? Are your performance measures aligned internally? Are you tracking program delivery in order to monitor accomplishments and identify where agency and user costs can be reduced? Do your performance measures adhere to federal highway legislation governing long-range planning and capital programming (Title 23 U.S.C. Sections 134 and 135)?: Do your performance measures capture the impacts of preservation, management, and oper- ations activities carried out by your agency? Do you have performance measures that are applicable to both long-range planning and transportation improvement program/state transportation improvement program (TIP/STIP) development? Do you have mode-neutral measures that can be used to evaluate modal alternatives to address transportation needs? Do you have measures that capture transportation's broad range of impacts on society, including safety and security, economic development, productivity and competitiveness, accessibility and mobility options for passengers and freight, fuel efficiency, environmental protection, and improved quality of life? There may be too many gaps to address at once. Some gaps may be fundamentally difficult to fill, given the lack of reasonable measures for some types of policy objectives, programs, and functions or constraints on data collection resources. In deciding where to focus, you will want to consider where additional performance measures will have the greatest impact on your agency's ability to make better resource allocation decisions, gain public confidence, and secure needed resources. Step 3: Define Selection Criteria There are many factors to consider in selecting performance measures for asset management. It is therefore helpful to explicitly define selection criteria that are important to your agency and follow a systematic evaluation process in which candidate performance measures are assessed based on these criteria. Most agencies find that they need to periodically change performance measures in order to respond to changes in leadership and policy. Having documentation that shows both the alternative measures that were considered and the basis for choosing current measures can provide a valuable resource for addressing future questions about why the measures were selected and for investigating future changes to measures. Not all selection criteria are applicable to all measures. Selection criteria should reflect the intended purpose, use, and audience for the performance measure. A performance measure to be used for 14

OCR for page 11
Volume II: Guide for Performance Measure Identification and Target Setting tracking agencywide progress toward targets needs to meet different criteria than one used to help prioritize locations for pavement work. A measure to be used to evaluate the progress of a highway division's ITS program in alleviating congestion does not need to meet criteria for mode neutrality, whereas a measure used to evaluate agencywide efforts to improve mobility likely would. The following criteria can be used to assess whether a given performance measure is a good candi- date for implementation. You may wish to add other criteria based on your agency's particular needs and history. You may also want to ignore criteria that are not relevant given the intended pur- pose of the performance measures. Criterion 1: Feasible Data Requirements. Does the measure have realistic and feasible data requirements? Does the agency currently have the data required for a particular measure? Data Quality and Accuracy. Can sufficient data quality, reliability, and accuracy be ensured to provide credible and usable information? Cost Versus Value Added. Would the benefits of having this measure available for decision making outweigh the cost of collecting the data required for it? Criterion 2: Policy Sensitive Reflects Policies. Can the measure be easily related to the agency's stated policy objectives? Does it provide a good measure of whether the outcomes intended by the policies are occurring? Meaningful. Does the measure convey meaningful information to decision-makers about the transportation system? Criterion 3: Supports Long-Term, Strategic View Trend Information. Are a baseline value and quality trend information available for this mea- sure? If not, is long-term tracking of trends feasible? Forecasts. Are reasonable and defendable methods available to forecast the future value of this measure (i.e., how the value of the measure would change in the absence of actions taken by the agency)? Life-Cycle Analysis. Is this measure suitable for incorporation into an analysis comparing long- term investment alternatives based on life-cycle costs and benefits? Criterion 4: Useful for Decision Support Feedback. Does the measure provide information that enables managers to understand prob- lems and suggest solutions? Responsiveness. Can the measure be tracked and reported in a timely enough fashion to sup- port the needs of decision-makers? (For measures to be used by operational managers, real-time monitoring may be required.) 15

OCR for page 11
Volume II: Guide for Performance Measure Identification and Target Setting Sensitivity. Is the measure sufficiently sensitive to reflect impacts of agency actions given the levels at which it will be applied (e.g., network, corridor, and project level and for short-, medium-, and long-range decision making)? Linkage to Actions. Are there methods available to predict how the value of this measure would change as a result of: A specific project or budget line item (e.g., an intersection improvement project or purchase of 10 new maintenance trucks)? Implementation of a program of defined projects (e.g., capital projects in the STIP)? Allocation of a given budget level to a specific program area (e.g., $10 million per year over the next 5 years to pavement preservation)? Scenario Testing. Is this measure amenable to "what-if" analysis, in which the performance implications of multiple budget scenarios are estimated? Can predictions be automated so that what-if testing is not overly time consuming? Neutrality. Is the measure sufficiently neutral to allow for evaluation of the impacts of a wide range of possible agency investments or actions (e.g., highway versus transit improvements to improve mobility)? (This question will not be relevant for measures that are used to track per- formance for a particular mode or asset class.) Agency Influence. Can this measure be used to help distinguish changes in transportation sys- tem performance that have occurred due to your agency's actions from changes that have occurred due to factors not under your agency's control? Criterion 5: Useful Across the Organization and Beyond Ease of Understanding. Can this measure be easily understood and interpreted by its intended audiences at technical, management, and executive levels of the organization and by outside stakeholders? Vertical Alignment. Can this measure function as part of a family of measures that can be used to describe performance at different levels of aggregation (e.g., corridor, district, and system), for different time horizons, and for different audiences? Is it consistent with other measures in use? Horizontal Alignment. Can this measure be used by multiple horizontal units of the organiza- tion (e.g., across units that manage different classes of assets and/or modes) in order to promote consistency in performance measurement and to facilitate investment tradeoffs across areas? Cross-Jurisdictional. Can this measure be used at a broad level outside of the organization across other jurisdictions in order to enable a consistent regional or statewide view of performance? Step 4: Identify Candidate Measures/Adjustments to Existing Measures Appendix A provides a list of performance measures that can be of value for asset management. Performance measures for consideration are organized by goal area: preservation, mobility and 16

OCR for page 11
Volume II: Guide for Performance Measure Identification and Target Setting accessibility, operations and maintenance, safety, economic development, environmental impacts, social impacts, and security. A list of delivery-oriented measures is also included at the end. Even though there are many performance measures listed, this is not intended to be a comprehensive set. There are a number of other references to consult for additional ideas--see the bibliography at the end of Volume I. Table 2 provides a sample format for organizing the measures that you select according to their intended uses and audiences. Table 2. Performance Measures: Uses and Audiences Example Format Daily Operations/Management Performance-Based Budgeting Long-Range Planning Project Prioritization Corridor Planning Strategic Planning Annual reporting Delivery Policy Objective/Performance Measure SYSTEM PRESERVATION a Percent Poor Pavements b Average Bridge Health Index c Remaining Value d Backlog e Customer Ratings MOBILITY/ACCESSIBILITY a Travel Time Index (Urban Freeways) b Annual Change in Travel Time Between Major Cities c Percent Population Within 10 Minutes of a State Highway d Time from End of Snow Event to Bare Pavement e Backlog--System Completion SAFETY a Serious Crashes/Million VMT b Number of Fatalities c Backlog--Economically Justified Safety Improvements 17

OCR for page 11
Volume II: Guide for Performance Measure Identification and Target Setting This classification will be of value for evaluating the measures in Step 5. Categories in the chart are as follows: Strategic Planning--Describing organizational objectives and establishing annual performance targets for the agency's strategic business plan. Long-Range Planning--Characterizing current systemwide or subnetwork performance, defining long-term system condition and service objectives, analyzing the impacts of differ- ent investment levels or strategies, and estimating investment needs associated with alter- native performance levels. Corridor Planning--Defining condition and service objectives for specific corridors and evalu- ating alternatives for corridor improvements. Performance-Based Budgeting--Allocating dollars to program areas in a performance-based budgeting process. Project Prioritization--Prioritizing and selecting projects for programming. Daily Operations/Management--Short-term and real-time decisions by operations and main- tenance program managers with respect to work prioritization and resource allocation across competing needs to maximize effectiveness. Delivery--Assessment of program and project delivery progress and effectiveness and evalua- tion of alternative delivery methods. Annual Reporting--Annual system performance reporting for establishing accountability and demonstrating good stewardship to external stakeholders and to the public at large and/or for informing transportation policy decisions considered by the agency management team and elected officials. Note that some measures will fit in several of these categories. For example, the remaining value of assets can be useful for high-level, strategic, and long-range planning decisions to make the case for additional investment. It can also be used at a corridor level as part of a study of future corridor devel- opment and management options in order to characterize future preservation needs. In a program development context, budget amounts for different categories of assets or for different geographic areas might be influenced by looking at the relative remaining value. Similarly, prioritization of reha- bilitation projects within a given program area might also be based on looking at the remaining value for candidate projects. At this point in the process, do not be too concerned about the specific functional form of the per- formance measure--this can be refined later using the process described in Section 3.3. Step 5: Assess and Select Measures for Further Design and Implementation Once your criteria are established, you can develop a format such as the one shown in Table 3 to assess each candidate measure. The table rates a variety of measures using the criteria described in Step 3. (Note that the table is meant as an illustrative example and that the ratings may vary based 18

OCR for page 11
Volume II: Guide for Performance Measure Identification and Target Setting Table 3. Performance Measure Assessment Example Format Can Link to Agency Actions Horizontal Alignment Meaningful for Policy FeasibleData/Tools Cross-Jurisdictional Trend Info Feasible Vertical alignment Useful Feedback Scenario Testing Can Forecast Sensitivity Neutrality Policy Objective/Performance Measure 1 SYSTEM PRESERVATION a Percent Poor Pavements -- -- b Average Bridge Health Index -- -- -- c Remaining Value d Backlog -- -- e Customer Ratings -- -- -- 2 MOBILITY/ACCESSIBILITY a Travel Time Index (Urban Freeways) b Annual Change in Travel Time Between Major Cities -- c Percent Population Within 10 Minutes of a State Highway -- d Time from End of Snow Event to Bare -- -- -- Pavement e Backlog--System Completion -- 3 SAFETY a Serious Crashes/Million VMT -- b Number of Fatalities c Backlog--Economically Justified Safety -- Improvements KEY: Meets criterion. May meet criterion (depends on measure formulation or application context). -- Does not meet criterion, or criterion is not applicable. on your agency's organizational, institutional, and technical characteristics.) Completing this eval- uation will provide a structured way to consider each of the criteria. There are likely to be questions raised along the way and action items for further investigation. For example, you may want to research how other agencies have approached prediction or impact assessment using some of the candidate measures and assess whether similar methods would work in your agency. 19