Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 11
Volume II: Guide for Performance Measure Identification and Target Setting
3.0 Framework for Performance
Measure Selection and Use
3.1 Overview
This framework for selecting and applying performance measures recognizes that each agency will
have a different set of circumstances and needs. Some agencies may already have a solid set of per-
formance measures in place and are looking to refine measures in a few specific areas. Other agen-
cies may have a large set of measures in place across different units that are not well coordinated or
integrated and are looking to rationalize or streamline the measures in use. Still others may want to
start from scratch and identify a complete new set of performance measures for use in their agency.
The framework also acknowledges that an agency's performance measurement efforts will likely
evolve over time in response to lessons learned. In some cases, targets will be adjusted to account
for new circumstances; in others, the measures themselves may be modified or completely replaced
as new data, system capabilities, and business processes evolve.
The framework is organized into three sections, as shown in Figure 3, which can be used indepen-
dently. Section 3.2 provides guidance for identifying performance measures that are in place, identi-
fying gaps, and considering new measures. Section 3.3 provides guidance for integrating performance
measures into the organization, including how to design related measures that can be used at differ-
ent levels and for different purposes. Section 3.4 provides guidance for establishing performance
targets within a resource allocation process.
3.2 Identifying Performance Measures
Hundreds of possible performance measures can be useful for asset management, and no single set
of measures will work well in every situation. The challenge is to identify a manageable set that can
be effectively implemented and used within a given organization.
Step 1: Inventory Existing Measures
Most agencies have a set of existing performance measures that provide a base from which to begin.
A recommended first step is to take an inventory of these measures and create a table, such as the
one shown in Table 1. This approach can identify existing measures that are currently not used or
that have issues that impede full implementation. Eliminating these measures can free up resources
that can be used for implementation of more useful measures.
11
OCR for page 12
Figure 3. Guidance for Performance Measures and Targets
Identify Performance
Identify Performance Measures
Measures
Inventory Define Identify Candidate Assess and Select Measures
Assess
Existing Selection Measures/Adjustments for Further Design and
Gaps
Measures Criteria to Existing Measures Implementation
Integrate Performance
Integrate Performance Measures
Measures into
into the
the Organization
Organization
Tailor Design consistent Identify Design Document
Engage
Measures to Measures Across Improvements to Communication Definitions and
Stakeholders
Decisions Program Areas Data and Tools Devices Procedures
Establish Performace Targets
Analyze
Define Establish
Select Scope Develop Consider Resource Consider
Contexts Targets and
of Measures Long-Term Funding Allocation Policy and
and Time Track
for Targets Goals Availability Scenarios Public Input
Horizons Progress
and Tradeoffs
OCR for page 13
Volume II: Guide for Performance Measure Identification and Target Setting
Table 1. Performance Measure Inventory
Example Format
Measure Owner How Used Comments/Issues
Percent good Materials section External reporting Stable and well-accepted
pavement
District performance targets
Crash rate Safety office External reporting Only includes serious crashes
Project prioritization
Travel rate index Traffic management Internal reporting Urban freeways only
center
Congested VMT Traffic management Internal reporting Urban freeways only
center
Asset value Finance office GASB 34 reporting Need to improve basis for
calculations
External performance
reporting
Step 2: Assess Gaps
The next step is to decide on the scope of your efforts by conducting a gap analysis of your agency's
existing measures. Gaps in performance measurement to be addressed include (1) gaps in coverage
of key policy goals or result areas, (2) gaps in use of performance measures to guide resource allo-
cation, and (3) gaps in alignment of performance measures both vertically and horizontally within
the agency. Even though measures in some areas may be defined and used to some extent for report-
ing, it is important to see where there may be disconnects between (1) information gathered and
reported for a particular policy objective and (2) the resource allocation decisions that are made that
impact achievement of that objective. For example, in Table 1, one might ask if congested vehicle-
miles traveled (VMT) is used to help determine priorities for corridor improvements or to support
decisions on allocation of funds across different operational program areas. It is also important to
assess the need for greater horizontal and vertical alignment across measures. Improved horizontal
alignment may be needed when existing measures do not adequately support tradeoffs across asset
classes, geographic areas, and/or investment types. Improved vertical alignment may be needed
when there are inconsistencies in measures used to make decisions related to a given policy objec-
tive. For example, if an agency uses pavement smoothness targets as the basis for setting the pave-
ment preservation budget level, but then uses structural condition as the basis for identification and
prioritization of projects, this reflects a lack of vertical alignment.
A logical way to assess gaps in coverage of performance measures is to review your performance
measure inventory against your agency's stated goals and objectives to see where there may be gaps
or areas where existing measures need to be improved. Figure 2 may also be helpful in determining
areas where additional measures are needed. The following list of questions can be used as a guide
in assessing gaps in performance measures:
· Given your agency's goals and objectives, what are the most important outcomes to be achieved?
Do your existing performance measures adequately cover these outcome areas?
13
OCR for page 14
Volume II: Guide for Performance Measure Identification and Target Setting
· Do you currently track the condition of all of your major assets?
· Do you have measures related to the level of service or function provided by these assets?
· Do you have measures that reflect customer perspectives?
· Do your performance measures align with national guidance, statewide priorities, governor's
initiatives, and so forth?
· Are your performance measures aligned internally?
· Are you tracking program delivery in order to monitor accomplishments and identify where
agency and user costs can be reduced?
· Do your performance measures adhere to federal highway legislation governing long-range
planning and capital programming (Title 23 U.S.C. Sections 134 and 135)?:
Do your performance measures capture the impacts of preservation, management, and oper-
ations activities carried out by your agency?
Do you have performance measures that are applicable to both long-range planning
and transportation improvement program/state transportation improvement program
(TIP/STIP) development?
Do you have mode-neutral measures that can be used to evaluate modal alternatives to
address transportation needs?
Do you have measures that capture transportation's broad range of impacts on society,
including safety and security, economic development, productivity and competitiveness,
accessibility and mobility options for passengers and freight, fuel efficiency, environmental
protection, and improved quality of life?
There may be too many gaps to address at once. Some gaps may be fundamentally difficult to fill,
given the lack of reasonable measures for some types of policy objectives, programs, and functions
or constraints on data collection resources. In deciding where to focus, you will want to consider
where additional performance measures will have the greatest impact on your agency's ability to
make better resource allocation decisions, gain public confidence, and secure needed resources.
Step 3: Define Selection Criteria
There are many factors to consider in selecting performance measures for asset management. It is
therefore helpful to explicitly define selection criteria that are important to your agency and follow
a systematic evaluation process in which candidate performance measures are assessed based on
these criteria. Most agencies find that they need to periodically change performance measures in
order to respond to changes in leadership and policy. Having documentation that shows both the
alternative measures that were considered and the basis for choosing current measures can provide
a valuable resource for addressing future questions about why the measures were selected and for
investigating future changes to measures.
Not all selection criteria are applicable to all measures. Selection criteria should reflect the intended
purpose, use, and audience for the performance measure. A performance measure to be used for
14
OCR for page 15
Volume II: Guide for Performance Measure Identification and Target Setting
tracking agencywide progress toward targets needs to meet different criteria than one used to help
prioritize locations for pavement work. A measure to be used to evaluate the progress of a highway
division's ITS program in alleviating congestion does not need to meet criteria for mode neutrality,
whereas a measure used to evaluate agencywide efforts to improve mobility likely would.
The following criteria can be used to assess whether a given performance measure is a good candi-
date for implementation. You may wish to add other criteria based on your agency's particular
needs and history. You may also want to ignore criteria that are not relevant given the intended pur-
pose of the performance measures.
Criterion 1: Feasible
· Data Requirements. Does the measure have realistic and feasible data requirements? Does the
agency currently have the data required for a particular measure?
· Data Quality and Accuracy. Can sufficient data quality, reliability, and accuracy be ensured to
provide credible and usable information?
· Cost Versus Value Added. Would the benefits of having this measure available for decision
making outweigh the cost of collecting the data required for it?
Criterion 2: Policy Sensitive
· Reflects Policies. Can the measure be easily related to the agency's stated policy objectives? Does
it provide a good measure of whether the outcomes intended by the policies are occurring?
· Meaningful. Does the measure convey meaningful information to decision-makers about the
transportation system?
Criterion 3: Supports Long-Term, Strategic View
· Trend Information. Are a baseline value and quality trend information available for this mea-
sure? If not, is long-term tracking of trends feasible?
· Forecasts. Are reasonable and defendable methods available to forecast the future value of this
measure (i.e., how the value of the measure would change in the absence of actions taken by the
agency)?
· Life-Cycle Analysis. Is this measure suitable for incorporation into an analysis comparing long-
term investment alternatives based on life-cycle costs and benefits?
Criterion 4: Useful for Decision Support
· Feedback. Does the measure provide information that enables managers to understand prob-
lems and suggest solutions?
· Responsiveness. Can the measure be tracked and reported in a timely enough fashion to sup-
port the needs of decision-makers? (For measures to be used by operational managers, real-time
monitoring may be required.)
15
OCR for page 16
Volume II: Guide for Performance Measure Identification and Target Setting
· Sensitivity. Is the measure sufficiently sensitive to reflect impacts of agency actions given the
levels at which it will be applied (e.g., network, corridor, and project level and for short-,
medium-, and long-range decision making)?
· Linkage to Actions. Are there methods available to predict how the value of this measure would
change as a result of:
A specific project or budget line item (e.g., an intersection improvement project or purchase
of 10 new maintenance trucks)?
Implementation of a program of defined projects (e.g., capital projects in the STIP)?
Allocation of a given budget level to a specific program area (e.g., $10 million per year over
the next 5 years to pavement preservation)?
· Scenario Testing. Is this measure amenable to "what-if" analysis, in which the performance
implications of multiple budget scenarios are estimated? Can predictions be automated so that
what-if testing is not overly time consuming?
· Neutrality. Is the measure sufficiently neutral to allow for evaluation of the impacts of a wide
range of possible agency investments or actions (e.g., highway versus transit improvements to
improve mobility)? (This question will not be relevant for measures that are used to track per-
formance for a particular mode or asset class.)
· Agency Influence. Can this measure be used to help distinguish changes in transportation sys-
tem performance that have occurred due to your agency's actions from changes that have
occurred due to factors not under your agency's control?
Criterion 5: Useful Across the Organization and Beyond
· Ease of Understanding. Can this measure be easily understood and interpreted by its intended
audiences at technical, management, and executive levels of the organization and by outside
stakeholders?
· Vertical Alignment. Can this measure function as part of a family of measures that can be used
to describe performance at different levels of aggregation (e.g., corridor, district, and system), for
different time horizons, and for different audiences? Is it consistent with other measures in use?
· Horizontal Alignment. Can this measure be used by multiple horizontal units of the organiza-
tion (e.g., across units that manage different classes of assets and/or modes) in order to promote
consistency in performance measurement and to facilitate investment tradeoffs across areas?
· Cross-Jurisdictional. Can this measure be used at a broad level outside of the organization across
other jurisdictions in order to enable a consistent regional or statewide view of performance?
Step 4: Identify Candidate Measures/Adjustments to Existing Measures
Appendix A provides a list of performance measures that can be of value for asset management.
Performance measures for consideration are organized by goal area: preservation, mobility and
16
OCR for page 17
Volume II: Guide for Performance Measure Identification and Target Setting
accessibility, operations and maintenance, safety, economic development, environmental impacts,
social impacts, and security. A list of delivery-oriented measures is also included at the end. Even
though there are many performance measures listed, this is not intended to be a comprehensive set.
There are a number of other references to consult for additional ideas--see the bibliography at the
end of Volume I.
Table 2 provides a sample format for organizing the measures that you select according to their
intended uses and audiences.
Table 2. Performance Measures: Uses and Audiences
Example Format
Daily Operations/Management
Performance-Based Budgeting
Long-Range Planning
Project Prioritization
Corridor Planning
Strategic Planning
Annual reporting
Delivery
Policy Objective/Performance Measure
SYSTEM PRESERVATION
a Percent Poor Pavements
b Average Bridge Health Index
c Remaining Value
d Backlog
e Customer Ratings
MOBILITY/ACCESSIBILITY
a Travel Time Index (Urban Freeways)
b Annual Change in Travel Time Between Major Cities
c Percent Population Within 10 Minutes of a State
Highway
d Time from End of Snow Event to Bare Pavement
e Backlog--System Completion
SAFETY
a Serious Crashes/Million VMT
b Number of Fatalities
c Backlog--Economically Justified Safety Improvements
17
OCR for page 18
Volume II: Guide for Performance Measure Identification and Target Setting
This classification will be of value for evaluating the measures in Step 5. Categories in the chart are
as follows:
· Strategic Planning--Describing organizational objectives and establishing annual performance
targets for the agency's strategic business plan.
· Long-Range Planning--Characterizing current systemwide or subnetwork performance,
defining long-term system condition and service objectives, analyzing the impacts of differ-
ent investment levels or strategies, and estimating investment needs associated with alter-
native performance levels.
· Corridor Planning--Defining condition and service objectives for specific corridors and evalu-
ating alternatives for corridor improvements.
· Performance-Based Budgeting--Allocating dollars to program areas in a performance-based
budgeting process.
· Project Prioritization--Prioritizing and selecting projects for programming.
· Daily Operations/Management--Short-term and real-time decisions by operations and main-
tenance program managers with respect to work prioritization and resource allocation across
competing needs to maximize effectiveness.
· Delivery--Assessment of program and project delivery progress and effectiveness and evalua-
tion of alternative delivery methods.
· Annual Reporting--Annual system performance reporting for establishing accountability and
demonstrating good stewardship to external stakeholders and to the public at large and/or for
informing transportation policy decisions considered by the agency management team and
elected officials.
Note that some measures will fit in several of these categories. For example, the remaining value of
assets can be useful for high-level, strategic, and long-range planning decisions to make the case for
additional investment. It can also be used at a corridor level as part of a study of future corridor devel-
opment and management options in order to characterize future preservation needs. In a program
development context, budget amounts for different categories of assets or for different geographic
areas might be influenced by looking at the relative remaining value. Similarly, prioritization of reha-
bilitation projects within a given program area might also be based on looking at the remaining value
for candidate projects.
At this point in the process, do not be too concerned about the specific functional form of the per-
formance measure--this can be refined later using the process described in Section 3.3.
Step 5: Assess and Select Measures for Further Design and Implementation
Once your criteria are established, you can develop a format such as the one shown in Table 3 to
assess each candidate measure. The table rates a variety of measures using the criteria described in
Step 3. (Note that the table is meant as an illustrative example and that the ratings may vary based
18
OCR for page 19
Volume II: Guide for Performance Measure Identification and Target Setting
Table 3. Performance Measure Assessment
Example Format
Can Link to Agency Actions
Horizontal Alignment
Meaningful for Policy
FeasibleData/Tools
Cross-Jurisdictional
Trend Info Feasible
Vertical alignment
Useful Feedback
Scenario Testing
Can Forecast
Sensitivity
Neutrality
Policy Objective/Performance Measure
1 SYSTEM PRESERVATION
a Percent Poor Pavements -- --
b Average Bridge Health Index -- -- --
c Remaining Value
d Backlog -- --
e Customer Ratings -- -- --
2 MOBILITY/ACCESSIBILITY
a Travel Time Index (Urban Freeways)
b Annual Change in Travel Time Between
Major Cities --
c Percent Population Within 10 Minutes of
a State Highway --
d Time from End of Snow Event to Bare
-- -- --
Pavement
e Backlog--System Completion --
3 SAFETY
a Serious Crashes/Million VMT --
b Number of Fatalities
c Backlog--Economically Justified Safety
--
Improvements
KEY: Meets criterion.
May meet criterion (depends on measure formulation or application context).
-- Does not meet criterion, or criterion is not applicable.
on your agency's organizational, institutional, and technical characteristics.) Completing this eval-
uation will provide a structured way to consider each of the criteria. There are likely to be questions
raised along the way and action items for further investigation. For example, you may want to
research how other agencies have approached prediction or impact assessment using some of the
candidate measures and assess whether similar methods would work in your agency.
19
