Cover Image

Not for Sale

View/Hide Left Panel
Click for next page ( 80

The National Academies of Sciences, Engineering, and Medicine
500 Fifth St. N.W. | Washington, D.C. 20001

Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 79
Guide for Target-Setting II-9 Step 2--Evaluate the Factors Influencing Target-Setting Asking the Right Questions In order to evaluate the factors affecting target-setting for a particular measure in a public agency or private organization, a practitioner can begin by asking a few simple questions. The recommended approach revolves around a process commonly known as the Five Ws and an H (Who, What, When, Where, Why, and How). WHY Is the Target Needed? "Why" requires thinking about the following factors: Internal support; Political/Legislative influence; Customer service focus; Stakeholder expectations; Commitment to regular communication and reporting; and Types of resources to be allocated. This involves developing an explicit understanding about why a target is being developed and the particular performance measure with which it is associated. It provides the link between the target and the larger PBRA process (i.e., the specific measure to which it relates and, in turn, the transportation goal/objective that it supports). This requires specifying the measures suitable for a target. Some performance measures may not lend themselves to quan- tification, while others may support long-term transportation goals but be outside the authority of the agency to influence and therefore inappropriate for defining a prescriptive target. Further, the need for a target related to a specific measure may stem from a need within the agency or from real or perceived needs from elected officials, the public, or other stakeholders. WHO Will Be Using the Targets? "Who" requires thinking about the following factors: Internal support/culture; Political/Legislative influence; Customer service focus; Span of control/agency jurisdiction; and Types of resources to be allocated. This involves defining who the end-user is for the target; namely, if that user is internal or external to the transportation agency. An internal user, for example, could be a technical staff person using the target to aid in project evaluation, while an external user might be an elected official, Board member, or member of the public who will be using the target to review invest- ment performance. If the end user includes external parties then this factor involves paying close attention to understanding the policy and public implications of setting particular targets (i.e., what is desired versus what is possible given certain resources).

OCR for page 79
II-10 Guide for Target-Setting and Data Management WHERE in the Process Will Targets Be Used? "Where" requires thinking about the following factors: Span of control/agency jurisdiction; PBRA history/evolution of state-of-the-practice; and Types of resources to be allocated. This involves defining the point in either the plan development or project delivery process where targets will support the decision-making process (e.g., project evaluation and selection, systems-level review, project design, project delivery, or monitoring of on-the-ground perfor- mance). This is closely related to how an agency has implemented the Performance Management Framework, as well as how much experience an agency has applying PBRA. WHEN Should Targets Be Attained? "When" requires thinking about the following factors: Timeframe; Types of resources to be allocated; Stakeholder expectations; and Commitment to regular communications and reporting. Political/Legislative influence This specifies the time horizon for when the target should be met. A distinction should be made between long-term goals about desired performance levels and short-term targets that rep- resent the best that can be done given known resources. HOW Will Targets Be Calculated and Achieved? "How" requires thinking about the following factors: Span of control/agency jurisdiction; PBRA history/evolution of state-of-the-practice; Financial resources; Technical resources/planning and forecasting capability; Timeframe; Political legislative influence; Organizational structure; and Internal support/culture. This question can be answered based on two levels of technical rigor. The first, which is less rigorous and perhaps most relevant to the long-range planning aspect of PBRA, is to simply define in broad terms the types of strategies intended to support meeting the target, whether it be by particular project types or investment strategies. The second, which is much more technically rigorous, relies on comprehensive technical analysis that defines how much improvement can be achieved given certain resource allocation tradeoffs and consideration of funding constraints

OCR for page 79
Guide for Target-Setting II-11 and various implementation scenarios. The second level of detail, if applied, should feed directly into what the target actually is, as it provides a more detailed perspective on what can actually be attained given certain real-world constraints. However, actual approaches for calculation and achievement of targets depend highly on internal resources and both internal and external influences. WHAT Is the Target? This step is the point where a target is actually established, given that the previous questions have been adequately answered. By asking these questions and considering the typical factors affecting target-setting, the practitioner can select the appropriate approach or approaches for setting targets and arrive at targets that suit an agency's needs. Typical Factors to Consider There are multiple factors that lend themselves to the development of a PBRA process within transportation agencies. These factors are documented in Volume I and in the Case Studies in Volume III. These are the factors that arise when asking the 5 W's and an H questions in the pre- vious section. When reviewing and refining performance targets, it is important to keep in mind that setting targets typically involves balancing a number of factors, which may vary in importance among different measures or products/services. The driving factor is not always the same. Political/Legislative Influence Perhaps the most immediate and direct factor influencing target-setting as an element of PBRA is the existence of a commission or other political body to which a transportation agency must report the performance of investment decisions. Political intervention in the process may result from controversy or the increasing public outcry over transportation services that force political attention on an issue. While political influence of this direct manner can have very complicated repercussions, it has shown to be one of the most positive indicators for implementation of target-setting, if done prop- erly. For almost every agency reviewed that is using targets as part of their PBRA process, politi- cal or legislative intervention provided the initial impetus for establishing discrete targets. Political intervention can be triggered by a number of issues, but the most common is the increasing lim- itation of transportation funding at all levels of government, which has created more competition for available funds and also made it more important to justify funding requests. The Minnesota Legislature and Department of Finance require that agencies use performance measures in biennial budget documents. Washington State's Legislative Transportation Com- mittee recommended a PBRA process in 1991 that's still in place today. Political and legislative influence also could result in an "edict from above," obviating the need for an agency to define its own target for a measure (MTC, ARC); sometimes the edict is broad enough that there is still room for more refined target-setting within an agency or division (FDOT). This can be difficult to negotiate if the process is not properly informed by knowledgeable transportation staff that guide the development of reasonable, attainable targets, considering planning, technical, and funding constraints for an agency (in terms of both ability to achieve and ability to measure progress towards a target). However, if specific targets already have been set externally, it may be necessary to complete the steps of the overall approach outlined in this

OCR for page 79
II-12 Guide for Target-Setting and Data Management guidance--namely, Establish Methods for Achieving Targets, Track Progress Towards Targets, and Adjust Targets Over Time--and through the feedback loop, better inform the target-setting process through the next planning cycle iteration. While political involvement can be challenging, it also can provide transportation staff the support they need to select projects that are proven to improve performance and therefore should be a priority for funding. Transportation agencies should guide the development of reasonable, attainable targets by legislatures. However, targets set by external edict may need to be refined through iteration of the target-setting process over time. Customer Service Focus Those public agencies and private organizations that have taken a clear customer-service approach to PBRA understand the need to use targets to be able to communicate to the sys- tem user, the "customer," the return on their investment. Customer satisfaction is a funda- mental aspect of performance for these organizations, and is therefore reflected in the types of measures selected, the measures that are given targets, and how the targets are created. They break down and analyze customer satisfaction or dissatisfaction from broad perspec- tives that address issues in areas such as social and community impacts and environmental impacts of transportation investments and, in more narrow terms, that address issues related to daily personal travel needs. In Coral Springs, Florida, customer input drives the decision-making process; target- setting serves as a metric for progress, providing feedback for the community. For ABC Logis- tics, targets stem from the promises made to its customers in their contracts with the logistics company. A Customer Care program monitors the company's performance relative to the targets and customer expectations, with feedback mechanisms to communicate results to the public and to link individual employee performance (and merit) to performance of the company. PBRA History/Evolution in State of the Practice Agencies that are only at the beginning of implementing a performance-based process gener- ally have less complete and less sophisticated target-setting processes. In general, there is a typical evolutionary path that agencies follow. A corollary to this evolution is the emergence of an agency's data sophistication. Mn/DOT's measures and targets, identified in its 2003 Statewide Long-Range Plan, were refined through application and adjusted in the 2009 plan. MTC's performance-based process also has been evolutionary during the development of its last three long-range plans. WSDOT's experience also confirms that target-setting requires a solid history of performance data as well as managerial comprehension and appreciation of that data, which comes with time and expe- rience. Managers must have the ability to understand transportation system behavior (i.e., "what the data are saying") and to discern what they can or cannot control. Even imperfect measures and targets, if they are well-established, are well understood and have previous reference points (DIY Company). Commitment to Regular Communication and Reporting Regular tracking of investment performance and reporting of results to the public and trans- portation stakeholders serves to focus attention on an issue over time, so that it is not lost in political and public discourse as new challenges arise. Regular reporting and communication of

OCR for page 79
Guide for Target-Setting II-13 progress helps to keep staff and the public focused on the particular challenge, especially as is in the private sector when it is tied to agency or even staff-level "merit"/compensation. This helps all involved to understand the nature of solving process problems over the long term, rather than focusing on immediate issues (e.g., fighting fires) that often distract from the larger mission of an agency. It also supports longer-term trend development that is needed to track the perfor- mance of investments over time. This commitment to transparency will affect the approach an agency selects to define targets as well as the timeframe of the targets. Reporting can be both internal and external for public sector organizations, and usually just internal for private sector organizations (with the exception of financial results for sharehold- ers). Many agencies use regular reporting to drive their own internal resource allocation (e.g., finances), as well as to satisfy external requirements (MLIT, OOCEA, WSDOT, MDOT). Report- ing mechanisms also interface strongly with managing stakeholder expectations and help to make an agency more customer-service focused. The regular development of reports or other "products" that are distributed to the public and used by decision-makers helps to maintain staff enthusiasm for the performance-based process and ensures the continued development of the necessary inputs for the report (Hennepin County). Span of Control/Agency Jurisdiction The span of agency control, whether it is through funding, modal authority, or geographic jurisdiction, plays a strong role in the development of measures and targets, because it controls the perspective from which each investment is evaluated. An agency that manages only highways will have a narrower set of measures than does an agency with jurisdiction for multiple modes. States responsible for all roads, rather than only the higher functional classes, face greater data- gathering complexities. This can influence how they set targets and how they use data to mea- sure progress towards those targets. For instance, Mn/DOT has direct control over the quality of pavement, but it can only influence transit service provided in Greater Minneapolis through funding. In all instances, the level of influence that the department has over a particular measure affects the target that is eventually set. Within DOTs, standard siloing of functions has led to strong asset management systems for roadway maintenance functions, but this process is trans- lating to other DOT functions. Financial Resources No constraint or factor in constraining the PBRA process and affecting target-setting is cited as much as financial resources. Financial resources are intimately intertwined with the resource allocation process, both determining an agency's ability to implement such a process by influ- encing other factors such as technical resources, and also potentially being determined by the process itself. Financial resources also can be used to create "financially constrained" targets that reflect historical or projected funding (MLIT). Performance data has played a key role in biennial state legislative budget allocations for Mn/DOT, and it also has played an important role in the debate for new transportation fund- ing. Mn/DOT quantified its highway performance measures and targets in its 2003 State Trans- portation Plan and concluded that Minnesota was underinvesting in its highway program by one billion dollars per year. This performance-based analysis was accepted by the legislature and virtually ended the legislative debate on level of need. The legislative discussion shifted from the question of need to the question of payment. In February 2008, the Minnesota Legislature over- rode the governor's veto and passed a funding bill which provided several billion dollars of new funding for transportation over 10 years. Similar to the influence of financial resources on target-setting, staffing and person-hours of available time can affect the depth of a PBRA program, including targets that an agency can assemble and monitor.

OCR for page 79
II-14 Guide for Target-Setting and Data Management Timeframe The timeframe of desired results affects how targets will be set and what they will be. Time- frame is sometimes determined by stakeholder and internal agency needs; it also can be dictated by forecasting capabilities. Although private sector senior managers and the finance departments usually have long-range goals, most corporate targets are set and performance moni- tored on an annual basis. To link individual performance to longer-term goals, companies sometimes use equity to incent employees rather than to share multi- ple annual targets with them. Many agencies have a variety of different timeframes for different planning and programming purposes, with targets for each of the timeframes. At Japan's MLIT, annual targets are derived in part from the latest major subjects of policy, planning, and programming to emerge from the funding reports from the MLIT and Road Bureau, the Road Bureau's Mid-Term Visioning Report, and the national government's 5-year Major Infrastructure Development Plan. Longer- term targets match this with a 5-year span. This information is used when determining feasible 5-year goals for the Bureau and results in what is essentially a financially constrained target. Maryland uses a similar approach with annual targets, 4-year targets, and 5- to 10-year targets, depending on the particular planning document. In Washington, the primary responsibility for translating long-term goals (dictated by elected officials) to short-term or "incremental" goals, objectives, and targets falls to the Department of Transportation, in consultation with executive and legislative members and staffs. This process centers on how to set and describe these incremental milestones, how to communicate them to the public, and what legal liability the State may incur by promoting these short-term targets publicly. WSDOT managers also may consider alternatives and adjustments in the engineering solutions to problems, in the methods of service delivery, and in the construction materials and techniques to be used in order to address these short-term targets. These options help to achieve stated targets within current funding and other resource constraints and thus maintain consis- tency between short-term program accomplishments and long-term aspirational goals. Technical Resources The presence or lack of forecasting tools can influence greatly the sophistication of forecasted targets. Agencies that have used HERS, PONTIS, and other tools for forecasting the results of long-term programs have greater insight with which to set long-term targets. Availability of analysis tools to identify performance impacts of projects realistically and effi- ciently and to track performance in relation to targets will determine what measures and targets can be used. Sometimes agencies develop desired measures and targets, even when data are not yet available, as a means of creating a "wish list" of data sources. Often it is difficult for decision- makers to see the need for data collection for a single performance metric, particularly if it appears to be part of a single endeavor (e.g., a long-range plan); if it is part of a larger, compre- hensive PBRA process, however, it is often easier to justify additional data needs. Evolution of tools over time makes it very difficult to track progress consistently. Change over time in tools, data, and analysis procedures, as well as differences between agencies and jurisdic- tions, can make it difficult for stakeholders--and even internal staff--to properly interpret the results. Staff turnover also can exacerbate this situation. Agencies must develop ways of main- taining their institutional knowledge base to properly utilize evolving tools and procedures.

OCR for page 79
Guide for Target-Setting II-15 Those with strong forecasting capabilities are more apt to have long-term goals and targets, whereas companies with weaker forecasting capabilities are more inclined to set annual targets only. For example, Corporation X, which has a robust financial modeling system, sets targets for several forecast periods. Development of strong tools and data give agencies the ability to not only calculate targets in a rational way, but the ability to measure progress towards those targets or the resources neces- sary to reach them. These abilities lend credibility to agencies with their stakeholders and the public (MTC). However, tools are often not equipped for a rigorous performance analysis at the project level for long-range planning or comparison between modes. Organizational Structure The organizational structure of an agency affects the structure of that agency's PBRA program and process as well as the development and purpose of targets. Centralized organization can allow the central office to work directly with the state government and stakeholders to establish targets. It also allows the organization to handle a somewhat complex and involved target- setting process (WSDOT). Private companies with multiple business units often hold up the highest-performing business unit as the benchmark that the others should strive to beat. The element of competition is often more important than the specific targets, so some companies let their divisions have a broad role in determining which metrics to use. Conversely, highly decentralized structures result in possibly different targets (and different resource allocation priorities and decisions) for different measures in different districts, but with broader and more flexible measures and targets at the central office (FDOT). This structure requires a strong but flexible performance management system to ensure consistency across districts in terms of achieving statewide goals. FDOT's overall Business Plan seeks to maintain accountability and transparency for processes that may not be standardized across the department. Stakeholder Expectations Similar to political and legislative influence, stakeholder influence can have a very significant impact on target setting. When external stakeholders become engaged in the process, they can influence the development of goals, measures, and targets. Typically, the use of specific targets derived from an internal process is seen in agencies with a more sophisticated and well-developed PBRA system that has developed over several iterations. Staff resources and, time permitting, an internally developed process can lead to very meaningful and effective targets within an agency's PBRA process. Agencies often establish targets through a committee process that provides for stakeholder input. As such, it provides an opportunity for dialogue about the transportation issues, constraints in funding, and other topics, which leads to the development of realistic and meaningful targets. This is absolutely critical for state DOTs and MPOs who make decisions in a very litigious environment. It is critical to communicate not only the lofty, long-term goals for transportation systems but the reality of fiscal and political and regulatory constraints as well; this will allow for stakeholder expectations to be managed from the beginning. Keeping flexibility in the target- setting process also is critical in properly setting stakeholder expectations. Further, transportation