Cover Image

Not for Sale



View/Hide Left Panel
Click for next page ( 2


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
I-1 SUMMARY Target-Setting Methods and Data Management to Support Performance-Based Resource Allocation by Transportation Agencies In July 2006, representatives of several state Departments of Transportation (DOTs), Met- ropolitan Planning Organizations (MPOs), and the Federal Highway Administration (FHWA) met in La Jolla, California, to discuss the use of performance measures in resource allocation and the data systems required to support an emerging business practice known as performance management. The workshop produced several research statements that were funded in early 2007 through the NCHRP program, including this one titled Target-Setting Methods and Data Management to Support Performance-Based Resource Allocation by Transportation Agencies. The study was designed to help public sector transportation agencies develop and improve performance management practices through the following three key objectives: To provide an overall description of Performance-Based Resource Allocation (PBRA); To provide a comprehensive description of the process and methods by which targets are set for use in PBRA; and To provide a comprehensive description of the data, information systems, and institutional arrangements needed to support PBRA decision-making. During the two years in which this study was conducted, there has been much discussion about the need to establish a performancedriven, outcome-based Federal Highway Program as a requirement of the next surface transportation authorization act. While this study was not intended to focus on a new national performance-based program, its findings and rec- ommendations are none-the-less germane to the issues currently being debated regarding target-setting and data systems. Framework for Performance-Based Resource Allocation PBRA takes place within an overall Performance Management Framework, depicted in Figure S.1, which is comprised of six basic elements: Establish Goals and Objectives. PBRA decisions are anchored in a set of policy goals and objectives which identify an organization's desired direction and reflect the environment within which its business is conducted. For example, many state DOTs have well-defined goals for the transportation system, including infrastructure condition, level of service and

OCR for page 1
I-2 Goals/Objectives Performance Measures Target Setting Quality Data Evaluate Programs and Projects Allocate Resources Budget and Staff Measure and Report Results Actual Performance Achieved Figure S.1. Performance management framework. safety, as well as goals reflecting economic, environmental and community values. Likewise the private sector frequently establishes policy goals to guide production of products and ser- vices while defining the environmental and community context for its investment decisions. Select Performance Measures. Performance measures are a set of metrics used by organizations to monitor progress towards achieving a goal or objective. The criteria for selecting measures often include the following: Feasibility, Policy sensitivity, Ease of understanding, and Usefulness in actual decision-making. Identify Targets. Targets are a quantifiable point in time at which an organization achieves all or a portion of its goals. These points set a performance level for each organiza- tional measure, such as achieving a 25 percent reduction in highway fatalities by 2030. The methods used to set such a target include: Establish Performance Management Framework; Evaluate the Factors Influencing Target-Setting; Select the Appropriate Method(s) for Target-Setting; Establish Methods for Achieving Targets; Track Progress Towards Targets; and Adjust Targets Over Time.

OCR for page 1
I-3 Allocate Resources. The allocation of resources (time and money) is guided by the inte- gration of the preceding steps into an organization's planning, programming, and project devel- opment process. To the extent possible, each investment category is linked to a goal/objective, a set of performance measures, and a target. Specific investment proposals are defined in rela- tion to specific targets. Measure and Record Results. The data for each performance measure must be regu- larly collected and periodically analyzed. The analysis should indicate how close the orga- nization is to achieving its targets and identify the actions necessary to improve results. Many public and private sector organizations have tracking systems in place to monitor perfor- mance allowing senior staff to make periodic budget adjustments. Create Data Management Systems. "Good" data is the foundation of performance management. Effective decision-making in each element of the performance management framework requires that data be collected, cleaned, accessed, analyzed, and displayed. The organizational functions that produce these requirements are called data management sys- tems. There are two key dimensions to creating and sustaining these systems. The two areas are equally important and must be synchronized within an organization to ensure the gen- eration and use of accurate, timely, and appropriate data. The first area centers on the tech- nical challenges associated with data systems, including development and maintenance of hardware and software, and the specifications for data collection, analysis, archiving, and reporting. The second area focuses on the institutional issues associated with data steward- ship and data governance. The Role of Targets in Performance-Based Resource Allocation As indicated, performance management is a business process that links organization goals and objectives to resources and results. Performance measures, and their corresponding tar- gets, are the lynchpin in the process. They provide the direct link between the stated goals of an agency and the effectiveness of its investment decisions in reaching those goals. Perfor- mance measures, used along with well-defined and well-communicated targets, provide transparency and clarity to the resource allocation decision-making process. Targets, them- selves, provide the critical context for evaluating the effectiveness of investment decisions. For example, a performance measure will define how an investment decision will be evalu- ated in terms of its impact in absolute terms. The corresponding target, provides the perspec- tive for evaluating the impact of the investment decision in relation to the desired end state, i.e., how significant is a particular investment in helping an agency attain a particular goal. Targets provide the means in which the relative effectiveness of a particular investment deci- sion can be clearly communicated. Because targets play such an important role in PBRA, this section of the study focused on the factors that influence target selection and the approaches by which targets are actually established. Towards that end, we have established the following steps for target-setting: Step 1--Establish a Performance Management Framework. Establish the framework that links organizational goals to resource results. Performance measures and their atten- dant targets are the link connecting goals to specific investments. Step 2--Evaluate the Factors Influencing Target-Setting. There are several internal and external factors in an agency that affect target-setting. These factors include--political/ legislative influence, customer and stakeholder perspective, agency experience in using

OCR for page 1
I-4 performance measures and targets, commitment to regular communicating and reporting, span of agency control, financial resources, and timeframe. In assessing these factors and others, an agency needs to answer the following questions: Why is target-setting needed? Who will be using the targets? Where in the agency decision process will targets be used? When should targets be attained? How will targets actually be calculated? How will targets be achieved? Step 3--Select the Appropriate Approaches for Target-Setting. Based on the factors in Step 2, select an approach or approaches for setting targets. Approaches for setting targets range from unilateral executive edicts based primarily on experience to collaborative senior staff deci- sions guided by relatively sophisticated modeling techniques available for some measures. In practice, however, most agencies use a hybrid approach in which they not only use different approaches for different measures but also multiple approaches for a single measure. For exam- ple, an agency could use modeling combined with customer/stakeholder feedback to arrive at a target that is both analytically grounded (to ensure a connection with predicted outcomes based on resources and existing plans) and satisfactory to the public and agency partners. Step 4--Establish Methods for Achieving Targets. Within the context of the Perfor- mance Management Framework, identify methods that orient the agency and its resources towards achieving the targets set in Step 3. Public and private organizations alike use several specific methods to achieve targets. Most critical to this, broadly speaking, is the integration of performance measurement into daily agency activities. This directs attention to key issues, including financial resources and data support systems. Other methods include establish- ment of funding allocation incentives and the integration of performance target attainments into personnel performance appraisals. Step 5--Track Progress Towards Targets. As part of the "Measure and Report Results" element of the Performance Management Framework, track performance progress specifi- cally against targets. Virtually all public and private organizations that employ performance management track the impact of their investments in achieving specific targets. Techniques vary. Some use a Balanced Scorecard in which numerous measures are evaluated and tracked in terms of multiple perspectives (customer, finance, internal processes, learning, and growth) and simplified into tables of information providing "warning lights" for areas in need of improvement. Other organizations prepare periodic performance measure "snapshots" in which red, yellow, and green colored shapes represent annual progress relative to targets by geographic area. Other agencies publish annual attainment reports. Step 6--Adjust Targets Over Time. Based on financial and political realities, ease, or difficulty of achieving targets, and increasing experience in PBRA, use the feedback loop in the Performance Management Framework to reevaluate and periodically adjust targets. Fac- tors driving possible need for adjustments from a policy perspective include changes in the level of funding or in the rules governing project eligibility. When adjusting targets, agen- cies also should consider resolution of issues relating to model updates and data collection methodologies that may be influencing the calculation of the target. Data Systems to Support Performance-Based Resource Allocation Recent calls for more accountability in government have focused attention on the methods by which public agencies make decisions and the underlying data upon which those decisions are based. At the same time, transportation agencies are struggling with budget issues forcing the issue of getting the most "bang for the buck" through the examination of all programs to

OCR for page 1
I-5 ensure maximum value to the agency. With the pending Transportation Authorization and its certain emphasis on performance measures and data to support them, the issue of establish- ing and maintaining data programs at the state and regional levels to support these needs is prevalent. Federal needs for data to support national reporting and programming will con- tinue to be a priority with programs such as the HPMS reassessment, Intellidrive, and Freight Data Management to name a few. Furthermore, state transportation improvement programs are emphasizing more collaborative transportation decision-making, which in-turn requires improved data programs. PBRA in any organization relies on the availability of timely, accurate, high-quality data which is easily accessible through a framework known as a data system. Various data systems throughout the organization serve the needs of decision-makers in multiple business areas. Data are the basic pieces of information which, when processed through a system, are avail- able for analysis. The core data pieces transform into information, and decision-makers then use this information to manage business functions across the organization. The process for ensuring that data is of the highest quality possible is known as a data management process. Data management programs are used to manage the data systems within the organization. Data management can be defined as the development, execution and oversight of architec- tures, policies, practices, and procedures to manage the information life-cycle needs of an enter- prise in an effective manner as it pertains to data collection, storage, security, data inventory, analysis, quality control, reporting, and visualization. There are many ways to approach estab- lishing a data management program, however, one of the most effective ways is to incorporate data management in concert with an overall data governance framework. Data governance can be defined as the execution and enforcement of authority over the management of data assets and the performance of data functions. The implementation of data governance includes participants from many areas of the organization. These individ- uals are usually already performing many of the roles identified with data governance, but their job functions have not been aligned within a formal data governance structure. For instance, persons within a business unit who enter data into a system and are responsible for the quality of the data are referred to as data stewards within the data governance model. A hierarchical relationship exists between data management, data governance, and data stewardship as illustrated in Figure S.2. Besides those who collect and provide data, there are users of the data, known as stake- holders. These stakeholders form a Community of Interest (COI) for the data system. The COIs serve a vital role by identifying needs for data and information and helping to deter- mine where the gaps exist in data programs. This leads to a formalized process for evaluat- ing and ranking priority needs for future data systems, and for justifying the costs of such data program development, based on Return on Investment (ROI) to the organization. Assessment of existing data systems also is a key component of a strong data management program. The following steps outline how transportation agencies can use data management and governance to strengthen existing Performance Measurement and Target-Setting programs in the agency. Step 1--Establish the Need for Data Management/Governance. Define the important relationship between data management and performance measurement and provide a maturity model to assess the "state of data governance" at the organization. Step 2--Establish Goals for Data Management. Once an agency has committed to mak- ing improvements in their data management practices, a plan to achieve this should be developed. A strong Data Management program improves data quality and limits poten- tial risks to the agency regarding loss of critical data and information.

OCR for page 1
I-6 Data Management Data Governance (DG Board, Stakeholders, DG Maturity Model) Data Stewardship (Stewards, Owners, Custodians) Source: Modified from Figure 1 Data Governance Team, The Data Governance Maturity Model. White Paper, RCG Information Technology, 2008. Figure S.2. Data management, data governance, and data stewardship. Step 3--Assess Current State of Data Programs. There are tools and techniques avail- able to assist the organization in assessing the current data practices and programs. These tools are known as Business Intelligence (BI) tools, and this report provides several exam- ples of the use of BI tools in the Case Studies documented in Volume III. Step 4--Establish Data Governance Programs. The agency should develop and imple- ment a data governance framework model that best meets the needs of the organization. There is not a one-size fits all data governance model. Figure S.3 illustrates a generic data governance framework which includes all of the traditional participants in a data gover- nance model. Strategic Vision, Mission, Goals for Data Data Division(s) Mission(s) Governance and Goals Board Data Steward Agency Data Programs Data Users and and Custodians Stakeholders Figure S.3. Standard data governance model.

OCR for page 1
I-7 An important part of establishing the data governance model is to align the goals of the data programs to the business objectives of the agency as a whole. This is accomplished through the following steps: Step 4.1--Identify the business objectives of the agency. Step 4.2--Identify the business functions or services of the agency that support the business objectives. Step 4.3--Identify which business functions are supported by which data programs. Step 4.4--Establish policies, standards, and procedures which mandate how data is to be collected and used within the agency. Step 4.5--Establish Data Action plans on both a data program and enterprise level, to address needs and gaps in data and information across the agency. Step 4.6--Establish a risk management plan for protecting data programs as valuable assets of the agency. Step 5--Technology for Data Management. The agency should utilize the tools available to support data management, including knowledge management systems, risk manage- ment systems, geographic information systems (GIS), and visualization tools, such as dash- boards and scorecards. Step 6--Link Data Program to Planning, Performance Measures, and Target Processes. The agency should demonstrate how the data programs are linked to planning, performance measures, and targets. This can be done through business use case examples and concept of operations documentation. Importance of a Data Business Plan The establishment of a data management program in a transportation agency can be achieved through the use of a formal data business plan. Many agencies incorporate com- ponents of their strategic plan into the data business plan, to ensure that data programs are aligned with the strategic mission and objectives of the agency. A data business plan helps to: Establish goals; Assess agency data programs; Establish data governance; Ensure proper use of technology/tools; and Link data management to performance measures and target-setting. There is a variable cross-section in state transportation agencies from those that have developed and implemented data business plans, such as Virginia DOT, to those who have made good progress in developing a data business plan, such as Alaska DOT, and those have just begun the process to develop a data business plan. Still others have not formalized a data business plan for their agency but can still derive benefit from exam- ining the examples provided by peer states. Target-Setting and Data Management Challenges and Opportunities It is important to acknowledge that there are both challenges and opportunities associated with setting performance targets and establishing a performance management data business plan. The opportunities derive from a transparent data-based decision process that clearly defines the nature of agency investments. The challenges are both institutional and technical in

OCR for page 1
I-8 nature. There are many examples, however, from both the public and private sector in which organizations specializing in transportation not only demonstrate the use of performance tar- gets but also illustrate how data management and data governance are used to manage data pro- grams which support performance measurement. These examples are all detailed in Volume III of the report, the Case Studies. Each of the challenges offers an opportunity to improve target- setting and the delivery of data and information to decision-makers across the organization. As illustrated in the Performance Management Framework in Figure S.1, data is a very important factor in the PBRA process. For instance, once an operational data management program is implemented, it needs to be integrated with the agency performance measures and target-setting process. The success factors to achieving this critical step are the following: Use a hybrid approach that employs modeling and benchmarking to establish agency tar- gets and performance measures. Do not use a one size fits all approach in establishing performance measures and targets. Use the correct metrics for making decisions. Focus on continuous improvement by revising/adding new metrics as needed. Link the performance measures and targets for a program to budget allocations, improv- ing participation by staff in supporting the performance measures and targets. The per- formance measure and target-setting process also can be used to motivate employees by linking their performance plans to objectives identified in specific performance measures and targets. Allow DOT transportation planning staff routine access to other planning offices (regional, district, etc.) and technical resources available in the agency. This strongly enhances a performance-based management process. Reward business areas which consistently meet targets and goals. Consistent achievement in meeting targets is a powerful motivator for behavior--success breeds success. Use external data sources, such as environmental, historic, and other planning agencies for GIS data layers to improve the data used for the performance measurement process when funds are limited to collecting this data using internal resources. Utilize software that is procured or developed internally to automate as much of the per- formance measurement process as possible. This will allow for more time devoted to the analysis of the performance results. Revise or stop using targets if performance data are not easily obtainable when a perfor- mance target is used. Programs which do not have a direct link between that program or project and performance should not be funded. Identify business units responsible for maintaining current metadata about each perfor- mance measure. This facilitates the analysis required for user requested data and informa- tion system changes and enhancements. Include objectives pertaining to resource allocation in the agency Business Plan. The cur- rent Business Plan at the Maryland Transportation Authority (MDTA), for example, has three separate objectives related to resource allocation. These include System Preserva- tion, Implementing and Asset Management System, and Integrating MDTA's financial system with other systems. Use external data sharing agreements to obtain data for performance measures that the agency does not have. For example, MDTA collaborates with other agencies for several measures that it needs additional data for, or does not have the necessary equipment to monitor itself. Establish performance targets through a streamlined process and revisit and revise (as needed) periodically.

OCR for page 1
I-9 Incorporate customer satisfaction as a measure in setting performance targets. Utilize incentives to facilitate meeting performance objectives, including awarding bonuses based upon job performance and using quantitative objectives embedded in professional employees' annual objectives. Arrange performance measures in a hierarchical order, allowing an agency to translate strategic goals/objectives into operational goals/objectives for each department. The U.S. DOT follows this approach among its various administrations (e.g., FHWA and FTA), allowing it to provide a performance budget that can be related to actual and planned accomplishments for each department. This same scenario would apply to a state DOT, with several divisions, districts, and/or independent offices. The performance in each area then becomes a key basis of resource allocation and budgeting. It is ultimately up to the transportation agency to take full advantage of the benefits that a fully functional data management program will offer for decision-making in a transportation environment.