U.S. and International Approaches to Performance Measurement for Transportation Systems (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

PLENARY SESSIONS

3The opening plenary session of the conference illus-trated and discussed the ways performance mea-surements are being used to assess the effectiveness of organizational operations and service performance at the regional, state, and national levels in the United States and abroad. Suggested topics for further research are identified in the breakout sessions corresponding to each of the ple- nary sessions and are compiled in Appendix A, pages 147–149. CONFERENCE WELCOME Lance A. Neumann Welcome to the Transportation Research Board’s (TRB’s) Third International Conference on Performance Mea- surement. A special welcome is extended to our interna- tional participants who have traveled from Japan, Mexico, Canada, Great Britain, and Sweden. I extend thanks to the members of the Conference Planning Committee, who did a great job of organizing informative and interesting sessions. I also thank the Fed- eral Highway Administration (FHWA) and the Federal Transit Administration (FTA) for their financial support of the conference. Representatives from four FHWA offices— planning, asset management, operations, and safety— are participating in the conference. Thanks also go to the TRB staff that assisted with the conference, especially Martine Micozzi. The use of performance measures by transportation agencies has increased significantly since the first confer- ence in 2000. The first conference focused primarily on the theory and concepts of performance measurement. By the second conference in 2004, we were able to talk about the actual experience of applying performance measures in transportation agencies at all levels of government. Speakers at this conference will highlight the use of performance measures as a strategic management tool at state departments of transportation, public transit agen- cies, railroads, other public agencies, and private busi- nesses. Many agencies have fully recognized the power of performance management. The Planning Committee has structured a mix of gen- eral sessions and breakout sessions to provide partici- pants with the opportunity to engage in discussion and dialogue. We hope to generate an agenda of research needs, workshops, technical assistance, and other activi- ties to promote the use of performance management con- cepts and performance measurement as a strategic management tool. WELCOME FROM THE CALIFORNIA DEPARTMENT OF TRANSPORTATION Will Kempton I thank TRB, FHWA, FTA, and Lance Neumann and the Conference Planning Committee for organizing this very important conference. It is vital that we take time to dis- cuss performance management and the importance it has for all of us in the transportation field. OPENING PLENARY SESSION Performance Measures as an Organizational Management Tool to Establish Accountability Will Kempton, California Department of Transportation Pete K. Rahn, Missouri Department of Transportation Archie Robertson, Highways Agency, United Kingdom John Gray, Union Pacific Railroad

I welcome Pete Rahn and Rhona Faught, my counter- parts in Missouri and New Mexico, respectively, who will also be participating in the executive roundtable dis- cussion this evening. I also welcome Archie Robertson, the chief executive of the United Kingdom Highways Agency. Why are performance measures so important? One of the concepts I learned from reading Peter Drucker’s books on management is that “what gets measured gets done.” I think that statement is true. I also think the pub- lic deserves to know that its money is being spent wisely. The public is beginning to demand to know more about how its tax dollars are being invested and is holding agencies more accountable for delivering on promises. As a result, government agencies have to be more effi- cient, more accountable, and more transparent. By trans- parent, I mean we have to provide information that demonstrates that we are spending the public’s tax dol- lars judiciously. Whether we are involved in preparing for road closures, tackling major construction projects, or implementing measures to reduce congestion— an important focus here in California— performance mea- sures ensure that we are operating more efficiently. From the beginning of his term, Governor Arnold Schwarzenegger has reported to the public on perfor- mance measures. One of his first activities after assuming office was to create a California performance review exercise. This exercise brought together experts to exam- ine current operations and to recommend ways to improve efficiency. More than just thinking outside the box, he wanted to blow up the box if necessary. We are committed at the California Department of Transportation (Caltrans) to a new philosophy and approach to doing business in the state. We like to say we are no longer a transportation bureaucracy, rather we are a mobility company. We have integrated key performance measures and strategies to help determine progress and success. There are three major components in our plan to implement performance management. First, we have an overall strategic plan. The strategic plan now drives the agency’s budgeting process. Second, the annual operations plan provides the implementation document for the agency. Third, performance measures have been set to monitor and evaluate whether we are meeting our objectives. It is vitally important that we evaluate our performance. Our strategic plan sets forth our vision, mission, goals, objectives, and strategies. It also includes Caltrans values. We felt it was important to communicate to our employees and the public that Caltrans stands for spe- cific values. We offered employees the opportunity to participate in developing these values, and we have empowered them to promote actions that reflect these values. Caltrans now has some 22,000 employees, including approximately 5,000 maintenance workers, whose access to e- mails is as frequent as for other staff. I com- municate with all Caltrans employees every 3 to 4 weeks by video e- mail. This message is provided to mainte- nance workers on a DVD, so that they can watch it as part of their regular safety meetings. About 3,300 employees actively participated in the strategic planning process. The Caltran’s operational plan identifies specific proj- ects, programs, and resources to meet the strategic goals and objectives. The performance measures are used to monitor progress toward meeting those goals and objec- tives. The performance measures also allow us to deter- mine whether a target is met and how our resources are used. This process ensures that resources are allocated to the highest priority projects and programs. For example, we work on programming and budget- ing with the California Transportation Commission, which is the state’s funding allocating agency. We use performance measures in the project selection process. State and regional agencies use the performance mea- sures to link projects to their regional transportation plans, which strengthens the connection between long- range planning and programming. Caltrans representa- tives will be providing more information on these processes in the breakout sessions. I will close by highlighting the governor’s Strategic Growth Plan. The plan is a $43 billion investment in infrastructure in the state. In November 2006 the voters approved a bond package for the program, which includes funding for highways, roadways, public trans- portation, ports, and other facilities. On the basis of the Strategic Growth Plan, we have an outcome- oriented program for transportation in the state. Outcome- oriented elements include reducing traffic congestion below current levels within 10 years. We have metrics in place to measure that outcome, and we have $19.9 bil- lion in approved bonds to fund a broad range of trans- portation projects to address congestion. The governor has promised on- time delivery, accountability, and trans- parency with the plan. John Njord, director of the Utah Department of Transportation (DOT), has done an excellent job of communicating with the public and with policy makers. Utah DOT’s objectives and performance measures are posted on the department website. The agency uses a dashboard to provide updated information on the status of projects and programs. This straightforward approach provides key information to various stakeholders. The performance measures we are proposing for the projects funded through the bond program are perfor- mance driven and action oriented. We also have an 4 U.S. AND INTERNATIONAL APPROACHES TO PERFORMANCE MEASUREMENT

industry expansion exercise under way because we want to ensure that the industry is able to accommodate some $20 billion in projects. We have also set targets for each of 12 Caltrans districts for program and project delivery. In the past 3 years, we have achieved close to 100% of project delivery commitments. In 2005 and 2006, of 174 major projects 173 met four specific project milestones, by quarter. Those projects were valued at about $2 bil- lion. In the past fiscal year, we delivered 100%—28 major projects, totaling $32.2 billion. Performance measures will play a key role in trans- portation agencies in the 21st century. Performance mea- sures translate into better decisions and better communication. I hope you will take what you learn at this conference and apply it to the challenges you face in your own agency and area. USING PERFORMANCE MEASURES TO ESTABLISH ACCOUNTABILITY Pete K. Rahn I was probably an illogical choice when I was appointed as the secretary of transportation in New Mexico. I did not have experience in either transportation or state gov- ernment. I have now been associated with the New Mex- ico DOT and the Missouri DOT. Both agencies were good organizations when I joined them. I believe the use of performance measures turned them both into superior organizations. When I first started at New Mexico DOT, I met with all the various agency staffs, including the district main- tenance workers. One of the first meetings was with maintenance personnel in one of the districts. I explained that it was the role of leadership to point out the direc- tion for the agency and that we will provide a safe play- ing field, and people throughout the organization will understand where the boundaries are. I further explained that we will empower our employees to use their talents and initiative to move the agency in the desired direction and that we will have performance measures to hold peo- ple accountable and to ensure that the agency is head in the direction the leadership is pointing. I noted that I was going to hold managers accountable for providing the tools and training for our employees to be successful and that I expected our employees to challenge their supervi- sors for the training and tools they need. An employee suggested that I was going to mess up the agency, because I would probably be gone in 4 years, after the next gubernatorial election, and their managers would remember the employees who had challenged them. This individual continued to challenge my com- ments during the meeting. It finally dawned on me that he was doing exactly what I was suggesting, to challenge management. I told him I would give him a day off with pay for having the courage to tell me what he thought. Another individual immediately raised his hand and said, “I don’t like you either.” This story highlights my belief that performance measures have to play a key role in the leadership and management of an agency. I will highlight how we are using performance mea- sures in Missouri. First, leadership points out the direc- tion for an agency. We establish boundaries for a safe playing field through the adoption of values. We empower our employees and we measure performance. In Missouri, we started with a strategic advance that involved 45 senior management personnel. After an extensive discussion, we agreed on a mission statement that states, “Our mission is to provide a world- class transportation experience that delights our customers and promotes a prosperous Missouri.” We sent the mission statement out to all our employ- ees and asked for their comments and reactions. I received 318 responses from employees. Most of those who responded did not like the use of “world class” because they thought we were setting ourselves up for failure. They also did not like the use of the word “delight.” It was suggested that making customers happy should be good enough. There is a difference between being just happy and being delighted. It is our vision to delight our customers, not just make them happy. After we agreed on the mission statement at the staff’s strategic “advance” (a term I prefer to use to connote brainstorming exercises and progressive thinking rather than the traditional term, staff “retreat”), we turned our attention to developing the playing field through our shared values. We identified 18 value statements. These value statements are to support and develop employees, to be flexible, to honor commitments, to encourage risk and accept failure, to be responsive and courteous, and to empower employees. Other value statements are to not compromise safety, to provide the best value for every dollar spent, to value diversity, to be one team, and to use teamwork to produce results. Still other value statements are to foster an enjoyable workplace, to be open and honest, to listen and seek to understand, to treat everyone with respect, to seek out and welcome ideas, and to strive to work better, faster, and cheaper. This list did not describe Missouri DOT at the time, and it does not describe Missouri DOT today. It does describe the organization we want to be, however. These values are used throughout the organization, including challenging leadership. We then discussed what the mission statement means to our customers and how we describe it in tangible ways that are understandable to our customers. We agreed on a series of tangible results including uninterrupted traffic 5PERFORMANCE MEASURES AS AN ORGANIZATIONAL MANAGEMENT TOOL

flow, smooth and unrestricted roads and bridges, a safe transportation system, roadway visibility, and personal, fast, courteous, and understandable response to cus- tomer requests. Other tangible results are to partner with others to deliver transportation services, to leverage transportation to advance economic development, to use innovative transportation solutions, and to promote fast projects that are of great value. Still further tangible results include environmental responsibility, the efficient movement of goods, easily accessible modal choices, cus- tomer involvement in transportation decision making, and convenient, clean, and safe roadside accommoda- tions. Being an advocate for transportation issues and providing the best value for every dollar spent, attractive roadsides, and accurate, timely, understandable, and proactive transportation information represent the final tangible results. On our website, MoDOT.org, we track some 130 per- formance measures. We add and delete performance measures as appropriate. We provide updated informa- tion on how we are doing on all the performance mea- sures. We also publish this information and distribute it to members of the state legislature, the governor, the media, and other groups. In developing performance measures associated with smooth roads, we focused on customer expectations. Missouri DOT conducts road rallies, which involve driv- ing randomly selected citizens and civic leaders along roads in the state. The participants grade road condi- tions related to pavement smoothness, lane and shoulder width, striping and signage, and other elements. This approach allows Missouri DOT to apply scores to what people find acceptable. The results were used to develop a baseline for measuring success. We also conducted sur- veys and focus groups to obtain more information from our customers. The results from the road rallies were used to develop our customer expectation standards. It is interesting that some of the items our customers rank highly are differ- ent from those we thought were important to them. Fea- tures with the highest ratings included the physical condition of the roadway, how well intersections are marked, traffic flow and congestion, ease of getting on and off roadways, bridge width, and smoothness. Some of the features we typically think people view as impor- tant, such as mowing and trimming and clearing road- side litter and debris, were rated low by participants. This information was used to define the characteris- tics of a good road and to develop benchmarks on how Missouri compares with other states. I have to confess; I do not believe in goals or in targets because agencies tend to set goals and targets they know can be met and then stop trying to improve once the target has been accom- plished for fear of using up any performance they may have to deliver the next year. I believe in continuous improvement. Our goal is to be the best in the country for every single measure. We will not let up until we accomplish that goal. Using the results from the road rallies, we examined the percentage of highways in good condition. In 2004, only about 44% of our major highways were in good condition. We had the third worst pavement ratings among states in the country. We were not meeting our customers’ expectations. We were able to use available bond proceeds to under- take a Smooth Roads initiative to address this problem. We dedicated resources to improving roadways in the state. By 2006, 74% of our major roadways were in good condition. Georgia has the highest percent of major roadways in good condition. We have a plan to overtake Georgia. We also examined the percentage of minor roads in good condition. We have experienced a slight decline in the percentage of minor roads in good condition— from 78% in 2002 to 69% in 2006—because of the higher priority of investing in major roadways. In 2006, a higher percentage of major roadways were in good con- dition. Georgia is also the best in the nation at maintain- ing its minor roadways. Again, we plan to overtake Georgia in that category. We also track the percentage of vehicle miles traveled on major highways in good condition to understand how many customers we are keeping happy. We monitor the percentage of deficient bridges on major highways. We have to do something drastic if we are going to improve those measures. We are not making improvements in the percentage of deficient bridges on either our major road- ways or our minor roadways. The need to do so has driven our behavior at Missouri DOT. We have been investing significant resources to bring deficient bridges up to standards. Our Safe and Sound Bridge Program targets 802 bridges in the state. We plan to contract with one team to reconstruct all 802 bridges within 5 years and to provide a 25-year warranty. We do more than just publish Tracker and perfor- mance measures. We follow up with quarterly Tracker meetings. Typically these meetings last from 6 to 8 h. The drivers that are the personnel responsible for each performance measure provide updates on what has been delivered, not what they are planning to do. I do not allow staff to discuss what they are planning to do. I actually blow a horn if people stray from talking about what they have been doing to what will be done. We focus on accomplishments and the delivery of our promises. All drivers are required to attend and partici- pate in the meetings. This approach creates a cross- divisions and cross- boundaries educational process for all staff. It provides personnel with a better understand- ing of activities throughout the agency. With 130 perfor- mance measures, the Tracker system provides a mechanism to hold people accountable. Through this approach we have been able to focus on our organiza- 6 U.S. AND INTERNATIONAL APPROACHES TO PERFORMANCE MEASUREMENT

tion and to significantly improve our delivery of services and programs to the citizens of Missouri. PERFORMANCE MEASURES: MANAGEMENT TOOL TO ESTABLISH ACCOUNTABILITY Archie Robertson I am passionate about the use of performance measures to drive an organization forward. Performance measures ensure accountability of the resources given to an organi- zation. Performance measures provide focus for an orga- nization. Performance measures enhance teamwork and ensure that you are satisfying your customers, that those who provided the funding are receiving good value, and that you are allocating resources appropriately. The Highways Agency is an executive agency of the U.K. Department for Transport. It is responsible for the operation and stewardship of the strategic road network, which is England’s motorways and major trunk roads, on behalf of the secretary of state for transport. I am the executive director of the agency and a member of the Department of Transport Board. Our funding comes from the U.K. government. We generate a small amount of funding from recovering the cost of building and oper- ating bridges. Increasing amounts of revenue are gener- ated from accommodating network enhancements for developers who want to develop property along our networks. The strategic road network includes approximately 4,800 mi of motorways and all- purpose trunk roads, representing about 2.8% of all roads in England. The counties are responsible for most roadways in England. Although it is only 2.8% of all roads, the network car- ries 31% of all road traffic and 62.3% of all freight. The network is critical to the economic viability and social well- being of the country. In total, there are about 80 bil- lion vehicle miles of travel every year. The value of the network is approximately £80 billion. We describe our role as being a network operator, rather than a road builder and maintainer of roads. This transformation to a network operator has occurred quite recently. We have approximately 3,500 civil servants at the Highways Agency. This figure has doubled in the past 3 years because we decided there was good value in putting traffic officers on the road network. Those indi- viduals are not police officers. Rather they focus on responding to accidents, clearing debris, and keeping traffic flowing. Approximately 1,500 staff are focused on road service and operate control centers. We spend approximately £1 billion annually on capi- tal projects and £1.2 billion annually on maintenance contracts and private finance initiative (PFI) payments. We have 13 PFIs in which we pay others to provide road services and information services. We are one of the lead- ing users of PFIs in the United Kingdom and in the global roads business. I calculate that it takes some 30,000 peo- ple to operate our network, so we rely heavily on con- tracted employees. As a result, I see myself not only as the leader of 3,500 agency employees, but as a champion for 30,000 workers that keep the network operating. Underpinning our work is our Customer Promise, which reinforces our commitment to put our customers first in everything we do. This promise focuses on help- ing our customers with their journeys. It states that we will help them make their journeys safely and reliably. We will provide value for money we receive and invest it in improved services. We will provide helpful informa- tion to enable our customers to make choices before and during their journeys. We will clear incidents quickly and safely. We will limit any delays when performing road- work and improvements. We will play our part in pro- tecting the environment. We will ask our customers for their views and act on feedback. We will deal with our customers promptly, courteously, and usefully. Everything we do is focused on fulfilling this promise. We have been successful in meeting the promise. Customer- focused organizations must listen to their customers. We conduct national and area surveys involving more than 8,000 customers throughout the year. These surveys include questions on the last journey the individual made. We have found that this focus on the most recent trip pro- vides very good information on our customers’ expecta- tions and satisfaction levels. The surveys provide an understanding of who uses our network and services and how they use them. The results allow us to compare per- formance against our objectives. The surveys also provide an indication of local concerns of road users and people liv- ing near our roads. Examples of issues identified by cus- tomers in the most recent survey include the number of roadwork projects under way, the accuracy of variable message signs, and concerns for the environment. Con- cerns about reliability, trip and travel information, and safety continue to be the three elements that are most important to our customers. When I joined the Highways Agency in 2003, the agency had 21 performance measures, which were nar- rowed down to four— improving safety, improving jour- ney time reliability, ensuring customer responsiveness, and supporting sustainability. As you know, unlike a rail or an airline network, a road network has no timetable. Travelers use the road network where and when they want. Providing accurate information to allow travelers to make decisions is an increasingly important part of our service. Travelers today are also much more con- cerned about the impacts of motoring activities on the environment. Road safety continues to be a major concern of our customers. The network remains one of the safest in Europe. We continue to exceed our road safety targets. Recently, there has been a 33% reduction in fatalities or 7PERFORMANCE MEASURES AS AN ORGANIZATIONAL MANAGEMENT TOOL

serious injuries and a 10% reduction in the rate of slight injuries, but still more than 2,000 people a year die or are seriously injured on our network. We know that engi- neering solutions are not sufficient in themselves. Driver information and education are becoming increasingly important. Improving the safety of our people who are working on roads continues to be an important initiative. We target journey reliability because that is what our customers tell us they want and because we feel we can make continued improvements through management and prevention of incidents on the network. Our target is to make journeys more reliable on the strategic road network by 2007–2008. This target is measured by improving in aggregate the slowest 10% of journeys for each of 98 routes, for each day of the week, for each time of day, measured in 15-min intervals, between the base- line period of August 2004 to July 2005 and the target period of April 2007 to March 2008. We manage this target through a subgroup of the Transport Board that meets on a monthly basis to review progress on various programs and projects. The group reviews the progress of interventions, the use of delivered interventions mea- sured against a range of indicators, and the actual per- formance measured against the target and the forecast performance. We have numerous efforts under way related to all our targets. As I mentioned, we are currently spending approximately £1 billion annually on capital projects. The road improvement program includes 113 schemes, involving roadway widening projects, bypasses, and junction improvements. A total of 47 schemes have been completed to date. Some 23 schemes are under construc- tion, at a value £1.8 billion. We have 37 schemes in the planning stage, at a value of more than £12 billion. The projects include the widening from six to eight lanes of key motorways used by commercial vehicles. We are working with suppliers and contractors to ensure that roadwork projects are better planned and executed through an approach I call “sweating the asset.” This involves increasing nighttime- only work to minimize the impact of roadwork on our customers, pro- viding better information to help our customers plan their journeys to minimize disruptions, using innovative equipment to speed up roadwork and improve safety, and making the best use of road space at all times. A number of initiatives under way are related to enhanced information services. These initiatives include providing real- time traffic information by digital radio, making real- time closed- circuit television images avail- able on our website, broadcasting quality images to travel news providers, and extending the trial of providing jour- ney and delay information on motorway message signs. We are also focusing on active traffic management, including the use of shoulder lanes as traffic lanes during peak periods. We use technology to assist with that process. We use variable speed limits during peak peri- ods and open the shoulder lane to general traffic. Com- puters monitor the entire roadway so that we are able to respond quickly to accidents or incidents. We have increased capacity on the network through this approach, without any new construction. We have learned a great deal during the past few years from asking our customers about their expectations and their satisfaction levels. We have learned that reliability is more of a concern to our customers than are actual travel times. Our experience indicates the importance of understanding performance measures before targets are set; targets should not be set hurriedly. In addition, the quality of data is important, as is ensuring that there is a reliable baseline to measure against. Finally, measures and targets should be kept simple. PERFORMANCE MEASURES AS AN ORGANIZATIONAL TOOL TO ESTABLISH ACCOUNTABILITY: PRIVATE- SECTOR COMMENTS John Gray In providing a private- sector perspective on the use of performance measures, my comments will focus on the performance measures associated with the freight indus- try in the United States, particularly the railroad industry. Let me begin by providing an overview of Union Pacific (UP) Railroad. We are a freight rail carrier with 32,300 mi of track. Our annual revenues are approxi- mately $15 billion, including some $100 million gener- ated by contracted passenger service. We operate in 23 states. UP owns approximately 105,000 freight cars and 8,500 locomotives. We operate some 2,700 freight trains on a daily basis and have approximately 50,000 employ- ees and 25,000 customers. Our annual capital improve- ment budget is about $3.2 billion, and our maintenance budget this year was $2.3 billion. Given these costs, you can see that we invest heavily in our infrastructure every year. Understanding the allo- cation of this capital is a critical issue for UP. The avail- ability and allocation of capital are essential issues in how one measures and looks at business. Although we operate in 23 states, including California and Texas, the state of Wyoming constitutes approximately 20% of our business. The major products are coal and soda ash, which are used at power plants that produce about 15% of the country’s electrical capacity. UP has a diversified business mix, which provides some balance in times of adversity in one area. In 2006, industrial products accounted for 21% of our business, compared with 20% for energy, 10% for intermodal, 8 U.S. AND INTERNATIONAL APPROACHES TO PERFORMANCE MEASUREMENT

16% for agriculture, 14% for chemicals, and 10% for automobiles. This year, our industrial business is lower, but our energy and intermodal businesses are higher. All components of the freight industry— rail, truck, water, and air— must measure operations and equipment utilization. For UP, measuring the infrastructure utiliza- tion in our terminals is critical. With the exception of rail and trucking, investment in transportation terminals for most modes is conducted in conjunction with one or more public agencies. Airports and water ports involve public agencies and public financing. The railroad indus- try is the only freight industry with a private line- haul infrastructure. For the most part we build, operate, and maintain our rail lines without public assistance. When we examine performance and effectiveness measures at UP, we have to deal first and foremost with the physical movement of commodities through our net- work. The value of the cargo is important, but is some- what secondary to the physical quantities and configuration of the cargo. The critical issue is how cargo is moved through the network. The value of cargo relates to mode selection, service selection within modes, and performance requirements. Network performance is an output measurement. Cargo value is one of the input requirements that define performance objectives. Ultimately, industry performance must be based on measures that will drive capital investment and the effi- cient management of network operations. Thus, perfor- mance measurements reflect the need to understand the efficiency of private capital utilization and the need for capital creation if the network is to be sustained. Perfor- mance measures also need to reflect requirements to maintain some degree of redundancy and network relia- bility within the limitations of network structure and capital availability. I would echo Pete’s comments related to basing the performance measurement structure on customer requirements. We have to deal with the same issue. Other than the measures related to the efficient operation of the network, our performance measures focus on our customers’ needs. Performance measures need to reflect the use of pri- vate capital. They also need to reflect the ability to drive management performance. Performance measures should reflect flexibility in the way the network operates. We have a number of measurement objectives at UP. We use them to describe network fluidity, network effi- ciency, and service expectations for customers. Other objectives address providing physical components for asset utilization and productivity measures, identifying opportunities for network management improvements, and identifying opportunities for capital investment. We have a number of network measures. Examples of these measures include 7-day vehicle loading rates, net- work vehicle inventories, network velocity, terminal dwell times, and terminal throughput versus terminal assets. Other measures are vehicle or shipment move- ment performance versus the plan, vehicle productivity, network assets unavailable for use, network mileage under constrained use limitations, vehicle delay time, and assets required to meet demand. Many of these mea- sures are examined for the total system and for specific components. Further, some measures are examined daily, some are examined over longer time periods, and some are meaningful only as a time series. One of the breakout sessions focuses on the limita- tions of performance measures. It is important to remem- ber that performance measures do have limitations. Performance measures should also be used judiciously. They represent only part of the measurement equation. Financial and economic components are also needed. They are usable over time only to measure the perfor- mance of a single carrier, network, terminal, or opera- tion. They cannot be used to compare carriers or modal performance. They cannot be used to compare perfor- mance between networks. Performance measures do not reflect the complexity of service and vehicle interchange between networks, either intramodal or intermodal. They do not reflect the impact on network performance of factors outside the control of the carriers or network managers, such as cus- tomer facility design or customer equipment utilization. Measurements typically have difficulty reflecting the impact of common network costs and the management decisions and policy making associated with, and derived from, the allocation of those costs. Ultimately, measurements must have a financial or economic component to be meaningful. They must relate to the cash flow production of assets, both network and vehicle. Usually a time component is needed because cap- ital costs tend to be time related. They need to reflect performance issues that drive customer satisfaction, cus- tomer costs, and customer logistics requirements. We have found that no single measurement is all- encompassing. The measurements must be used in con- cert along with management judgment of their relative importance at a particular time or under particular circumstances. Examples of measurements used in the freight indus- try include vehicle revenue or margin productivity over time and operating ratio, which is the ratio of revenue to operating cost either in total or for discrete operating entities. Other measures are return on assets, either in total or for discrete operating entities, and return on equity. Revenue or margin productivity of terminals over time, line segments or service offerings and revenue, and cost or margin per unit of production over time repre- sent still other measures. The Association of American Railroads website includes the rail public performance reports, which pro- 9PERFORMANCE MEASURES AS AN ORGANIZATIONAL MANAGEMENT TOOL

vide useful information. Performance measures in the private sector continue to focus on capital components, network performance, and manager performance. Lance A. Neumann, Cambridge Systematics, Inc., mod- erated this session, and Katherine F. Turnbull, Texas Transportation Institute, served as rapporteur. 10 U.S. AND INTERNATIONAL APPROACHES TO PERFORMANCE MEASUREMENT

11 PLENARY SESSION 2 Communicating Performance Results Effectively to Your Customers G. J. (Pete) Fielding, University of California, Irvine David Kuehn, Federal Highway Administration Shintaro Terabe, Tokyo University of Science, Japan Daniela Bremmer, Washington State Department of Transportation This plenary session discussed methods, techniques,and applications used by public and private agenciesin the United States and abroad for implementing performance measures to assess customer satisfaction. Research needs are identified in each of the correspond- ing breakout sessions on pages 55–68 and in Appendix A, page 147. COMMUNICATING PERFORMANCE RESULTS G. J. (Pete) Fielding My comments focus on the presentation of performance measure results to different audiences. I will also discuss benchmarking and potential issues associated with the use of transportation performance measures. Figure 1 illustrates transit performance concepts. It highlights the three dimensions of transit performance, service effi- ciency, service effectiveness, and cost- effectiveness. Ser- vice efficiency relates to how well transit agencies and operators are using public investments. Service effective- ness relates to how well the service is used by customers. Cost- effectiveness represents the link between the ser- vices provided and ridership levels. Numerous perfor- mance measures can be used with each of these three categories. I recommend using a small number of key performance measures. Figure 2 illustrates road maintenance performance concepts. It includes the same three dimensions, service efficiency, service effectiveness, and cost- effectiveness. Public investments are used to develop and operate the road network, which is used by customers. A variety of performance measures can be used with roadway main- tenance and operation, including maintaining the condi- tion of assets to a specified level, clearing incidents within a certain length of time, and reducing fatalities. Performance measures may be focused internally and externally. Customers for internal measures are agency staff; customers for external measures are policy makers, other stakeholders, and the public. Different methods of reporting performance results should be used with these groups. Different methods should also be used within an agency. Technical staff need more detailed information, whereas senior management personnel are interested in key trends. The challenge is to develop a small set of crit- ical performance indicators for each of the three dimen- sions of performance. Information on these indicators can be displayed using different formats depending on the audience. Two international examples contributed to the per- formance measures presented in Figure 2. First, the Organisation for Economic Co- operation and Develop- ment (OECD) established an expert group to examine performance indicators for road sectors in 1996. The report prepared by this group identified 15 performance indicators. Examples of these indicators are expenditures for maintenance and operation per vehicle kilometer, value of assets, roughness by road class, kilometer of congested roads, travel time variability, and fatalities and serious injuries per vehicle kilometer. Second, Transit New Zealand, the agency responsible for the highway network, has gradually shifted management of road maintenance to suppliers. The agency is using long- term

contracts in which the contractor is responsible for both management and physical work for a lump sum price. These contracts use performance measures to ensure compliance with effectiveness objectives, while leaving service efficiency to profit- maximizing contractors. The performance measures focus on planning and supervi- sion, including timely inspections and processing of applications for activities in the corridor, operational measures related to routing maintenance, and measures related to long- term pavement management. It is always an advantage to provide information on improving conditions and to use higher numbers for improving conditions. Typically, we associate higher numbers with better performance. Presenting the num- ber of lane miles of roadways with pavement in good condition is more effective than presenting the number 12 U.S. AND INTERNATIONAL APPROACHES TO PERFORMANCE MEASUREMENT Public Investments Finance, Labor, Materials Service Produced Client Utilization GOVERNING AGENCY GOALS FINANCING PERFORMANCE APPRAISAL AUDITS SURVEYS MEDIA ANALYSIS COST-EFFECTIVENESS SE RV IC E EF FI CI EN CY SERVICE EFFECTIVENESS Vehicle Hours/Miles Capacity Miles Passengers/Pass. Miles Operating Revenue Travel Time Delay in Peak Hours Regional Impacts FIGURE 1 Transit performance concepts. Public Road Network Asset condition/Value FY2 Standards for surface quality, roadscape, markings, and structural integrity Public Investments Client Utilization GOVERNING AGENCY GOALS FINANCING PERFORMANCE APPRAISAL AUDITS SURVEYS MEDIA ANALYSIS COST-EFFECTIVENESS SE RV IC E EF FI CI EN CY COST-EFFECTIVENESS Asset Condition/Value FY1 Finance, Labor, Materials Vehicle/Ton Miles, Fuel Taxes Casualty Accidents, Time to Clear Obstacles, Travel Delays FIGURE 2 Road maintenance performance concepts.

of lane miles that are in poor condition. In addition, pre- senting information graphically is more effective than presenting it with data tables. I think it is more effective to combine measures than to use ratios. For example, typically transit performance measures focus on vehicle hours of service and the cost to operate service. It is also important to address the impacts of inflation when using financial information over time. For example, examining the operating costs of one tran- sit provider during a 10-year period without accounting for inflation would indicate that operating costs have increased from $70 to $90 per revenue vehicle hour. When the effects of inflation are removed, however, the increase in operating cost during the 10-year period was from $90 per revenue vehicle hour to approximately $ 95. Other typical measures used in public transportation include operating expenses per vehicle revenue mile, operating expenses per passenger mile, and unlinked pas- senger trips per vehicle revenue mile. Displaying the trends in these performance measures in graphs can be effective because they represent the major dimensions of transit operations. Information on these key measures can be used internally and externally. Internally, transit agencies are highly interested in monitoring operating costs. Performance measures help identify where cost increases are occurring and why they are occurring. Pro- viding this information to policy makers and stakehold- ers is also important to sustain the ongoing funding of transit services. Often, an examination of absolute measures provides findings different from an examination of relative mea- sures. For example, the annual number of assaults on transit passengers and operators during a 10-year period is related to the amount of service provided. As a result, examining passenger trips per assault may be a more appropriate measure than total assaults. Making comparisons between transit agencies or other agencies is not recommended. Differences in oper- ating objectives, operating environments, and other fac- tors make meaningful analysis of performance difficult. Comparisons between transit agencies were attempted in the 1990s as part of a congressional mandate for trien- nial audits of transit agencies. Although peer compar- isons may assist in establishing reasonable benchmarks for performance goals, detailed comparisons between agencies is not effective. Comparing one agency over time is a much more effective way of encouraging improvement in public agencies. It is important to be honest in reporting performance. The most important use of performance measures is to improve an agency’s internal performance. Maintaining the trust of policy makers and the public by presenting accurate performance assessments is also critical. Be honest with the use of performance measures. MEASURING THE VALUE AND IMPACT OF AGENCY COMMUNICATION WITH THE PUBLIC David Kuehn At the conference 3 years ago three gaps in research related to communication were identified—assessing various communication methods, examining how per- formance measures influence behavior, and exploring the difficulties in communicating risk. This presentation focuses on examining research and transportation agency practices during the past 3 years that are related to those gaps. It is important to start by asking why we want to com- municate performance measures. Although the answer may seem self- evident to most of us, focusing on the dif- ferent reasons for communicating performance measures provides a common basis for discussion. Seven reasons for communicating performance measures were identi- fied through a review of recent research literature. The reasons identified are legislative direction, public aware- ness, support for new revenue, customer feedback, accountability, trust building, and collaboration. It is important to first define the audiences you want to communicate with concerning information on perfor- mance measures. I have used a very broad definition of customers for performance measures. Certainly, decision makers and partner agencies are key customers, as are businesses, shippers, residents, and suppliers. Travelers, which include commuters, visitors, transit riders, cyclists, and pedestrians, are also customers. Individuals obviously may be in more than one of these groups. The Florida Department of Transportation (DOT) conducts customer surveys of residents, local officials, visitors, seniors, and commercial drivers. These targeted surveys provide important feedback from these different customer groups, which have different needs and per- spectives. The City of New York has used the American customer satisfaction index survey. Results are captured by borough, ethnicity, and income. The results are used to better understand the needs and priorities of different groups. For example, results indicated that residents in the outer boroughs and those with lower incomes view transit services as important. FHWA has partnered with the Gallup organization to conduct surveys of staff at other agencies related to the environmental process and environmental stewardship. Communicating performance measures can enhance partnerships among agencies. This communication can improve cooperation, build new partnerships, and expand on existing partnerships. The Wilmington Area Planning Council (WILMAPCO), a bistate metropolitan planning organization in the Wilmington, Delaware, area, provides one example of using performance mea- sures to enhance partnering and coordination. The long- 13COMMUNICATING PERFORMANCE RESULTS EFFECTIVELY TO YOUR CUSTOMERS

range transportation planning process at WILMAPCO coordinated performance measures and data from the Delaware DOT, the Maryland State Highway Adminis- tration, and other agencies. Addressing safety perfor- mance measures in most areas requires multiple agencies and groups to work together, including nontraditional partners. The value of communication performance measures to customers can be thought of as stair steps, with bene- fits progressing from building awareness, to developing trust, to obtaining support for specific measures, pro- grams, and policies. Different performance measures may be appropriate at these different levels. Different methods of communication may also be appropriate at the various levels. The City of Baltimore uses CitiStat, a database linked to a geographic information system, to monitor street repair, snow removal, and other mainte- nance and operational activities. Presenting the results of these measures to the mayor and other policy makers regularly has been very effective in building support for transportation projects and programs. Surveys used as part of the Tracker system at Missouri DOT ask resi- dents specifically about their perception of the agency’s transportation expertise. Responses to this question pro- vide an indication of the public’s trust in the Missouri DOT. The Canadian Smart Commute Initiative provides an example of developing and using a benchmarking tool to track support for a program among the business com- munity and employees. The impact of customer communications has been examined in a few cases. Impacts represent a long- term or outcome measurement. FHWA used a quadrant analysis to assess some elements of the 2005 national survey of travelers. This analysis was used to focus on the critical weakness area, which is the quadrant of high importance, but low perceived performance. By examin- ing the differences between the expectations for and the delivery of the program, FHWA was able to refocus delivery of the program. A variety of methods and techniques can be used to communicate performance measures to different cus- tomers. Although there is no right or wrong method, some approaches are more appropriate for various audi- ences. Charts, tables, dashboards, score cards, report cards, system maps, and narrative summaries represent some of the commonly used communication methods. More detailed information is provided to technical staff, decision support information is provided to policy mak- ers, and more general information is provided to the media and the public. A number of transportation agencies use dashboards for communicating performance measures. Dashboards provide high- level, up- to- date information on key per- formance measures. Dashboards are linked to auto- mated databases, so the information is updated on an ongoing basis. Score cards or report cards represent another high- level approach. This technique, which typ- ically assigns a letter grade to different measures, tends to be updated periodically. The use of interactive maps appears to be increasing. These maps provide system information, usually in real time. Users are often able to obtain more specific information on the website, which provides feedback to the agency on what is important to users. In conclusion, I think we will continue to see an increasing use of performance measures as a marketing tool, especially to promote a strategic partnering. Using performance measures to coordinate system operations, environmental stewardship, safety, and other multia- gency topics will increase. Second, I think we will see the use of performance measures to communicate customer choice in transportation and other public service, focus- ing especially on quality- driven measures. Finally, I think communicating performance measures will be used more to support data- driven decisions and shared outcome orientations. PERFORMANCE MEASURES AND CUSTOMER SATISFACTION IN THE JAPANESE ROAD SYSTEM Shintaro Terabe To provide an international perspective on the use of per- formance measures, I will discuss the performance indi- cators used in Japan and how they relate to customer satisfaction. Performance- based management at the national level in Japan started in 2003. The policy evaluation law was approved in 2002, and legislation addressing the long- term plan on the major development of infrastructure was approved in 2003. The central government currently uses 21 performance measures focusing on seven cate- gories or policies. These categories are international competitiveness, traffic congestion and linking regions, safety, environment, asset management, use of the high- way network, and road administration. A systematic approach is used with performance mea- sures. A base value from 2002 was identified for each measure, and targets were established. Progress on meet- ing each measure is tracked. Many of the measures are similar to those used here in the United States. Different approaches have been used nationally to measure customer satisfaction with various elements of the transportation system. An Internet survey is used annually, with some 20,000 residents completing the sur- vey each year. The survey includes 15 questions, using a 5-point scale for responses. Overall, customer satisfac- tion has been improving. The overall rating was 2.6 in 2003, 2.7 in 2004, 2.9 in 2005 and 2006, and is esti- 14 U.S. AND INTERNATIONAL APPROACHES TO PERFORMANCE MEASUREMENT

mated to be 3.0 in 2007. Low satisfaction levels were recorded on tolling of the expressways, however. As you might expect, customer satisfaction levels vary by different elements of the transportation system and by geographic regions. In 2004 the overall customer satis- faction rating was 2.7. Ratings are higher in rural areas in which congestion levels are lower. Customer satisfaction related to tolling of the expressways was 1.8. Customer satisfaction related to road construction improved from 2.2 in 2003 to 2.3 in 2004, partly as a result of changes in the way construction activities are undertaken. Performance- based management is also important at the prefecture, or local government, level. A total of 47 prefectures will develop their own performance mea- sures. Of these, 44 prefectures have completed the initial development of performance measures, which focus on regional characteristics. Some 272 performance mea- sures have been developed in the 44 prefectures. More than half of these measures are unique to individual pre- fectures. Examples of unique measures in the traffic safety area include the percentage of safe routes in school zones and the elimination of dangerous roads due to passing trucks. Customer- related measures at the prefec- ture level also reflect unique local characteristics. Exam- ples of local measures include the number of users of information kiosks, the distance between roadside park- ing for cellular telephone use, and the number of sight- seeing spots accessible within 30 min from the expressway exits. The Japan Highway Public Company was privatized in 2005. It was divided into three separate companies covering the eastern, central, and western parts of the country. A customer satisfaction Internet survey has been conducted since 2004. Some 8,400 individuals responded to the survey, which included 48 questions and used a 5-point scale. The survey addressed customer satisfaction in a num- ber of categories. Elements in the safety and comfort cat- egory focused on markings, maintenance, lighting, shoulder width, visibility, snow and ice, cleanliness, veg- etation, and scenic visibility. Measures in the reliability category addressed congestion, lanes for slow traffic, number of toll booths, frequency of road closures, and clearance time for accidents. Measures in the informa- tion category focused on accuracy, highway telephone services, information available through the Internet, con- gestion forecasts, information boards, information kiosks, highway radio, information by ITS, and signage. Measures addressing roadside rest areas focused on con- gestion in parking lots, number and cleanliness of restrooms, safety, lighting, pavement, and vegetation. Survey results are used to track changes in customer satisfaction over time. The results indicate that customer satisfaction related to reliability has improved during the past 3 years. Results are also examined to identify areas in which improvements should be made to effect improvements in customer satisfaction. The customer satisfaction surveys provide insight into communication with the public. Customer satisfaction is based on the level of service, but public investment is limited. Com- munication with customers is very important. This com- munication should focus on providing information on current conditions and future plans. It should be used to obtain feedback from customers, input on possible projects, and ideas for new activities. One example of a recent method to enhance commu- nications with customers is providing real- time traffic information through the Internet and mobile devices. Information on future plans for the expressway network is also posted, including schedules for the opening of different road sections. Another example is providing ongoing press releases and newspaper articles on spe- cific projects, such as the opening of a new tunnel and the positive impact on traffic after it was opened. Another method, which has been in use since 1998, is driver hotlines. Some 31,500 calls were received in 2006. An emergency hotline for road maintenance was established in 2005. About 30,000 calls were made to this hotline in 2006. Other techniques that can be used to communicate with customers include meetings and site visits, adopt- a- highway programs, and public involvement strategies. Examples of site visits include a road day and a civil engi- neering day. Adopt- a- highway programs focus on road cleaning and vegetation management. Collaborative planning represents a public involvement technique that has been used recently. A number of issues should be considered in monitor- ing customer satisfaction levels and communicating with the public. First, Internet- based customer satisfaction sur- veys may introduce a bias because to participate individ- uals must have access to the Internet. This limitation can be addressed through the use of additional telephone or mail- based surveys. Second, the ability to change cus- tomer satisfaction levels depends on the ability to make changes and improvements in services. These services can be divided into two groups— providing the right service and providing attractive service. Both types of services need to be addressed to improve customer satisfaction. COMMUNICATING MAXIMUM THROUGHPUT: THE DOUG MACDONALD CHALLENGE Daniela Bremmer My comments focus on the difficulty of communicating transportation performance measures to the public and the experience in Washington State with a contest on techniques to effectively communicate the concept of 15COMMUNICATING PERFORMANCE RESULTS EFFECTIVELY TO YOUR CUSTOMERS

maximum throughput. The contest highlighted the importance of effective communication with the public, the media, and policy makers. Effectively communicating transportation principles, measures, and results to the public can be a difficult task for many transportation agencies and transportation professionals. We have a tendency to use technical jar- gon to explain performance measures to diverse audi- ences. It is difficult to present complex performance data in easy to understand formats. Simple questions relating to why the transportation system performs in a certain way, whether performance is improving or getting worse, what transportation agencies are doing to improve per- formance, and whether taxpayers and decision makers are getting their money’s worth are not easy to answer. Concepts of system efficiency and maximum through- put can be especially problematic to explain to the public and policy makers. Washington State DOT emphasizes system efficiency in its management of congestion on state highways and in investment decisions. The depart- ment uses maximum throughput as the basis for measur- ing system efficiency. Maximum throughput is defined as the maximum number of vehicles that can pass through an individual lane every hour, which is approximately 2,000 vehicles per lane per hour on highways and is achieved when traffic on a roadway is traveling at approximately 70% to 85% of the posted speed limit. Washington State DOT publishes a comprehensive annual congestion report and has explored different methods to communicate system efficiency issues and results to policy makers, the media, and the public. To highlight congestion and delay levels, we have used three- dimensional graphics modeled after the delay maps that have been used in Japan. We have also used other approaches to present information and to help tell our story. It is difficult to communicate to citizens the concept of managing the flow of traffic on highways by regulating vehicle volume and speed. We are asking the public to accept congestion thresholds and strategies that do not manage the system to free flow or posted speeds, but rather to a perceived standard. Further, maintaining maximum throughput may require the deployment of operational strategies, such as variable pricing and tolling, which may be controversial in some areas. Douglas MacDonald, the former Washington State DOT secretary, attempted to address this dilemma in a unique way. He announced a contest, called “The Doug MacDonald Challenge,” to find an effective communica- tion tool for conveying the concept of maximum throughput to the general public, the media, and policy makers. The contest guidelines allowed each entry to submit a 175-word description, with an additional chart or graph to supplement the written explanation. The contest was sponsored by TRB and the TRB Congestion Pricing Committee. The prize for the winning entry was $1,000, which was donated by Mr. MacDonald. The winner was announced at the 2007 TRB annual meeting. A total of 110 people took on Doug MacDonald’s challenge and submitted entries. The finalists used a number of innovative techniques to explain maximum throughput. A few entries explained the “clogging” con- cept through examples such as using grass clippings in a lawn mower. Other entries were more theoretical and demonstrated the physics and logic behind the concept of maximum throughput. Still others suggested more practical solutions for achieving maximum throughput, such as adding toll lanes adjacent to existing highways to lower the overall volume on the highway. The winning entry was submitted by Paul Hasse, a freelance writer from Sammamish, Washington. He demonstrated the concept of maximum throughput by pouring a bag of dry rice through a funnel. If too much rice, denoting vehicles, is poured into the funnel, denot- ing the highway system, at one time, the spout of the fun- nel, or the highway, becomes clogged, and very little rice makes it through the spout, representing traffic conges- tion during peak periods. By pouring the rice slowly through the funnel, the rice organizes itself in a more efficient manner and flows out of the spout at a more rapid rate of up to 33% faster. The media, including television stations, featured the winner and Secretary MacDonald pouring rice through a funnel to demonstrate the concept. The winning entry demonstrated that controlling the volume and flow of traffic through a highway system is essential to achiev- ing maximum throughput and efficiency. It demon- strated to the public how Washington State DOT may control volumes and traffic flow through ramp meter- ing, HOV–HOT lanes, variable congestion pricing and tolling, and variable speed limits. The challenge had some interesting secondary effects. Soon after the contest was announced, the Seattle Times erroneously reported that the contest was for the best idea for relieving traffic congestion, rather than explain- ing maximum throughput and efficiency. In response to this misinformation, Washington State DOT received an impromptu opinion poll from more than 250 individu- als. This unsolicited information provided the agency with valuable insight on the public’s views and knowl- edge of transportation issues in Washington State. Some of the suggested ideas included eliminating HOV lanes, teaching drivers how to merge, lowering speed limits, raising speed limits, and tolling of highways. In addition, 16 entries suggested building more roadways. The experience with the challenge provides benefits to ongoing communication methods. First, communicating difficult- to- understand concepts, such as maximum throughput, and politically sensitive subjects, such as tolling and congestion pricing, is best done in an engag- 16 U.S. AND INTERNATIONAL APPROACHES TO PERFORMANCE MEASUREMENT

ing, humorous, and disarming manner. Second, using a presentation to engage a wide variety of audiences, such as engineers and the general public, is useful in imple- menting new and complex transportation concepts. Third, the demonstration was presented at the White House, to the governor and legislators, at national trans- portation conferences, and at chamber of commerce meetings, building interest and support for transporta- tion. Finally, it helped in creating a successful partner- ship between the city of Seattle, King County, and the Puget Sound Regional Council for a multimillion dollar grant from the U.S. Department of Transportation for tolling SR-520 in King County. Howard Glassman, Florida Metropolitan Planning Organization Advisory Council, moderated this session, and Katherine F. Turnbull, Texas Transportation Insti- tute, served as rapporteur. 17COMMUNICATING PERFORMANCE RESULTS EFFECTIVELY TO YOUR CUSTOMERS

18 EVENING SESSION Perils and Pitfalls of Organizational Measures Pete K. Rahn, Missouri Department of Transportation Archie Robertson, Highways Agency, United Kingdom Rhonda Faught, New Mexico Department of Transportation This session featured an international roundtable ofsenior transportation executives who shared theirinsight and experiences and described the challenges they encountered when introducing and sustaining the use of performance measures in their respective transportation agencies. USING PERFORMANCE MEASURES IN A NEW ORGANIZATION Pete K. Rahn When I became Director of Missouri Department of Transportation (MoDOT), I was not sure how my expe- rience at New Mexico Department of Transportation (NMDOT) would relate to a new organization. I had a very positive experience with the development and use of performance measures at NMDOT. I was unsure whether the same approach would produce similar results in Missouri. I am pleased to note that state departments of trans- portation face very similar issues. When I began promot- ing the use of performance measures at NMDOT in 1995, there were no public- sector agency models avail- able to use as examples. We were really creating a new model. At that time, there was distrust, and even some fear, among employees within NMDOT that the perfor- mance measures would be used against them. We imple- mented the program on sheer will. Finding the right person to manage the development and implementation of performance measures is critical to the success of the effort. I was fortunate to have an individual who did an excellent job of shaping the program. When I became director at MoDOT in 2004, the department had been working on performance measures for several years. Support from the director appeared to be lacking in the effort, however. I had to overcome skep- ticism on the part of some employees who viewed per- formance measures as just a passing fad. Convincing midlevel managers that I was serious about performance measures took time and effort. Midlevel managers in many state departments of transportation have assumed the role of protecting lower- level staff from the whims of new secretaries or directors after changes in governors. Gaining and maintaining the support of midlevel man- agers is critical to successfully developing and using performance measures. CREATING A CULTURE OF PERFORMANCE MEASURES Archie Robertson Performance measures are just one part of the manage- ment package. You also need to establish the agency mis- sion, culture, and vision. Creating a new culture at a public agency can be challenging. There is a need to con- tinually reinforce the vision and the culture. We have an additional challenge at the Highways Agency with the large number of contract workers. Our

contractors are well in tune with our performance mea- sures. The performance measures process at the High- ways Agency has gone very well. BUILDING ON EXISTING PERFORMANCE MEASURES Rhonda Faught I was fortunate to follow Pete as Director at NMDOT. Although performance measures were well established at NMDOT, there was a need to ensure support from the new governor. We have been able to sustain and build on previous efforts. Although state departments of transportation face many similar issues, we all do things a little differently. For example, as part of an AASHTO- sponsored effort, we examined how seven states assessed on- time and on- budget measures. We found that each state was using a different approach. We can learn from each other and improve the way we deliver projects and services. ADDITIONAL DISCUSSION After the opening comments, participants and the speak- ers discussed a wide range of topics. The following top- ics were covered in the session: • Participants discussed the challenge of maintaining a commitment to performance management at state departments of transportation and other transportation agencies when there is a change in governor, mayor, or other elected officials. Suggestions for maintaining an ongoing commitment included establishing a strong agency culture focusing on performance measurement and maintaining support at the commission level. Another suggestion focused on (a) developing a strong relationship with the legislature or other policy body so that performance measures are expected to be used or (b) linking performance measures to specific legislative reporting requirements. It was also noted that support from midlevel managers, as well as top agency leaders is critical. • Another suggestion was to be proactive when there is a change in the agency director, commissioner, or com- missioners after an election. An example of agency staff taking the initiative to meet with a new commissioner to discuss the use of performance measures and their importance was provided to support this suggestion. Conducting periodic meetings with policy makers to review performance measures was noted as another strategy. It was also noted that goals, policies, and direc- tions may change with new leadership, but the perfor- mance measurement process should be robust enough and strong enough to be maintained. • The differences in the organizational structures of state departments of transportation were discussed. These differences will influence the application of per- formance measures and the ability to sustain their use overtime. It was suggested that no one approach fits all situations. Rather, the measures, benchmarks, and mile- stones need to be tailored to the characteristics of individual states. • The need to develop performance measures related to climate change, sustainability, transportation and land use, and transportation and economic development was discussed. Sharing examples of currently used measures, as well as those in the development stage, was identified as important. Examining the experience in some Euro- pean countries with these types of measures was sug- gested as a possible follow- up activity. • Participants discussed the approaches that have been used at different federal agencies during changes in administrations. Focusing performance measures on the core mission of an agency or organization was noted as an important approach to maintaining continuity. Although goals and priorities may change with a new administration, the performance measures process remains intact. • The role performance measures can play when agencies are facing reduced funding levels, as well as increased funding, was discussed by participants. It was suggested that measurement systems based on accurate data are needed in both situations. Performance mea- sures provide the basis for sound decision making. Although it was suggested that performance measures, in and of themselves, will not result in increased funding, performance measures can be used effectively to estab- lish the need for increased funding with policy makers and the public. Performance measures provide valuable information to communicate with policy makers on transportation funding needs. Keeping the message sim- ple by focusing on a few critical measures was noted as important. • Participants discussed the use of different methods to communicate with policy makers, the media, the pub- lic, and other stakeholders. Methods noted as effective included dashboards, charts and graphs, summary tables, brief narrative summaries, and other techniques. Exploring the use of other visualization techniques was suggested as a follow- up activity. • Using performance measures to forecast future trends was discussed. The approach can be used to focus the attention of decision makers on the need for changes in current approaches, new initiatives, or other pro- grams. Different future scenarios can also be presented 19PERILS AND PITFALLS OF ORGANIZATIONAL MEASURES

on the basis of different courses of action and federal funding levels. • Participants discussed recent experiences with legis- lation and voter initiatives to increase transportation fund- ing in different states. In many cases, suggested efforts have been linked to specific projects. Many of the mea- sures also include specific reporting and accountability requirements. State legislatures and other policy bodies are demanding more accountability as part of increased funding. Participants discussed the safety area as an exam- ple of how performance measures can be developed and applied to build support for increased funding. • Participants discussed the use of performance mea- sures tied to personnel evaluations and pay in public agencies. There are examples of bonuses or merits being applied in the public sector if targets are met or exceeded. Participants noted the experience at MoDOT, FTA, and the U.K. Highways Agency. Sharing information on these approaches was suggested as a follow- up activity. Examples of bonuses tied to both individual and to group performance were noted. The need to explore what is allowable in different states was noted as important. • Participants discussed the roles state departments of transportation may play on multiagency teams focus- ing on broader issues. Examples of these types of groups include corridor coalitions, economic development ini- tiatives, and environmental coalitions. The benefits of participating in these types of efforts were described, along with some of the issues that may need to be considered. Lance Neumann, Cambridge Systematics, Inc., moder- ated this session, and Katherine Turnbull, Texas Trans- portation Institute, served as rapporteur. 20 U.S. AND INTERNATIONAL APPROACHES TO PERFORMANCE MEASUREMENT

21 PLENARY SESSION 3 Data and Tools Timothy J. Lomax, Texas Transportation Institute Mark E. Hallenbeck, University of Washington John J. Collins, Traffic.com, Inc. This plenary session addressed data needs and toolsrequired to construct performance measurementsystems, as well as different types of data and the process of integrating such data into performance mea- surement systems. The practice of data sharing among public and private agencies in the United States and abroad was discussed, including the need for data at the corridor and regional scale. Research needs are identified in each of the corresponding breakout sessions on pages 69–81. DATA FOR BETTER PERFORMANCE MEASUREMENT (AND BETTER DECISIONS, TOO) Timothy J. Lomax To help set the stage for the breakout sessions, this pre- sentation will discuss the recent TRB transportation information needs assessment. I worked with Johanna Zmud from NuStats, Joe Schofer from Northwestern University, and Tom Palmerlee from TRB on the assessment. The TRB transportation needs assessment was ini- tially intended to support a policy study mandated by the Safe, Accountable, Flexible, Efficient Transportation Equity Act: A Legacy for Users. The assessment was undertaken to better understand the role of data and information in the transportation decision- making process, including problem identification, planning, operations, prioritization, and resource allocation. A sec- ond intent of the assessment was to establish an ongoing process to monitor and assess data needs. A third intent was to raise the awareness of the importance of data in transportation planning, programming, operations, and decision making. The assessment began with a survey distributed to TRB committees. Some 650 ideas were submitted from 144 TRB committees. We used these ideas and other information to create a framework to consider data needs. The framework, which is contained in TRB Cir- cular E-C109,* is illustrated in Figure 3. Objective and subjective information, as well as policies, feed into the decision- making process. We hope that over time, the information gaps will be identified, providing a feedback loop to improve transportation data. We also created an information needs framework. As illustrated in Figure 4, the subject areas of the frame- work include data items, tools, access to information, and improvement of the practice. The framework also addresses information needs subjects, which include the physical inventory, travel data, condition and perfor- mance characteristics, and externalities. These informa- tion needs exist in both a national and a regional framework. In many cases, these two levels feed back and forth. They can also be divided into physical and operational characteristics. We examined situations in which the use of data made a difference in the decision- making process. Interviews were conducted with key decision makers in the case study examples to obtain a better perspective on the use of data and to help establish a context for data and deci- * Transportation Information Assets and Impacts: An Assessment of Needs, http://onlinepubs.trb.org/onlinepubs/circulars/ec109.pdf.

sion making. A number of common themes emerged from the interviews. First, the interviews established that data are important in the decision- making process. It is also clear from the interviews, however, that decisions will be made with or without data. If accurate, credible, and understandable data are available at the time a deci- sion is being made, the data will be considered in the decision- making process. This theme speaks to the need to have accurate and up- to- date data sets. It also suggests that we should focus on providing the best available data to answer specific questions rather than trying to develop perfect data. This approach supports the need for a feed- back loop to continue the investments in transportation data. Providing accurate and timely data to policy mak- ers reinforces the need for better data for future decisions. A variety of information needs were identified in the interviews. Examples of needed information included demographic trends, infrastructure conditions, and traf- fic volumes. Information on system performance and the outcomes of previous actions was also noted as impor- tant. Key information attributes were identified; infor- mation should be timely, responsive, meaningful, simple, and concise. These attributes do not mean that the data should be less complex. Rather, they suggest that we need to funnel the data into performance measures that can be understood by different audiences. This approach focuses on data as assets. Data add value and contribute to the decision- making process. Obtaining, analyzing, and maintaining data also require financial and personnel resources, time, and commit- ment. Good decisions are the hallmark of an effective information program. There is a critical need for sus- tained national data sets for national and local use. In many cases, regions are using the national data sets as default values because they do not have local data. There is a need to obtain feedback from policy mak- ers on what data are of benefit and how the data can be improved. This feedback can then be used to improve data collection and analysis tools and techniques, along with presentation methods. We are seeing a growth in the use of partnerships for transportation data collection and analysis. These part- nerships include public–public partnerships, as well as 22 U.S. AND INTERNATIONAL APPROACHES TO PERFORMANCE MEASUREMENT Transportation Data Objective Information Data have a role here, too Problems Options Outcomes Subjective Information Decision Process DecisionsValues Opinions Biases Politics Identify Information Gaps FIGURE 3 Data needs framework. Type Data Items Tools Procedures, models, estimating, forecasting Access Data, archiving, data, and methods Practice Programs, funding, variety of actions Inventory, Travel, Quantity Condition Performance Externalities Emissions System, Users System, Users Pavement, Bridge Pavement, Bridge Miles Persons, Freight Miles Persons, Freight Information Needs Subject National Physical Operation Physical Operation Regional FIGURE 4 Information needs framework.

public–private partnerships. The notion of sharing data— both within agencies and across different agencies and private- sector groups— is becoming more important. This trend minimizes data collection efforts, maximizes the use of data, and increases the value of data. Advances in technology are enhancing data collec- tion, data analysis, and data sharing capabilities. For example, new techniques allow the tracking of trip time reliability, which is a measure of interest to the public and policy makers. Technology and partnerships are improving data acquisition efficiency and allowing the use of new performance measures. It is important to understand how data are collected and analyzed. We call this “collecting and connecting.” And it shows up in a number of situations. The results from travel surveys provide information on travel behav- ior and the movement of goods. Household travel and commodity flow surveys provide information on current and future demands. Asset history is important for understanding the condition and needs of different facilities. Real- time traffic data and weather and incident monitoring data should be tied together. Transportation policies and decisions should not be made in a vacuum. Decisions should be based on an understanding of current conditions. Combining trans- portation and land use information can provide power- ful decision support tools. My challenge to you is to examine the data sources you currently use. Discuss these sources with personnel at other agencies, private- sector groups, and policy mak- ers. Encourage other businesses and leadership groups to participate in this discussion to expand your potential sources of data and user groups. Recruit representatives from agencies, businesses, and community organizations to use your data and information. COOL THINGS YOU CAN DO WITH DATA AND WHAT THAT MEANS TO YOU Mark E. Hallenbeck My comments focus on the importance of data in the decision- making process. There is much we can learn from the private sector concerning the use of data for management and operations. Successful, well- run companies such as Wal- Mart, UPS, and FedEx have excellent data systems. These com- panies use their data systems to make informed business decisions. Without accurate data systems it is difficult for a business or an agency to know what is working, what is not working, and what can or needs to be done better. No one, including businesses and public agencies, likes to pay for data. Data collection is expensive. Fur- ther, data collection, in and of itself, is not a source of revenue. In a well- run business, data are collected for a purpose. That purpose must be important enough to warrant the expense of collecting and processing the data. The value of the data has to be more than the expense of collecting it. In the business world, most data collection is done as part of the routine business process. It is rarely collected as part of a special study by a specialized data collection division. Furthermore, good businesses are always exploring new applications that take advantage of the data currently collected. The intent of the applications driven by their data collection is to use those data to gen- erate revenue for the company or to reduce the com- pany’s cost of doing business. The benefits of the data collection, which include revenue generation or cost reduction/containment, must exceed the cost of data col- lection, or the company stops collecting those data. For example, the data system at Wal- Mart tracks inventory, allowing individual stores to maintain adequate, but not overstocked, supplies of products. This process ensures that the least amount of money is spent on maintaining inventory, while ensuring that items desired by customers are available in the store. Typically, in businesses data systems are used for many different management purposes. Thus, Wal- Mart uses these same data to manage its general purchasing decisions on what products to buy, to forecast revenue, to plan store layouts, and to manage profits. Few public transportation agencies, including state departments of transportation and public transportation agencies, are run like businesses. The attitude at many transportation agencies is that funding is not available for data collection and analysis. Most transportation agencies have limited budgets and significant, obvious, needs. Allocating limited funding for data collection is often viewed as an unnecessary luxury. I would suggest that transportation agencies often make decisions based on public and political influence. I would also suggest that transportation agencies would benefit from being managed more like private businesses. Although trans- portation agencies must respond to multiple goals and objectives, good data systems can aid in the decision- making process. There is a ballot measure this November in the Puget Sound region on funding for Sound Transit and the Regional Transportation Improvement District. There has not been a great deal of data presented as part of this ballot measure. It is difficult to respond to questions raised by this measure relating to how current funding is being spent, whether agency goals and objectives are being met, and how the transportation system could be operated more effectively, without having data systems capable of addressing these subjects. For example, congestion is common on most road- ways today. Yet, in many cases, road performance is not 23DATA AND TOOLS

measured. As a result, funding and performance are not always linked. Typically, funding priorities focus on road- way construction and roadway maintenance. Funding for operating roadways efficiently is not usually a priority. Public spending is typically aimed at maintaining public support. Public support is not tied directly to the actual performance of the system. Rather, public support is often based on the visible results from the expenditures of tax dollars. Many significant operational improvements are not readily apparent to the public, but could be measured and reported, thus gaining public visibility and support. By not measuring and reporting roadway performance, we help ensure that the public does not connect conges- tion with suboptimal operation of the transportation sys- tem. It is also hard to show or explain congestion that does not occur as being the result of an operational improvement or action. As a result, the transportation system may not be operated as efficiently as it could be, with many roadways operating below capacity. You can ask whether the transportation system would function differently if a transportation agency’s funding was a function of roadway performance or whether your salary was based on how well roadways actually oper- ated. Would transportation agencies operate the system differently if revenue was directly generated by roadway use? What if additional revenue was generated by improved roadway performance? Would more funding be spent on operating facilities we already have more efficiently if the pay of transportation agency personnel was also related to the performance of those facilities? If revenue increased if more people could use the roadway system during peak periods, would transportation agen- cies do things differently to maximize the number of peak- period travelers? If revenue, or your salary, related to performance, the first task those involved would need to perform would be to define good performance. It would also be neces- sary to report actual performance, and there would be significant incentive to determine the causes of poor per- formance and then to remove or mitigate those causes. Transportation agencies would continually be looking for ways to improve performance and would more closely assess the expected benefits of proposed improve- ments. I predict that under that scenario, roadway oper- ations would be a priority, travel choices would expand, and resources would flow to enhancements. Managed lanes, HOT lanes, and congestion pricing reflect that approach. They also provide the financial incentives that help drive its success. In addition, they pro- vide tools that help make active management of the sys- tem possible. Active operational management means understanding current conditions, predicting future demands and conditions, analyzing the impacts of alter- native management actions, and selecting and implement- ing the best actions. To adopt active traffic management, the necessary agency or jurisdictional agreements for implementing these actions must also be in place. Agencies must also determine exactly what goal they wish to actively manage to achieve the desired results. Possible goals of active operations management might include maximizing the number of vehicles served, max- imizing the number of people served, maximizing the revenue collected, minimizing delays, or maximizing the benefits gained from available funding. One might sug- gest that the current goal of our transportation system management is more closely tied to providing everyone with equal access to congestion. Data are key to making active system management work. Data are needed on system demand, or road use, and on road performance, such as speed and delay. The actual versus optimum performance of a roadway can then be computed. Systems to better detect use and per- formance are needed, as are more flexible operational controls and more capable analytical systems. Data on facility operation would include volume by vehicle type and performance data related to speed, delay, and con- gestion. The status of control and enforcement systems would be needed, as would data on external events such as weather, incidents, construction, and special events. Data collection needs to be a routine element, rather than a special activity. Data should be collected once, but should be used many times. Data should be viewed as an asset, not just a cost. In many cases, the secondary uses of data may be more valuable than the preliminary uses. For example, we may collect truck weight data to enforce weight laws, but by using those data to support pave- ment warrantees we may save hundreds of millions of dollars. Software is available for archiving, retrieving, analyzing, and reporting data. Decision support software is also available. Traditional uses of roadway data include identifying where congestion is occurring, analyzing the causes of congestion, and assessing changes in congestion based on system expansion and operational improvements. Examples of operational improvements that may be monitored include incident response programs, changing traffic signal timing plans, and dynamic message signs. The effectiveness of changing traffic control settings for different weather conditions, including rain and snow, can be monitored. Traffic data can also be used to ana- lyze changes in pricing, including increasing or decreas- ing tolls and varying tolls by vehicle type or time of day. Still other uses include assessing incident response staff utilization and response times and examining the effec- tiveness of routing plans. Operations data are important for more than just mak- ing operating decisions. Operations data can be used to analyze pavement warrantees to determine whether actual traffic volumes meet the warrantee targets. Operations data are important in analyzing the effectiveness of spe- 24 U.S. AND INTERNATIONAL APPROACHES TO PERFORMANCE MEASUREMENT

cific safety treatments and the factors influencing crashes. Operations data are also important for system planning. Using data and performance measures reflects a proactive business culture. This culture actively analyzes and reviews performance against goals, uses objective data to judge performance, rewards good performance, and changes business practices that cause poor perfor- mance. Most transportation agencies do not reflect this proactive business culture. Moving toward this culture involves setting operational goals and establishing incen- tive and disincentive systems so that the agency works toward these goals. INDUSTRY VISION FOR REAL- TIME PERFORMANCE MEASUREMENT John J. Collins This presentation focuses on the role private industry can play in collecting data on various aspects of the transportation system. As has been noted, the use of per- formance measures based on reliable and accurate data is a key attribute of successful businesses. Performance measures are becoming more widely used in the public sector, including transportation agencies at all levels. It is important to collect data once and use them for multiple purposes. Avoiding “stovepiping” of data col- lection, which can sometimes occur in agencies and busi- nesses, is also important. Given the expense of gathering data, the private industry vision is to collect the key data, making sure there are multiple customers for those data. We follow this approach in our business. We could not run a successful business if we sold our data only to one television network. Our customers include multiple tele- vision networks and radio stations. We also use the data in other ways. Multiple users of data collected by public agencies maximize resources and build support and champions. On the business side, it is important to have enough cus- tomers to keep the cost to individual users reasonable. Collecting data once and using it multiple times is impor- tant in the public and the private sectors. The private sector can assist public agency data col- lection efforts in a number of ways. First, private busi- nesses can add data to the data public agencies currently collect. For example, under a contract with FHWA, we have added sensors to fill gaps in 27 areas across the country. We also bring agency data into the same data- base to avoid stovepipes. Further, the private sector can provide additional applications and tools to enhance the use of data in the decision- making process. Private industry uses numerous methods to collect transportation- related data. Examples include fixed sen- sors, Global Positioning System (GPS) data from truck- ing firms, and cell phone data from cellular telephone providers. We collect data from these and other sources, process the data, and provide the data to state depart- ments of transportation and other customers in usable formats. We can compare current and historical data, provide short- term and long- term forecasts, and conduct other analyses. It is important to turn data into intelli- gence that can be used to make better individual and agency decisions. NAVTEQ owns and operates traffic sensors on more than 2,500 lane miles of roadways. These sensors pro- vide volume, speed, length- based vehicle classification, and lane- occupancy data. GPS and cell phone coverage are used in areas to augment sensor data, providing additional spatial and temporal verification and accuracy. Real- time information can be provided on websites, historical data can be analyzed, and future trends can be examined. Data can also be used for plan- ning and operations. For example, the Illinois Tollway uses our data summarized in Excel spreadsheets to determine lane closures for construction and mainte- nance. More detailed analysis can also be conducted, including examining traffic patterns by time of day and day of the week. Friday afternoons used to be the busiest traffic period on the toll road. With more peo- ple working alternative hours or taking 3-day week- ends, the heaviest traffic now tends to occur on Thursday afternoons. The tollway also used our data to adjust the hours during which contractors are per- mitted to close lanes for road repairs. Under our contract with FHWA, we are required to provide 95% availability. Every morning we examine all the sensors across the country to identify problems. Missing data means unhappy customers. From a busi- ness perspective, we cannot afford to have sensors mal- function. We are in the data business; not the sensor business. The heart of what we do is data processing and data storage. We follow the previously described approach of collecting data once and using it for multi- ple purposes. We develop specialized delivery systems for different customers. For the Winter Olympics, Utah DOT developed an extensive real- time information system with more than 2,000 sensors. Although it is a very robust system, it was not designed to serve all Utah DOT’s current data needs. We were able to assist the agency by working with the Bureau of Transportation Statistics and Performance Measurement System (PeMS) applications to add appli- cations that Utah DOT wanted to our data warehousing system. The methodology is relatively simple. NAVTEQ is collecting data under the federal program. Agency data and other data collected by NAVTEQ enhance the data- base. PeMS provides the additional traffic management tools and transportation applications. There are multiple uses for the data. The uses include monitoring the status 25DATA AND TOOLS

of sensors and traffic operations, analyzing performance measures, and supporting policy decisions. The program with FHWA was modeled after the FAA map that shows congestion levels in the nation’s airspace. It provided a powerful tool to show policy makers the need for improvements. FHWA did not have anything similar to highlight traffic congestion across the country. Private industry can be an important partner in col- lecting, archiving, and analyzing transportation data. We can add value by providing new data sensors and access to intelligent transportation system data. We can also provide new tools and applications. Remember, collect data once, but use it for multiple purposes. Johanna Zmud, NuStats, served as facilitator for this session, and Joe Zietsman, Texas Transportation Insti- tute, served as rapporteur. 26 U.S. AND INTERNATIONAL APPROACHES TO PERFORMANCE MEASUREMENT

27 PLENARY SESSION 4 Hot Topics Kotaro Nagasawa, Mitsubishi Research Institute, Inc. Steven Gayle, Binghamton Metropolitan Planning Transportation Study Mark Larson, Minnesota Department of Transportation Kimberly Spence, Virginia Department of Transportation This plenary session presented current and upcominginnovative practices in performance measurementapplied both in the United States and abroad, including Japan’s shakai- jikken evaluative process used in project decision making; goal setting at the metropolitan planning organization (MPO) level; a tradeoff analysis method used to measure the performance of a multi- modal system; and target setting and techniques used in planning and evaluating sustainable transportation sys- tems, road pricing, and safety and freight planning. Research needs are identified in each of the correspond- ing breakout sessions on pages 82–97 and in Appendix A, page 148. SOCIAL POLICY EXPERIMENTS IN ROAD MANAGEMENT: INTRODUCING JAPAN’S APPROACH TO ACHIEVING BETTER PERFORMANCE Kotaro Nagasawa My comments focus on an interesting road management strategy in Japan called shakai- jikken. The term shakai- jikken means a short- term trial with stakeholders’ par- ticipation. The concept is close to a social policy experiment, but it is more practical and less theoretical. Before- and- after comparisons are conducted as part of the strategy. Although the road network in Japan has improved over the past 30 years, problems related to traffic man- agement are becoming more complex and more serious. For example, traffic congestion is a problem in metro- politan areas, particularly during the peak commuting periods. Road safety in all parts of the country is also a concern. In most areas, constructing more highways is not possible. Consensus building with stakeholders is a key part of the process of addressing these concerns. In Japan, we typically rely on computer simulation when we try to introduce a new policy in traffic man- agement. Using a report based on that computer simula- tion, we start discussions with stakeholders to build a consensus and to move to the next stage of implementa- tion. Performance measurement is a major element of the implementation phase. However, questions may be posed regarding whether the proper policy has been selected for implementation. Computer simulation may not be fully reliable, particu- larly in complex cases such as demand management. It is against this the background that the concept of shakai- jikken emerged in Japan. Stakeholders are unable to fore- cast what will happen when a new policy is introduced. The shakai- jikken concept, which focuses on testing and evaluating a new approach or concept, was first proposed by the Road Committee in 2001. The first shakai- jikken was introduced in 2003. The first step in the process is a conference with stakeholders and academics organized by the local government. The conference participants develop the plan and run the shakai- jikken. The central government subsidizes the shakai- jikken. I will highlight a shakai- jikken related to charging or pricing on expressways in Japan. The cost to use an expressway is normally approximately 30 cents per mile, yet there is no charge to use local roadways. As a result,

local roads are congested, especially during peak com- mute times. The expressways have capacity at these times. The idea of reducing the expressway charge to encourage traffic to move from the local roads to the expressway and to better use the infrastructure repre- sents one shakai- jikken. The shakai- jikken of flexible charges started in 2003. There were 22 trials in 2003, 41 trials in 2004, and 12 trials in 2005. These 75 trials are located in different areas and test different ways of introducing flexible charges for use of the expressways. Issues examined in these trials include the most effective time periods to reduce charges, the level of the reduction, and the type of vehicles to target for the reduced fees. Different methods for stakeholder participation in the shakai- jikken are also being explored to help identify the most effective techniques. Examples of the time periods tested for the reduced fees on expressways included 7:00 a.m. to 9:00 a.m. and 5:00 p.m. to 7:00 p.m., 6:30 a.m. to 9:30 a.m., and all day. In the trials providing reduced charges during the morning and afternoon peak periods, traffic on the local roads was reduced by 3% to 6%. The length of traffic congestion or queuing also decreased. In addition, the most effective discount rate was examined. Although drivers would welcome no toll at all, this change is not realistic to consider. The elasticity of the charges was calculated on the basis of the result of the surveys conducted with some of the shakai- jikken trials. When the discount rate increased from 30% to 50%, the elasticity went up slightly. The elasticity went down, however, when the discount rate went from 50% to 70% based on one sample. I do not think we have enough data to confirm the most desirable discount rate is 50%, however. Information was also obtained on drivers’ reactions to different discount rates. Approximately 30% to 60% of the drivers reported satisfaction with a discount rate of 30%. Some 80% to 90% of drivers were satisfied with a discount rate of 50%. These results suggest that there is no reason to set the discount rate higher than 50%. The trials also highlighted the importance of public relations and public information in successful projects. The first stage of public information on one project included press releases and a website. At the end of the first stage, only 10% to 30% of local drivers knew that there was a plan to reduce road charges in the area. To increase public awareness, the organizers used a large banner, distributed fliers, and placed a newspaper adver- tisement. Four weeks after these approaches were imple- mented, the number of local drivers who knew about shakai- jikken increased to 90%. Experiments with con- gestion charging have also been conducted in Sweden and other countries. In September 2002, the Swedish government and the Stockholm City Council proposed to introduce congestion charges in the Stockholm area. The congestion pricing system was introduced during the first half of 2006. The infrastructure included an IBM electronic toll collection system. The results during that trial period indicated that ridership on public trans- portation increased by 6%, and road traffic decreased by 25%. The pricing system was supported by approxi- mately 53% of Stockholm residents, but residents in sub- urban areas did not favor the program. Shakai- jikken should be considered in the context of pol- icy management. There are three key points to the use of this approach. First, shakai- jikken is a strong tool to ensure more realistic plans that can be implemented. Second, shakai- jikken provides a method to introduce new types of policies. In Japan, it was difficult to predict what would happen when expressway charges were changed because there was no prior experience. The shakai- jikken approach provides a way to test these approaches. Third, shakai- jikken provides the opportunity for stakeholders to com- municate with each other. Cooperative processes enhance the chances of successful implementation of new policies. Shakai- jikken has been discussed at some academic conferences in Japan. Some elements need to be exam- ined more closely. First, the costs and benefits of each shakai- jikken should be evaluated. Second, there may be a bias in the result of shakai- jikken, because stakehold- ers are highly motivated while a trial is in operation. Third, more thought needs to be given to the consensus- building process included in a shakai- jikken. Although the shakai- jikken research has a strong practical side, theoretical research is going to become more and more important. USING TRANSPORTATION SYSTEM PERFORMANCE MEASURES IN PLANNING: METROPOLITAN PLANNING ORGANIZATION PRACTICE Steven Gayle My comments focus on the use of transportation system performance measures in the planning process conducted by metropolitan planning organizations (MPOs). I high- light examples from MPOs throughout the country. Sys- tem performance is important to MPOs for several reasons. First, MPOs are stewards of the multimodal metropolitan transportation system. Second, MPO board members, who are mostly local officials, under- stand the importance of measuring performance. Third, the public, who are MPO customers, know when the 28 U.S. AND INTERNATIONAL APPROACHES TO PERFORMANCE MEASUREMENT

transportation system is not working because of conges- tion, mistimed signals, and other problems. MPOs have traditionally been involved in measuring and monitoring the performance of physical assets, including pavement deterioration and deficient bridges. The modeling and forecasting process used by many MPOs has historically been based on recurring conges- tion. We are just beginning to examine how measuring the operational performance of the system can benefit MPO planning tasks. System performance measures have many uses and applications. They can be used to identify the attributes of the transportation system that are most important to a metropolitan region. Performance measures provide information on current system conditions and perfor- mance. They provide a metric for communicating with decision makers and the public about past, current, and expected future transportation system conditions. Sys- tem performance measures also serve as criteria for investment decisions made in the transportation plan- ning process, and they can be used to evaluate the suc- cess of projects and programs. The Chicago Metropolitan Agency for Planning Regional Transportation Plan (RTP) provides one exam- ple of the use of performance objectives. One objective in the RTP is to develop a transportation system that (a) facilitates management and operation communication abilities and real- time decision making; (b) improves the transportation system information available to travelers and system operators; (c) reduces nonrecurring delay by reducing the number and duration of highway incidents and improves transit system on- time performance; and (d) reduces recurring delay through access and speed management, value pricing, improved design, and incen- tives encouraging alternate modes of travel. Another objective in the RTP is to promote transporta- tion proposals that (a) reduce highway congestion; (b) improve system reliability; (c) provide improved trans- portation management capabilities; (d) maximize perfor- mance benefits through intensive management, (e) increase person throughput in congested corridors by increasing vehicle occupancy, providing transit options, and encour- aging transit use; (f) increase the share of trips made by walking, bicycling, and transit; and (g) improve coordina- tion and connectivity between and among different modes. The Capital District Transportation Committee in Albany, New York, has a comprehensive measurement program that includes a series of principles. The first prin- ciple is that management of demand is preferable to accommodation of single- occupant vehicle demand growth. A second principle is that cost- effective opera- tional actions are preferable to physical highway capacity expansion. A third principle is that incident management is essential to effective congestion management. Still another principle is that any major highway expansion considered by Capital District Transportation Committee will include a management approach. These principles are linked to performance measures, which focus on access, accessibility, congestion, and flex- ibility. The planning time index is a measure of reliabil- ity and predictability of travel time, reflecting the importance of nonrecurring delay on expressway seg- ments. Expressway data come from the New York State Department of Transportation Management Informa- tion System for Transportation. This index is used with decision makers and other stakeholder groups. The North Central Texas Council of Governments (NCTCOG) is the MPO in the Dallas–Fort Worth Metroplex. NCTCOG prepares and publishes an annual Transportation State of the Region, which is used by the policy board in decision making. Performance measures used by NCTCOG include the congestion index, fatal and injury crashes, air emissions, and transit ridership. Metro is the MPO in the Portland, Oregon, area. Metro has freeway and roadway performance measures that use intelligent transportation system data. Examples of measures include the severity and duration of freeway congestion and the variability in speed. Travel time con- tours, which measure how accessibility is affected by congestion, are being developed. Metro is also establish- ing baseline performance measures. The Metropolitan Transportation Commission (MTC) is the MPO in the San Francisco Bay Area. The MTC also operates elements of the transportation sys- tem. The MTC’s Transportation 2030 report states that “Traffic management and operations strategies, such as incident detection and real- time information, and increased use of new technologies, are key to reducing the impact of traffic congestion on people’s lives and businesses.” The MTC’s state- of- the- system report includes congestion measures that focus on peak period vehicle hours of delay, average commute time on speci- fied routes, the buffer index, local road congestion, and transit on- time performance. As other speakers have noted, there are challenges and limitations with the use of performance measures by MPOs. One challenge is selecting regionally important performance measures. Other challenges include instru- menting the transportation system, especially beyond freeways, to provide needed data; obtaining data on all transportation modes; and archiving and analyzing data. It is also important to consider the link between opera- tion and management performance measures and model- ing. In air quality nonattainment areas, travel models must be accepted in the conformity process, but many lack an accepted method for modeling the benefits of 29HOT TOPICS

operational strategies. A final challenge is the trade- off of credibility versus accuracy of real- time performance data. STRATEGIES TO ACHIEVE TARGETS: EXPERIENCE IN SAFETY Mark Larson My comments focus on approaches to setting aggressive performance targets and the strategies needed to achieve them. I will highlight the safety and the Toward Zero Deaths (TZD) program in Minnesota as a case study. I will describe elements of the program and the results experienced to date. I will also highlight possible impli- cations of using performance targets in other policy areas, including economics and freight. Performance targets provide a powerful tool for driv- ing improvement. Performance targets provide a basis for establishing a common vision within an agency, across a state, or among multiple organizations. They can provide organizational motivation and mobilization of people, strategies, and organizations. Performance targets establish a basis for common action with part- ners. Measures and targets also provide a basis for data- driven analyses of factors that influence performance. Performance targets and target levels can be based on a number of different factors. Historical trends can be used to establish baselines and predict future trends. Eco- nomic measures, such as benefit–cost ratios and life- cycle costs, may be used to establish performance targets. Fis- cal measures addressing available resources and con- straints provide input for establishing performance targets. Other sources for identifying performance tar- gets include engineering factors and customer input or feedback. Performance targets should reflect the values and the vision of an agency, organization, business, or program. Legal or regulatory requirements may also be used for setting performance targets. Transportation agencies, including state departments of transportation and MPOs, influence, but do not fully control, outcomes related to safety, freight, the economy, and sustainability. The public expects leadership from us in these areas, however. The lack of results on key public issues may undermine respect for governmental agen- cies, which may affect funding. Other countries and international bodies have measures and targets for safety, the economy, and the environment. For example, the United Nations Millennium Development Program has set aggressive targets for reducing poverty. Transporta- tion is an important element supporting this effort because it provides access to jobs and health care. The transportation system is expected to contribute to broader societal goals. Transportation organizations typically need to part- ner with other agencies, organizations, and the private sector to achieve the desired results for important soci- etal measures. Transportation agencies may take the lead or may be a member of a coalition. The TZD initiative in Minnesota provides an example of a coalition developed to reduce roadway fatalities. It was started in 2001 as a partnership between the Min- nesota Department of Transportation (Mn/DOT), the Minnesota Department of Public Safety, the Minnesota Department of Health, counties, FHWA, the University of Minnesota’s Center for Transportation Studies, and 25 community coalitions. Other Mn/DOT initiatives support the effort. The Mn/DOT 2003 statewide plan sets targets for reducing fatalities. District plans and reg- ular State Transportation Improvement Program “ check- ins” support and monitor progress toward meeting these targets. The Strategic Highway Safety Plan includes cus- tomized analyses and strategies by district. It was com- pleted this year, building on the Comprehensive Highway Safety Plan (CHSP) established in 2005. Min- nesota was among the first states to create such a plan. In 2003, Mn/DOT, under the leadership of then Com- missioner Elwyn Tinklenberg, set its first target in the statewide transportation plan for the reduction of road- way fatalities. The absolute number of fatalities was selected as the measure, rather than the fatality rate, and targets were set for 20 years into the future. A straight- line trend projection showed a steady increase in fatali- ties from the base of 633. After extensive debate, two levels of targets were established for reduction in fatali- ties over 20 years— a moderate target of 600 and an aggressive target of 550. Later, a far more aggressive TZD target was set at 500 for 2008, on the basis of improving at a rate parallel to the FHWA target. What strategies would be used to achieve such chal- lenging targets? The CHSP started with data analysis of factors driving fatalities over the base period of 2001 to 2005. Factors influencing fatalities, such as driver behav- ior and infrastructure elements, were ranked. The top two fatality factors were related to driver behavior. A total of 52% of the fatalities involved unbelted vehicle occupants, and 36% were alcohol related. Further, 28% were speed related, and 24% involved drivers younger than 21 years of age. A total of 33% of the fatalities occurred at intersections, 32% were single vehicle run- off- the- road crashes, and 20% were head- on and sideswipe crashes. This analysis was used by the CHSP multiagency team to develop strategies focusing on the top five critical emphasis areas identified from the data. The first empha- sis area is to increase seatbelt use and reduce impaired driving. The second strategy targets improvements to intersection design and operations. The third strategy focuses on lane departure issues to reduce head- on and 30 U.S. AND INTERNATIONAL APPROACHES TO PERFORMANCE MEASUREMENT

across- median crashes, to keep vehicles on the roadway, and to minimize the consequences of vehicles leaving the roadway. The fourth strategy focuses on young drivers and curbing aggressive driving. The fifth strategy addresses increasing driver safety awareness and improv- ing information and decision support systems. In all, there are 15 critical strategies. Under enforce- ment, the strategies focus on providing adequate law enforcement resources, passing and implementing a pri- mary seatbelt law, implementing automated enforce- ment, implementing a stronger graduated driver licensing system, supporting the enforcement of traffic safety laws, and targeting enforcement. The strategies under engineering address cost- effective lane departure improvements, cost- effective intersection improvements, roadway maintenance, and road safety audits. The edu- cation strategies include a communication and market- ing taskforce, a high- level traffic safety panel and a legislative action committee, and enhanced driver educa- tion. The emergency medical services strategy is develop- ment of a statewide trauma system. In the area of enforcement, a speed management strat- egy and program was created. The speed limit on 905 mi of two- lane, two- way roads with excessive average motorist speeds was raised from 55 to 60 mi per hour (mph) in 2005. At the same time, enforcement was increased on these segments and others where fatalities were increasing. A public education campaign was also undertaken. Mn/DOT provided supplementary funding to the state patrol for the increased enforcement. The results of these efforts, evaluated by the University of Min- nesota Center for Transportation Studies, indicated a sig- nificant decline in vehicles traveling more than 70 mph. To address lane- departure fatalities, Mn/DOT is installing cable median barriers on at- risk segments of freeways. Currently, 36 mi of cable median barriers have been installed in two districts. An additional 53 miles of cable median barriers have been planned and funded, and 155 more mi have been planned but are not yet funded. The installation cost for cable median barriers is approximately $100,000 per mi. A preliminary evalua- tion indicates that at least two lives have been saved dur- ing the first year since the cable median barriers were installed. A new local government partnership strategy has been initiated to fund safety projects in individual counties with $4 million from a central safety fund. A total of 63 applications for project funding were received from 46 of 87 Minnesota counties; 40 projects in 34 counties were funded. Projects address lane departure strategies, inter- section lighting, enhanced signing, guardrail upgrades, geometric improvements, and road safety audits. It is important to monitor progress in achieving per- formance targets. Ideally, reports should cover the trend in the primary outcome measure compared with the tar- gets and the outcome or output results broken down by causal factor or by strategy areas. Mn/DOT’s Office of Traffic, Safety, and Operations reports annually on fatal- ity trends to the executive staff, districts, and other stake- holders through the Safety Performance Report. The Freight and Commercial Vehicle Operations Office also reports annually on fatality trends versus targets. Fatalities on roadways in the state have declined every year since 2003. There were 494 fatalities in 2006, which was the lowest number since 1945 and below the TZD’s targeted number of 500 fatalities. A new TZD target of not more than 400 fatalities for 2010 has been set. Nonengineering solutions have played a major role in reducing fatalities, including the TZD partnership and the passage of .08 alcohol legislation. From a bench- marking perspective, in 2005, the Midwest states had lower fatality rates than the national average, and Min- nesota had the fourth lowest fatality rate in the country. Additional safety strategies will be implemented in the future. A primary seat belt law has not yet been passed by the state legislature. This law is estimated to raise seatbelt use from 83% to 93% and to prevent some 40 deaths a year. Implementation of the Statewide Trauma System is estimated to reduce fatalities by 9%. Reducing truck- related fatalities in the state is also important. Mn/DOT’s Freight and Commercial Vehicle Operations Office is responsible for heavy truck permit- ting and safety. The 2005 Heavy Vehicle Safety Plan includes a target for reducing annual truck- related fatali- ties from 81 in 2004 to 70 or fewer by 2008, building on reductions achieved the previous 5 years. The office is part of the Strategic Highway Safety Plan partnership. Strategies to reduce truck- related fatalities focus on law enforcement and inspection, cost- effective road and road- side improvements, stronger commercial drivers license requirements, and the use of four- cable median barriers. I would like to briefly discuss insights from other pol- icy areas, including economics and freight. The Min- nesota State Aviation Plan, required by FAA and prepared by Mn/DOT’s Aeronautics Office, provides an example of targets related to economics that are beyond the con- trol of the department. One of the targets is that 90% of the state’s population should be within a 60-min drive of scheduled airline service. Currently, 86% of the popula- tion has airport access within 60 min. Mn/DOT’s Aero- nautics Office is working aggressively with the Minnesota city of Marshall, a major employer (Schwan’s Foods), and Northwest Airlines to try to fill a service gap in the southwest region. Freight is another policy area where performance indicators for outcomes beyond agency control are being used. Freight tonnage in Minnesota is expected to increase by 60% overall by 2020. Freight tonnage car- ried by rail is forecast to increase by 36%, and tonnage carried by truck is estimated to increase by 80% by 31HOT TOPICS

2020. The impact of these trends on pavements and bridges, as well as the impact on congestion, has been discussed by senior staff. The discussion evolved into possible influences and roles the department could play to affect rail capacity and the railroad share of freight. The state currently has only a small grant program to help short- line railroads. If these indicators become per- formance measures at Mn/DOT, we would need to estab- lish new partnerships and strategies. In conclusion, I will suggest several elements of suc- cessful practice. First, set forth a vision and high- profile targets to motivate people. Second, analyze data to break down key factors driving results and identify strategies and champions. Third, develop partnerships— one agency does not always have all of the necessary resources to meet the targets. Steps in developing successful part- nerships include opening discussion, developing a plan, developing measures and common targets, sharing tech- nical resources, and sharing program resources. MULTIMODAL TRADE-OFF ANALYSIS FOR PLANNING AND PROGRAMMING Kimberly Spence I will discuss some of the key elements in the resource paper prepared for this conference on multimodal trade- off analysis for planning and programming. I would like to acknowledge the coauthor of the paper, Mary Lynn Tischer, who was unable to attend the conference. My comments focus on defining multimodal trade-off analy- ses, describing some of the barriers to multimodal trade- off analyses, and highlighting the state- of- the- art and the state- of- the- practice with multimodal trade-off analyses. I will also present examples from Virginia and describe our experience with the use of multimodal trade-off analyses. I started my career as a consultant. I realized at the time that clients could get projects completed three ways— good, fast, and cheap— but they could only get two of the three. That is, clients could get a project cheap and fast, but they would probably not have a good prod- uct. They could get a project done well and fast, but it would probably not be cheap. These examples represent trade-offs. You have to identify what is important to you and whether you are willing to sacrifice performance in one category for performance in another category. Multimodal trade-offs can be defined as the process of evaluating potential solutions by considering the trade-offs of investing in one mode or program over another to determine the best overall investment. Several potential barriers to multimodal trade-off analysis can be identified. First, there is limited flexibility in many federal and state funding programs. Second, transporta- tion planning tends to be organized around individual modes. That means planning and implementing multi- modal transportation projects can be difficult because of the complex and cumbersome process of coordinating the efforts of multiple agencies and departments. Fre- quently, multimodal transportation plans are simply the aggregate of individual modal plans rather than an inte- grated analysis of multimodal transportation. Other bar- riers include the lack of mode- neutral performance measures and the lack of data and analytical tools to facilitate comparisons across modes. Performance data on different modes are available at various levels of detail. Finally, politics may be a barrier to multimodal trade-off analysis, because politicians may champion a specific project regardless of the analysis results. A number of different approaches are being used by state transportation agencies to conduct multimodal trade-off analyses. Benefit–cost analyses are one of the most commonly used methods. This technique converts project benefits and costs to a single ratio. This approach converts disparate impacts to a common metric, so it lev- els the playing field. In practice, benefit–cost analyses can be data intensive and often require value judgments in assigning monetary values to qualitative measures. This approach can also disguise the magnitude of costs and benefits. For example, a high cost–high benefit proj- ect and a low cost–low benefit project may have similar ratios. There are different types of benefit–cost models. In Virginia, we have a benefit–cost model that is used to assess projects to be considered for rail enhancement funding. Cost- effectiveness models represent a second approach. These models reduce complex impacts to a single monetary value. Objectives or outcomes are iden- tified, and the cost to achieve each is compared. The models measure how closely a project corresponds to a goal in relation to its cost. The approach facilitates com- parison of alternatives rather than identifying a single best solution. The Hampton Roads Planning District in Virginia uses a cost- effectiveness model to program con- gestion mitigation/air quality projects. The cost per ton of emissions reduced is identified for each project and compared. Least- cost planning represents another approach. Least- cost planning is similar to benefit–cost analyses and cost- effectiveness models in that it converts project impacts to a single monetary value. It measures the degree to which a project meets a predefined goal. The difference from the other approaches is that least- cost planning identifies the lowest cost project that meets the performance goal. This approach enables comparisons of different types of projects. Least- cost planning is used in Washington State. The Puget Sound Regional Council 32 U.S. AND INTERNATIONAL APPROACHES TO PERFORMANCE MEASUREMENT

has developed a series of performance measures to prior- itize highway, high occupancy vehicle (HOV), and tran- sit projects in the congestion mitigation plan. Mode- neutral approaches represent still another tech- nique for conducting multimodal trade-off analyses. This approach facilitates the comparison of competing modes and permits an unbiased assessment of modal alterna- tives. Person miles of travel provide an example of a mode- neutral measure, compared with vehicle miles of travel, which reflects motor vehicle and roadway use. It is difficult to find measures that are not dependent on a particular mode or program category, however. Further, not everything can be measured the same way. For exam- ple, auto accessibility may be measured by auto owner- ship, whereas transit accessibility might be measured by the distance to a transit stop. Also, different geographic scales are frequently associated with different modes. Finally, this approach may limit the objectives that are addressed. A final method that may be considered is multicrite- ria/goals achievement analyses. This approach measures the degree to which a given improvement meets broader goals. A scoring system is used to evaluate alternatives over a common set of evaluation objectives. This tech- nique may incorporate weights to reflect policy objec- tives. The use of the goals–achievement matrix can be more transparent than other techniques because the sim- plified scoring scheme may be easier for policy makers and the public to understand. Many state departments of transportation tie long- range goals to performance measures. Most states also monitor system performance. Some states use perfor- mance measures to identify projects for incorporation into the long- range plan. Further, some states use perfor- mance measures to identify projects for programming. In most cases, performance measures are used to compare projects within a specific funding program or category. States also use different approaches for conducting mul- timodal trade-off analyses. Oregon uses a benefit–cost analysis to assess trade-offs among 10 different facility types, including bus, rail, port, and air. The approach used in some states assigns additional points to projects that incorporate multiple modes. The Virginia Department of Transportation uses performance- based planning. Virginia’s long- range transportation plan, VTrans2025, addresses highways, rail, transit, ports, and airports. State legislation prohib- ited the plan from being just an aggregation of individual modal plans. It includes a long- range vision and goals to guide transportation decision making in the state. Per- formance measures are identified for each goal. The original intent of the plan was to identify major statewide multimodal corridors and to prioritize projects within the corridors. A goals- achievement matrix was used to evaluate projects according to the performance measures. Weights were assigned to reflect the impor- tance of policy priorities. For example, the goal related to the efficient movement of people and goods was given more weight than the quality- of- life goal. Ultimately, the prioritization system was only applied to highway con- struction projects. More than 1,000 highway projects were evaluated using the goals- achievement matrix. Each performance measure had a specific scale associated with it, and points were given to projects that benefited mul- tiple modes. Projects were sorted into tiers. Based on their scores, Tier 1 projects were identified as immediate needs and were suggested for qualitative review. The qualitative information was provided to decision makers for use in making programming decisions. Additional elements were also included in the evaluation. Examples of these elements were HOV usage, truck counts, and bicycle and pedestrian access. The review of the practices in different states identi- fied some common trends. First, almost all states include multiple measures in the long- range transportation plan- ning process. Second, most states use criteria for select- ing projects for a specific program or mode. Most states allocate funds within specific programs, however, and prioritize projects within these programs. Many MPOs flex funds between programs, and a few states prioritize across modes. This review highlights that multimodal trade- off analyses can be conducted using complex methodologies or simplified nonquantitative approaches. Some states are using these approaches. Decision making for project selection is becoming more closely linked to the planning process, regardless of the exact analysis technique used. Consideration of multiple modes is being incorporated into the planning process. The planning process is also being more closely linked to the overarching goals con- tained in the long- range plan. And, as we all know, polit- ical considerations will continue to play an important role in programming decisions. Randall Halvorson, Cambridge Systematics, Inc., mod- erated this session, and Josias Zietsman, Texas Trans- portation Institute, served as rapporteur. 33HOT TOPICS

34 PLENARY SESSION 5 Performance- Based Contracting and Measuring Project Delivery Paresh Tailor, Highways Agency, United Kingdom Amado Rubio Athie, Secretariat of Communication and Transport, Mexico Stephen C. Beatty, KPMG LLP Sidney Scott III, Trauner Consulting Services, Inc. This plenary session identified and discussed tech-niques used in the contracting process to convertexisting project standards into a performance- based framework for improving road quality and operating con- ditions. With an emphasis on public–private partnerships used both in the United States and abroad, the following methods were detailed: design, build, finance, operate (DBFO); concession agreements; provision of service; and asset utilization and bonus incentives to promote better highway planning and traffic operations. Research needs from the corresponding breakout sessions are identified on pages 98–108 and in Appendix A, pages 148–149. PERFORMANCE- BASED CONTRACTING IN THE UNITED KINGDOM Paresh Tailor My presentation focuses on performance- based contract- ing in the United Kingdom, including the use of the DBFO concept. DBFO represents the United Kingdom’s form of private finance initiative (PFI) contracts for roadways. We have a long history of using PFI contracts in the United Kingdom. Early PFI projects included the Queen Elizabeth II Bridge, the Second Severn Crossing, and the Birmingham North Relief Road. DBFO is part of the government’s public–private part- nership (PPP) initiative. This initiative focuses on having the private sector take over traditionally public- sector roles to provide better value for the money. As the name indicates, this approach includes having the private sector design, build, finance, and operate a roadway, bridge, or other transportation facility. The DBFO concept was pre- sented in a 1993 green paper titled “Paying for Better Motorways.” The 1993 green paper identified several objectives for DBFO. These objectives included to ensure that the road is designed, maintained, and operated safely; to transfer the appropriate level of risk to the private sector; to pro- mote innovation, both technical and financial; to foster the development of a private- sector road- operating industry in the United Kingdom; and to minimize finan- cial contributions from the public sector. These objec- tives are still relevant to DBFO projects today. There have been two phases of DBFO projects awarded. The first phase, split into two groups, consisted of eight projects, all awarded in 1996. The projects var- ied in size and cost from approximately $19 million for small new works to $428 million for significant new motorways. The projects were financed using a payment based on shadow tolls. The initial DBFO contracts were seen as a precursor to real tolls; therefore, shadow tolls were selected. The government at the time anticipated a “pay- as- you- go” or pricing system at some point in the future. This transition has not occurred. In 1997, there was a change of government in the United Kingdom. The new government initiated a review of the roads program, including the DBFO pro- jects, and all but one of the anticipated DBFO projects were stopped. In 2000, after the review, a revised pro- gram of 80 possible future projects was announced. Since 2000, this program has increased to include up to 113 possible projects. The projects include a mix of

motorway widening, trunk road widening, junction improvements, and bypasses. Approximately 47 of these projects have been completed to date. The total value of the remaining projects is approximately $24 billion. In 2000, the government also approved a second phase of three DBFO projects. The three projects were all high value and involved a variety of payment mechanisms. Two projects used a congestion pricing mechanism, and one used a lane availability pricing mechanism. The risk sharing in DBFO contracts is also important. The general basis of the contracts is for the public sector to transfer as much risk as possible to the private sector. The one risk retained by the government is the acquisition of land for the projects. The government ensures that the right- of- way is available so that the projects can move forward. To date, 11 DBFO contracts have been awarded by the Highways Agency. Three PFI contracts have also been awarded. These contracts are the M6 Northern Relief Road (M6 Toll), which is the first and only toll road in England; the National Traffic Control Center, which is part of the agency’s improvement in services as it moves toward being the network operator; and the National Roads Telecommunication Center, which is updating and improving the telecommunication networks adjacent to the Highways Agency network. These three contracts have a capital value of just less than $4 billion. The next DBFO, which is currently being developed, is on the M25. It is anticipated to be awarded in 2008. The typical DBFO structure involves numerous public- and private- sector groups. All the contracts include some type of performance mechanism that must be monitored. The Highways Agency has overall management responsi- bility for the DBFO contract. The lead DBFO company typically includes contractors from the design, construc- tion, operation, maintenance, and financial sectors. The core service delivery requirements for DBFO companies are defined in each contract. These require- ments include providing a safe highway, including reduc- ing casualties and maintaining a safe and serviceable network. Other requirements focus on minimizing dis- ruptions and delays from incidents and maintenance, as well as keeping travelers informed, including maintain- ing the necessary technology. Several benefits may be realized using DBFO contracts. Potential benefits include fostering development of a private- sector road- operating industry and improved cost certainty. Other possible benefits include making more reliable and accurate expenditure forecasts, transferring appropriate levels of risk to the private sector, and improving partnerships between the public and private sectors. Still other potential benefits include promoting innovation in all areas, including finance, and providing a better value for the money. An independent audit indi- cated a 15% savings using DBFO contracts, compared with the traditional public- sector approach. The DBFO process provides benefits to the public sec- tor, the private sector, and customers. The public sector obtains a better value for the money. The private sector benefits from a secure revenue stream. Customers bene- fit from safe and more reliable roadways and other trans- portation facilities and services. Finally, let me provide a little information about the latest DBFO project. The M25 London Orbital roadway is the next DBFO project, which is scheduled to be awarded in 2008. It will be the largest DBFO project awarded by the agency. The overall project, including construction, operation, and maintenance is valued at approximately $10 billion. The M25 is the largest ring road in the United Kingdom and is currently very con- gested. With the impending 2012 Summer Olympic Games being hosted in London, one of the main require- ments of the contract will be that no construction can be scheduled during the Olympic period. This makes the contract a little more difficult. We are also looking at how to maximize management of the new lanes. The five main performance criteria for the contract will focus on lane availability, route performance, condition, safety performance, and proactive management. PERFORMANCE- BASED CONTRACTING AND MEASURING PROJECT DELIVERY: MEXICO Amado Rubio Athie We all share a common problem of obtaining adequate funding for transportation projects. Innovative ap- proaches are needed to address transportation financing requirements. Mexico needs significant investments in the highway sector. On an annual basis, Mexico’s road construction and maintenance needs require an average of approxi- mately $5 billion. Available public funds allow the fed- eral government to invest less than half the required amount. To close this gap, Mexico has developed three public–private partnership (PPP) models that seek to attract private capital to highway investments. The three models are highway concessions, PPPs or projects for the provision of service (PPS), and asset utilization. High- way concessions use toll income, PPSs use service pay- ments, and asset utilization maximizes existing toll road income flows. These models have been applied in the health, educa- tion, and transportation sectors. Six performance- based contracts have been awarded in transportation. The first service contract, to modernize the Irapuato- La Piedad federal road, was successfully awarded in August 2005 and will be finished in the middle of 2008. These approaches are helping increase the roadway inventory in Mexico. 35PERFORMANCE- BASED CONTRACTING AND MEASURING PROJECT DELIVERY

The PPP model has several objectives. These objec- tives include allowing an earlier development of Mex- ico’s free roads, although the model has also been used with new toll roads. Other objectives include increasing the amount of road investments with private participa- tion, adding value to the services offered to users, and increasing the efficiency and productivity of public ser- vices. Still other objectives are to create new opportuni- ties for the private sector and to better distribute and manage highway project risks. The PPP model has a number of characteristics. A con- cession is awarded through a public bidding process, which also grants the concessionaire the exclusive right to sign the service contract. The duration of the service con- tract is fixed, from 15 to 30 years. The contract is fixed in time and amount. The contract establishes an association between the ministry and a private firm to design, finance, build, maintain, and operate a highway. The private firm provides services in exchange for periodic payments. The periodic payments are based on the availability of the road and traffic levels. Each bidder requests a periodic payment determined as a function of construction, maintenance, and operat- ing costs; the rate of return on equity, including financial costs; estimated annual traffic; and duration of the con- tract. The net present value of the periodic payments is the decision criterion used to award the concession, if the winner complies with technical, legal, and financial requirements. The government always maintains the property of the road. At the end of the concession period, the road is returned to the government. When the model is applied to a toll road, the periodic payment is made with a combination of toll revenues and budgetary funds. The risks in PPP projects are divided between the gov- ernment and the private sector. The government retains and manages the risks associated with planning and permitting, regulatory changes, and demand. The risk associated with design, construction, productivity, obso- lescence and hidden faults, operation, and financing are transferred to the private sector. The risks related to force majeure, archeological findings, and inflation are han- dled and managed by the government and the private sector. The legal structure begins with the bidding process. The bids are evaluated, and an award is made to the selected private group. The title of concession includes the purpose and duration, general conditions, obliga- tions, and an early termination clause. The contract includes the scope of services, the concessionaire obliga- tions, the payment mechanism, inspection procedures, penalties, and guidance for transferring the road to the government at the end of the concession period. The Queretaro- Irapuato Road provides an example of the typical approach. The private service contract for improving this federal toll- free road includes widening 48 km of roads to four lanes, maintaining the 93-km road for 20 years, building seven interchanges, building four overpasses, providing emergency assistance for users, and building a 4.4 km boulevard. To adequately specify the expected services, the road is divided in 16 subsections. The activities to be performed in each sec- tion include building additional lanes; widening shoul- ders; and providing maintenance, traffic signs, mechanical assistance, incident attention, and communi- cations and lightening in suburban areas. Technical stan- dards are set, and inspection, supervision, and control procedures are defined. Payments to the service provider are based on perfor- mance. The payment mechanism considers the availabil- ity of the road to users, traffic levels, the shadow toll requested by the service provider, and deductions when the road is not available for use. Payments are scheduled on a quarterly basis and are applied for each subsection of the road. The performance of the service provider is also mea- sured. The service provider will have to design, improve, maintain, and operate the road according to government requirements related to the physical road characteristics, specification of the operation activities, maintenance requirements, other services on the road, and road char- acteristics at the end of the contract. The government designates a representative who is responsible for ensur- ing that the requirements are met throughout the duration of the contract. These models are used primarily to improve the char- acteristics and services of toll- free roads. The main crite- ria that are applied to select road sections to be improved include the number of users who will benefit, the value for money through risk transfer to the private sector, and the importance of the road at the regional and state lev- els. Other criteria include the ability to complete key transport corridors, the availability of alternative sources of funds, and the potential to support economic develop- ment in specific regions, such as in southeastern Mexico. In its first stage, contracts have been awarded on six roadway modernization and improvement projects using these approaches. The bidding process will be initiated soon on two additional projects. PERFORMANCE- BASED CONTRACTING AND MEASURING PROJECT DELIVERY: CONCESSION AGREEMENTS Stephen C. Beatty My comments focus on the use of concession agreements and possible management, reporting, and measurement issues. It is important that control points are identified throughout the entire project when developing a conces- 36 U.S. AND INTERNATIONAL APPROACHES TO PERFORMANCE MEASUREMENT

sion agreement. One of the potential benefits of PPPs is the freedom for concessionaires to be creative. It is important that this freedom occur within the parameters established by the transportation agency, however. It is important to know what you are asking for and to clearly ask for what you want. Key elements to consider in developing a concession contract include design and construction, operations, tolling, maintenance, expansion, rehabilitation, and handback. Jurisdictions that have been successful in the use of concession agreements have tied the measurement and management requirements back to the project objec- tives. The ability to direct and control must be set out in concession contracts. The concession agreement is the control mechanism for the implementation of a PPP. We favor two types of requirements— specific and general— for use in a conces- sion agreement. The first type of requirement is very spe- cific about a certain action or activity. An example of a specific requirement would be specifying the width of the travel lanes or the depth of the pavement. The second type of requirement provides general direction. An example of a more general requirement would be to operate the facility in a safe manner. A significant issue to address in design and construc- tion is who designs the facility and to what standard. To maximize the potential creativity of the private sector, performance standards may be used in design and con- struction. One of the most difficult tasks that public agencies face is converting their specifications, needs, and measures into performance requirements. In Texas, it took 1 year to accomplish this task, with more than 100 people involved in the process. Elements to include in a concession agreement on design and construction are identifying who will certify the project design and who will assume the liability for the design. Independent engineers are frequently used in PPPs to address poten- tial liability concerns. We have found that the International Organization for Standardization methodology is beneficial in check- ing construction standards. On the finance side, the lender’s engineer will be focusing on construction stan- dards. Public agencies may also use a quality auditor or quality oversight approach. Other issues to consider are substantial completion and total completion. In a toll road or payment situation, you want to tie the ability to charge tolls to substantial completion, but you also want to ensure total completion. In terms of operation, you want to create an incentive for the concessionaire to do the right thing. You want the developer to act as the pub- lic agency would. Elements to consider include incident response, snow and hazard clearing, spills, utilities and other rights holders, safety, and customer service. The use of shadow tolls, availability payments, and conces- sions may also be addressed. Recent advances in technology have improved accu- racy and verification in tolling. Enforcement and collec- tion have significant implications for the concession agreement. There may be issues with respect to equity among users: those who pay and those who do not pay. Setting toll rates probably gets the most media and pub- lic attention. The concession agreement should define who has the authority to set tolls and the guidelines or criteria that will be used to increase or decrease toll rates. The expansion of a facility is often overlooked in con- cession agreements. This issue is important in long- term concessions and should be addressed in the agreement. You want the concessionaire to have the economic incen- tive to expand the facility when demand warrants. Expansion triggers are one method to accommodate these needs. The concession agreement should also include how additional interchanges or access points will be accommodated. A concession agreement should also address rehabili- tation needs. Dealing with rehabilitation has become eas- ier with the widespread acceptance of the independent examination concept. Contractual provisions should address the timing of rehabilitation, including late- in- the- term rehabilitation. Handback represents the final major element of con- cession agreements. The public sector should expect a well- maintained, 50-year old road in 50 years, rather than a brand new road in 50 years. Requiring inspection of a facility 5 to 10 years from the handback date and the development of a work list for needed repairs is a good approach. The concession agreement should identify how disputes will be resolved. Requiring a performance secu- rity, such as a letter of credit, is a good approach. Other issues may emerge during PPPs. It may be nec- essary to address defects in existing sections of a facility. Public agencies may have strong desires for control, which often need to be tempered with the concession- aire’s need to run a business. Both groups should focus on their appropriate roles and responsibilities. PERFORMANCE- BASED CONTRACTING: A VIABLE CONTRACT OPTION? Sidney Scott III My comments focus on performance contracting from a broad context. I am currently working on a Strategic Highway Research Program 2 (SHRP 2) project examin- ing performance specifications. This project is exploring techniques to convert existing standard specifications into a performance- based framework. It is important to start by examining the traditional way of doing business, which tends to be “Just tell me what to do; I want to build it and move on.” The tradi- 37PERFORMANCE- BASED CONTRACTING AND MEASURING PROJECT DELIVERY

tional method is price driven, with the burden of risk on the owner to design, specify, and control the work. There are exceptions to this traditional approach, however. For example, warrantees have been used since the late 1800s. Therefore, some of the concepts we are considering today are not really new. We are rediscovering approaches, as well as developing new techniques. We use methods and materials or specifications that describe the methods or process requirements and the materials or products to be used. In a general sense, per- formance contracting can be described as a goal- or results- oriented process. It is best- value driven in that price and other factors are considered. It does not dictate how to do something, except essential needs or mini- mum functions and standards. Performance contracting rewards innovation and rewards “better- than- the- minimum” outcomes. Innovation often comes from the private sector’s providing new ideas. We need to interject more innovative ideas in our business models in the transportation sector. At the same time, performance contracting ensures that public agencies, acting as stew- ards of public funds, obtain the maximum value of available resources. In this country, we have focused on prescriptive requirements in highway contracting. This approach focuses on the result. We are moving more toward the use of performance contracting, however. In addition to PPPs, other models include performance warranties, performance- based maintenance or asset management contracts, and design–build. “Highways for Life” is an FHWA- sponsored performance contracting program to develop a framework within the traditional system. We are seeing a number of performance- based con- tracting initiatives in the United States today. Many of these initiatives are modeled after the techniques being used in Europe and other parts of the world. The SHRP 2 project, Performance Specifications for Rapid High- way Renewal, is developing a performance framework and guide or standard specifications to capture the essen- tial benefits. We started by examining the following questions that were identified in a 2004 FHWA Strategic Roadmap. What do we want? How do we order it? How do we measure what we ordered? How do we know we got what we ordered? What do we do if we do not get what we ordered? These are some of the basic questions that should be asked in the development of a performance framework. The first question relates to defining what a public agency wants in a project. As other speakers have noted, a good place to start is with the goals identified by the agency in the planning process or the project develop- ment process. Typically, there are both high- level goals and project- specific goals. High- level goals may relate to safety, congestion reduction, innovation, and long- term performance. Examples of project goals include time sav- ings, cost reduction, and product quality. Other project goals might focus on traffic management, work zone safety, and the environment. I think innovation is an important goal. Allowing the contractor to use innovative approaches, within mini- mum requirements, can provide numerous benefits. Innovation is a common thread throughout many of the new contracting approaches. It is through innovation that value is added to these new contracting methods. The second question focuses on the actual procure- ment method. There are many ways to buy services. The European model tends to be qualifications based. Char- acteristics of this approach include a prequalification process and negotiated contracts. Public agencies in the United States have historically focused on price consid- eration. This approach uses open bidding and fixed- price contracts. PPPs reflect characteristics of the European approach, especially providing flexibility for innovation and involving the contractor early in the process. Several items need to be considered in performance- based contracts. First, it is important to determine whether the work can be described in terms of end result performance. For example, pavement may be described in terms of smoothness, strength, durability, aesthetic features, life, and safety. Second, does the contract pro- vide for multiple means and methods or alternatives for achieving intended results, or does it specify one approach? Third, it is important to consider whether the specified alternatives are practical and economical. Fourth, if multiple factors contribute to the desired result, determining the relative importance of their con- tribution will be necessary. Finally, it is important to use factors that are measurable and testable. There are several ways to measure what public agen- cies ordered. Traffic management can be measured by the travel time through work zones. Worker safety can be measured by incident rates. Pavement quality can be mea- sured by smoothness. Pavement safety can be measured by skid resistance. Schedule adherence can be measured by the percentage ahead of schedule. Surveys can be used to measure user satisfaction. In addition to trust, public agencies can use a variety of tests to ensure the final prod- uct meets the specifications. Tests need to be rapid and reliable, repeatable, and achievable and economical. The final question addresses what public agencies can do if the project does not meet the requirements or spec- ifications. There are many approaches that can be used if this situation occurs. Incentive–disincentive strategies in a contract can be used to promote better- than- the- minimum performance, as well as addressing specific performance criteria. Pay adjustment systems can be linked to incentive–disincentive strategies, with payment based on value received. Shared risk contingency repre- sents another approach that establishes a contingency fund. This technique provides incentives to the contrac- 38 U.S. AND INTERNATIONAL APPROACHES TO PERFORMANCE MEASUREMENT

tor to exceed targets, because it receives a share of the dollars in the contingency fund at the end of the project. In summary, performance contracting represents a new way of doing business in the United States. We are incorporating ideas from approaches used in countries throughout the world. Performance contracting does represent a cultural shift for public transportation agen- cies and for contractors. We tend to have numerous smaller contracting firms in the United States, whereas other countries have larger, integrated contracting firms. It is also important to remember that performance con- tracting is not appropriate for all projects. Performance contracting requires evaluating risk and the potential for risk–reward sharing. It requires a framework and speci- fications. Numerous benefits can be realized from per- formance contracting, including innovation, better value for the public investment, and improved service to the public. Anthony Kane, American Association of State Highway and Transportation Officials, moderated this session, and Josias Zietsman, Texas Transportation Institute, served as rapporteur. 39PERFORMANCE- BASED CONTRACTING AND MEASURING PROJECT DELIVERY