Performance Metrics for Public–Private Partnerships (2021)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

101   A P P E N D I X E Proposed Research Needs Statements Problem Title Guidebook for Developing Effective P3 Project Performance Metrics Research Problem Statement NCHRP Project 20-05/Topic 51-19: “Performance Metrics for Public-Private Partnerships” identified a gap in the body of knowledge regarding the implementation of transportation infrastructure public–private partnerships (P3). A P3 is defined as a contract between the public and private sectors for the delivery of a project or service in which the private partner provides the majority of the necessary financing. The synthesis found that no definitive guidance exists on how to quantify asset performance. The literature reviewed determined that an effective metric is focused on outcomes rather than merely outputs. At the theoretical level, this seems like a simple difference. However, in the world of complex megaprojects, the sheer magnitude of the requirement to develop and deploy metrics that cover everything from administrative procedures to reasonable residual lives for a plethora of asset classes is daunting. Therefore, practical guidance is needed to provide for differentiating between an output and an outcome. The objective of this project is to sample successful P3 projects and determine the types of metrics in use and their relative effectiveness in quantifying asset performance. The effort would likely consist of sampling a variety of performance metrics for common elements of a given asset, like pavements or incident response, from a variety of P3 projects. The actual performance of each of these would then be measured against the standards found in the concession agreement and the actual payment adjustments to provide a measure of efficacy. Simultaneously, both the agency and Concessionaire would be surveyed to furnish perceptional data that could be compared to the quantitative data to identify trends and potential disconnects between actual and perceived effectiveness. In recent years, the establishment of the U.S. DOT’s Build America Bureau and enabling legislation in many states have facilitated the implementation of P3 projects in the transportation sector. However, the selection of P3 implementation, just like any other project delivery method, is driven primarily by the goals and objectives of the DOT and the specific project. DOTs are still faced with many challenges in aligning these goals and objectives to the delivery methods of choice. In addition, once a project delivery selection is made, monitoring project performance over the length of the project, given this decision, becomes vital for assessing and improving future projects’ performance and delivery decisions. Performance metrics in P3 projects could address the public client’s overall strategic plan and mission objectives, as well as the overall quality and performance of a given asset. These performance metrics should include key indicators of travel time reliability, safety, and overall project physical condition, as well as other project elements. Despite the importance of performance metrics, knowledge is relatively limited about the most effective practices for selecting metrics, setting metrics performance levels, and reasonable approaches to deductions and/or point systems for failure of the Concessionaire to meet contract requirements that ultimately align with the DOT’s goals and objectives.

102 Performance Metrics for Public–Private Partnerships • What are the different arrangements of P3s used in the transportation industry? What are the key selection parameters for each, in light of DOT project goals and objectives? • What are the key performance metrics for P3 projects? How can they be measured? How effective are they? • What are effective ways to tie performance metrics to payment mechanisms (i.e., lane availability, route performance, condition criteria, safety performance, unplanned events) to the P3 Contractor? • If the P3 Concessionaire does not comply with performance standards, what actions (e.g., nonconformance reports or penalty point notices) should be taken? • If the P3 Concessionaire maintains or exceeds the level of performance specified for the majority of the contract term, what incentives, if any, should be provided? • What techniques have been used to avoid and resolve disputes? Which have been most effective and which have been less effective? Objectives The objective of this project is thus to develop a structured decision-making model for DOTs on implementing an effective P3 project, from inception to operations, that is driven by the DOT goals and priorities and assessed by performance metrics pertinent to the selected P3 arrangement (from availability payments, to sharing revenue, to full privatization, among others). The decision- making model might take into consideration factors such as various P3 arrangements and financing options and resources and time constraints. Based on the results of this study, a guidebook will then be developed to provide information on practices that may assist in selecting various P3 arrangements as well as assess the project using key performance metrics that align with the project goals and objectives. The purpose of this project is to (1) investigate the different P3 arrangements used in the transportation industry and link them to DOT project goals and objectives, (2) examine performance metrics for P3 projects used by DOTs as well as other countries in the delivery of transportation infrastructure, (3) document lessons learned, and (4) make implementation recommendations that will improve U.S. policy and practice. This project will produce an empirical guide based on effective practices regarding the selection of performance measures for P3 projects and provide guidance on selecting different P3 arrangements/structures given DOT goals and objectives as well as effective practices for implementing performance measures. Specific tasks of the research to accomplish the main objective include • Task 1 – Conduct a comprehensive literature review of studies related to P3 arrangements and how they are linked to DOTs and projects’ goals and objectives as well as to performance measurement. • Task 2 – Benchmark the state of the practice, across the various transportation agencies, related to P3 selection choices and establishing project performance metrics for P3 projects. • Task 3 – Prepare a research work plan that describes the details of the research methodology and methods for identifying effective practices and developing conclusions. The proposed research could address the following questions: • Task 4 – Conduct representative detailed case studies on current and completed P3 projects to identify selection mechanisms and constraints as well as effective performance measurement practices and lessons learned. • Task 5 – Execute the research work plan and prepare an interim research report that articulates the data collection and analysis as well as emerging conclusions, effective practices, and lessons learned; develop a proposed outline for the guidebook and the case example report, and draft language. • Task 6 – Publish the guidebook to assist DOTs in implementing an effective P3 project, from inception to operations that is driven by the DOT goals and priorities and assessed by performance metrics pertinent to the selected P3 framework and based on proven effective practices. • Task 7 – Prepare a final research report detailing the complete results of the research. Estimate of Problem Funding and Research Period The estimated project budget is $500,000, and the period is 24–36 months.

Proposed Research Needs Statements 103   Urgency, Payoff Potential, and Implementation During the development and operation phases of a project, performance metrics will be compared with established performance objectives that serve as a baseline to determine how successful organizations (and the private sector partner) have been in attaining project objectives. The payoff of this research will be improved decision making for DOTs on various P3 arrangements, as well as administration and control of P3 projects, which ultimately will result in higher public satisfaction. The guidelines will become available to transportation agencies through the traditional NCHRP distribution processes. Related Research Federal Highway Administration. Public-Private Partnerships for Highway Infrastructure: Capitalizing on International Experience. International Technology Scanning Program 2009. Gransberg, D.D., DeWitt, S., and Touran, A. NCHRP Synthesis 563: Performance Metrics for Public–Private Partnerships. Transportation Research Board, Washington, D.C., 2021. Yuan, J.F., Zeng, A.J.Y., Skibniewski, M.J., and Li, Q.M. Selection of Performance Objectives and Key Performance Indicators in Public-Private Partnership Projects to Achieve Value for Money. Construction Management and Economics, 27(3), 2009, 253–270. Yuan, J., Skibniewski, J., Li, Q., and Zheng, L. Performance Objectives Selection Model in Public-Private Partnership Projects Based on the Perspective of Stakeholders. Journal of Management in Engineering, 26(2), 2009, 89–104.

104 Performance Metrics for Public–Private Partnerships Problem Title Guidebook for Developing P3 Project Handback Criteria Research Problem Statement NCHRP Project 20-05/Topic 51-19: “Performance Metrics for Public-Private Partnerships” identified a gap in the body of knowledge regarding the implementation of transportation infrastructure public-private partnerships (P3). A P3 is defined as a contract between the public and private sectors for the delivery of a project or service in which the private partner provides the majority of the necessary financing. The synthesis found that no definitive guidance exists on how to develop handback criteria when an asset is returned from the Concessionaire to the agency. Since few projects have reached a point where the contractual handback criteria have been tested, it will be more difficult to conduct an empirical study to produce the necessary guidance. However, the synthesis found current P3 projects use an approach where different features of work (e.g., pavements, bridges, or lighting) in a single P3 project have different handback periods. Therefore, it might be possible to empirically study the shorter termed features. The World Bank has used build-operate-transfer (BOT) delivery to build toll roads and other infrastructure in developing countries with 5 to 15-year O&M periods and contain a robust set of handback criteria that has been tested. This data might be a good departure point for the effort. The synthesis also identified the use of residual life as a key component of a number of P3 projects’ handback processes. While the concept of residual life has been well-researched in the life-cycle cost literature, the concept remains highly theoretical and requires a number of assumptions to be made to conduct the residual life computation, which in turn is highly sensitive to the assumptions themselves. Hence a pragmatic approach to rationally computing residual life that is directly applied to the P3 project handback process is needed to assist in managing this risk by the public agency. The research would sample a variety of handback criteria for common elements of a given asset, like pavements or incident response, from a variety of P3 projects. The actual performance of each of these would then be measured against the handback standards found in the concession agreement to provide a measure of efficacy. Simultaneously, both the agency and Concessionaire would be surveyed to furnish perceptional data that could be compared with the quantitative data to identify trends and potential disconnects between actual and perceived effectiveness. The proposed research could address the following questions: • What are the different arrangements of P3s used in the transportation industry, and how do handback criteria vary between each arrangement? • What are the key handback criteria for P3 projects? How can they be measured? How effective are they? • What techniques have been used to avoid and resolve disputes? Which have been most effective and which have been less effective? Objectives The objective of this project is thus to develop a structured decision-making model for DOTs on implementing an effective P3 project handback criteria pertinent to the selected P3 arrangement (from availability payments, to sharing revenue, to full privatization, among others). The decision- making model might take into consideration factors such as various P3 arrangements and financing options and resources and time constraints. Based on the results of this study, a guidebook will then be developed to provide information on practices that may assist in selecting handback criteria for various P3 arrangements, as well as assessing the project using key handback metrics that align with the project goals and objectives. Specific tasks of the research to accomplish the main objectives include • Task 1 – Conduct a comprehensive literature review of studies related to P3 arrangements, how they are linked to DOTs and projects' goals and objectives, as well as to performance measurement. • Task 2 – Benchmark the state of the practice, across the various transportation agencies, related to P3 selection choices and establishing handback criteria for P3 projects.

Proposed Research Needs Statements 105   • Task 3 – Conduct representative detailed case studies on current and completed P3 projects to identify handback criteria selection mechanisms and constraints, as well as effective performance measurement practices and lessons learned. • Task 4 – Execute the research work plan and prepare an interim research report that articulates the data collection and analysis, as well as emerging conclusions, effective practices, lessons learned; develop a proposed outline for the guidebook, the case example report and draft language. • Task 5 – Publish the guidebook to assist DOTs in implementing an effective P3 project handback criteria development program, and assessed by performance metrics pertinent to the selected P3 framework and based on proven effective practices. • Task 6 – Prepare a final research report detailing the complete results of the research. Estimate of Problem Funding and Research Period The estimated project budget is $300,000 and the period is 24–36 months. Urgency, Payoff Potential, and Implementation The increased access to and use of private funding to rectify the nation’s crumbling infrastructure has furnished a valuable contribution to the transportation system. P3 projects have fixed concession periods, and it is critical that the DOT has the means in place to ensure that the asset is in satisfactory operating condition when it is handed back to the agency. The payoff of this research will be improved decision making for DOT on various P3 arrangements, as well as administration and control of P3 projects, which ultimately will result in higher public satisfaction. The guidelines will become available to transportation agencies through the traditional NCHRP distribution processes. Related Research Federal Highway Administration. Public-Private Partnerships for Highway Infrastructure: Capitalizing on International Experience. International Technology Scanning Program, 2009. Gransberg, D.D., DeWitt, S., and Touran, A. NCHRP Synthesis 563: Performance Metrics for Public–Private Partnerships. Transportation Research Board, Washington, D.C., 2021. World Bank Group. Concessions, Build-Operate-Transfer (BOT) and Design-Build-Operate (DBO) Projects. Legal Resource Center, 2018. https://ppp.worldbank.org/public-private- partnership/agreements/concessions-bots-dbos [September 22, 2019]. Yuan, J.F., Zeng, A.J.Y., Skibniewski, M.J., and Li, Q.M. Selection of Performance Objectives and Key Performance Indicators in Public-Private Partnership Projects to Achieve Value for Money. Construction Management and Economics, 27(3), 2009, 253–270. Yuan, J., Skibniewski, J., Li, Q., and Zheng, L. Performance Objectives Selection Model in Public-Private Partnership Projects Based on the Perspective of Stakeholders. Journal of Management in Engineering, 26(2), 2009, 89–104.

106 Performance Metrics for Public–Private Partnerships Problem Title Guidebook for Developing Life-Cycle Incentive/Disincentive Schemes for P3 Projects Research Problem Statement NCHRP Project 20-05/Topic 51-19: “Performance Metrics for Public-Private Partnerships” identified a gap in the body of knowledge regarding the implementation of incentive/disincentive schemes in public–private partnerships (P3). A P3 is defined as a contract between the public and private sectors for the delivery of a project or service in which the private partner provides the majority of the necessary financing. The synthesis found that no definitive guidance exists on how to compute incentive/disincentive (I/D) amounts for P3 projects. Although there has been past research on the overall topic, it is focused on the design and construction phase of a project’s life cycle and does not extend beyond construction completion. One well-documented benefit of P3 is its life-cycle contract that imposes a fiduciary responsibility on the Concessionaire for the performance of the designed and constructed product after it is put in service. As a result, there is a need to tie the O&M period to the preconstruction and construction phases in a manner that encourages life-cycle decision making. I/D schemes have proven to be quite effective in non-P3 projects and are in use on most P3 projects. Thus, research to determine the manner in which non- P3 I/D schemes can be better adapted to P3 projects is required to ensure that the life-cycle benefits of P3 project delivery are indeed accrued by the public agency. At the present time, there is no comprehensive understanding of a) What transportation agencies are using as I/Ds. b) How the I/Ds are implemented through contract provisions. c) What criteria and evaluation are considered when applying I/Ds to projects. d) If I/Ds are cost-effective and fair to all parties. e) What impacts I/Ds have on project quality, delivery, and safety for all modes of transportation. This research is needed to provide answers to the items stated above and to identify best practices on the use of Incentives and Disincentives on P3 transportation projects. The study could address the following questions: a) Which transportation agencies have used I/Ds on construction projects? What has worked? What has not worked, and why? b) How did the transportation agency develop the contract provisions? Was the contracting industry involved? c) How were projects that used I/Ds selected? d) How was the value of the I/Ds determined? What was the true cost of the I/Ds (including added transportation agency cost through oversight, administration, staffing, etc.)? e) How was it determined that the I/Ds had been earned (e.g., data/information gathered or citizen input)? f) What impact did I/Ds have on the quality and safety of the project for both the users and the construction activities? tasks: • Task 1 – Survey transportation agencies as to their use of I/Ds. • Task 2 – Conduct case studies on P3 project and document contract provisions, criteria for project selection, value of I/Ds, lessons learned, true cost, and impact on quality and safety. • Task 3 – Prepare guidelines that include a process for selecting, computing, implementing, and evaluating I/Ds for P3 projects. Estimate of Problem Funding and Research Period The estimated project budget is $500,000, and the period is 24–36 months. Objectives This objective of this project is to empirically evaluate the use of incentives/disincentives (I/Ds) in P3 projects. The research objective will be achieved through the completion of the following

Proposed Research Needs Statements 107   Urgency, Payoff Potential, and Implementation During the development and operation phases of a project, performance metrics are compared with established performance objectives that serve as a baseline from which I/D is executed as part of the project payment process. The payoff of this research will be increased life-cycle decision making during the design and construction phase of P3 projects, as well as administration and control of P3 projects, which ultimately will result in higher public satisfaction. The guidelines will become available to transportation agencies through the traditional NCHRP distribution processes. Related Research Federal Highway Administration. Public-Private Partnerships for Highway Infrastructure: Capitalizing on International Experience. International Technology Scanning Program, 2009. Gransberg, D.D., DeWitt, S., and Touran, A. NCHRP Synthesis 563: Performance Metrics for Public–Private Partnerships. Transportation Research Board, Washington, D.C., 2021. Leontescu, M., and Svilane, E. Incentive Mechanisms for Large Public-Private Partnerships: Empirical Evidence from SESAR. MS thesis. Jonkoping University, Smaland, Sweden, 2012. Yuan, J.F., Zeng, A.J.Y., Skibniewski, M.J., and Li, Q.M. Selection of Performance Objectives and Key Performance Indicators in Public-Private Partnership Projects to Achieve Value for Money. Construction Management and Economics, 27(3), 2009, 253–270. Yuan, J., Skibniewski, J., Li, Q., and Zheng, L. Performance Objectives Selection Model in Public-Private Partnership Projects Based on the Perspective of Stakeholders. Journal of Management in Engineering, 26(2), 2009, 89–104.

Abbreviations and acronyms used without definitions in TRB publications: A4A Airlines for America AAAE American Association of Airport Executives AASHO American Association of State Highway Officials AASHTO American Association of State Highway and Transportation Officials ACI–NA Airports Council International–North America ACRP Airport Cooperative Research Program ADA Americans with Disabilities Act APTA American Public Transportation Association ASCE American Society of Civil Engineers ASME American Society of Mechanical Engineers ASTM American Society for Testing and Materials ATA American Trucking Associations CTAA Community Transportation Association of America CTBSSP Commercial Truck and Bus Safety Synthesis Program DHS Department of Homeland Security DOE Department of Energy EPA Environmental Protection Agency FAA Federal Aviation Administration FAST Fixing America’s Surface Transportation Act (2015) FHWA Federal Highway Administration FMCSA Federal Motor Carrier Safety Administration FRA Federal Railroad Administration FTA Federal Transit Administration HMCRP Hazardous Materials Cooperative Research Program IEEE Institute of Electrical and Electronics Engineers ISTEA Intermodal Surface Transportation Efficiency Act of 1991 ITE Institute of Transportation Engineers MAP-21 Moving Ahead for Progress in the 21st Century Act (2012) NASA National Aeronautics and Space Administration NASAO National Association of State Aviation Officials NCFRP National Cooperative Freight Research Program NCHRP National Cooperative Highway Research Program NHTSA National Highway Traffic Safety Administration NTSB National Transportation Safety Board PHMSA Pipeline and Hazardous Materials Safety Administration RITA Research and Innovative Technology Administration SAE Society of Automotive Engineers SAFETEA-LU Safe, Accountable, Flexible, Efficient Transportation Equity Act: A Legacy for Users (2005) TCRP Transit Cooperative Research Program TDC Transit Development Corporation TEA-21 Transportation Equity Act for the 21st Century (1998) TRB Transportation Research Board TSA Transportation Security Administration U.S. DOT United States Department of Transportation

Perform ance M etrics for Public–Private Partnerships N CH RP Synthesis 563 TRB TRA N SPO RTATIO N RESEA RCH BO A RD 500 Fifth Street, N W W ashington, D C 20001 A D D RESS SERV ICE REQ U ESTED N O N -PR O FIT O R G . U .S. PO STA G E PA ID C O LU M B IA , M D PER M IT N O . 88 ISBN 978-0-309-67392-1 9 7 8 0 3 0 9 6 7 3 9 2 1 9 0 0 0 0