National Academies Press: OpenBook

Performance Metrics for Public–Private Partnerships (2021)

Chapter: Chapter 2 - Literature Review

« Previous: Chapter 1 - Introduction
Page 10
Suggested Citation:"Chapter 2 - Literature Review." National Academies of Sciences, Engineering, and Medicine. 2021. Performance Metrics for Public–Private Partnerships. Washington, DC: The National Academies Press. doi: 10.17226/26171.
×
Page 10
Page 11
Suggested Citation:"Chapter 2 - Literature Review." National Academies of Sciences, Engineering, and Medicine. 2021. Performance Metrics for Public–Private Partnerships. Washington, DC: The National Academies Press. doi: 10.17226/26171.
×
Page 11
Page 12
Suggested Citation:"Chapter 2 - Literature Review." National Academies of Sciences, Engineering, and Medicine. 2021. Performance Metrics for Public–Private Partnerships. Washington, DC: The National Academies Press. doi: 10.17226/26171.
×
Page 12
Page 13
Suggested Citation:"Chapter 2 - Literature Review." National Academies of Sciences, Engineering, and Medicine. 2021. Performance Metrics for Public–Private Partnerships. Washington, DC: The National Academies Press. doi: 10.17226/26171.
×
Page 13
Page 14
Suggested Citation:"Chapter 2 - Literature Review." National Academies of Sciences, Engineering, and Medicine. 2021. Performance Metrics for Public–Private Partnerships. Washington, DC: The National Academies Press. doi: 10.17226/26171.
×
Page 14
Page 15
Suggested Citation:"Chapter 2 - Literature Review." National Academies of Sciences, Engineering, and Medicine. 2021. Performance Metrics for Public–Private Partnerships. Washington, DC: The National Academies Press. doi: 10.17226/26171.
×
Page 15
Page 16
Suggested Citation:"Chapter 2 - Literature Review." National Academies of Sciences, Engineering, and Medicine. 2021. Performance Metrics for Public–Private Partnerships. Washington, DC: The National Academies Press. doi: 10.17226/26171.
×
Page 16
Page 17
Suggested Citation:"Chapter 2 - Literature Review." National Academies of Sciences, Engineering, and Medicine. 2021. Performance Metrics for Public–Private Partnerships. Washington, DC: The National Academies Press. doi: 10.17226/26171.
×
Page 17
Page 18
Suggested Citation:"Chapter 2 - Literature Review." National Academies of Sciences, Engineering, and Medicine. 2021. Performance Metrics for Public–Private Partnerships. Washington, DC: The National Academies Press. doi: 10.17226/26171.
×
Page 18

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

10 This chapter documents the results of the literature review (Appendix A), which was con- ducted to collect the necessary details to develop the survey questionnaire. The topics discussed in this chapter include • P3 usage throughout the nation, • Key performance metrics for various DOT P3 contracts and how the metrics are categorized, • Handback requirements, • Methods by which performance requirements are enforced, and • Lessons learned. P3 Usage A P3 is a cooperative effort between public and private entities utilizing the expertise and resources of each to meet the public needs by delivering an infrastructure project. The use of this delivery method extends back about three decades and has increased in the United States in the past 10 years. Benefits of P3s include accelerating project delivery by the elimination/ reduction of large capital outlay for construction at the beginning of the project by the public entity, the potential for reducing the owner’s risk, and leveraging the innovative capabilities of the private sector for delivering a high-quality project within budget and allocated time, ensuring life-cycle optimization. P3 projects include new projects that are structured as project delivery with a finance component [with Design-Build-Finance-Operate-Maintain (DBFOM) being the most comprehensive type] or existing facilities where the public sector leases a toll facility to the private sector for toll collection and maintenance of the existing asset. Transportation has been one of the main sectors utilizing P3s. According to the Design- Build Institute of America (DBIA), all but eight states have full or partial authorization to utilize P3s as of 2020. However, this chart does not specifically address transportation proj- ects. The National Conference of State Legislators (NCSL) reported in 2010 that 29 states had enabling legislation for the use of P3 in transportation projects. As of 2018, the FHWA indi- cated that a “statutory framework for implementing transportation-related P3 projects (emphasis added by authors) could be found in 36 states, the District of Columbia, and Puerto Rico.” That data are shown in Figure 3. A recently completed survey for an FHWA project entitled Tools and Technical Assistance for Evaluation of Alternative Contracting Methods identified 26 state DOTs that had some form of P3 authorization as of 2019. The differences between the NCSL, the FHWA’s P3 website, and the FHWA study numbers are attributed to state-level P3-enabling legislation that did not include the state’s DOT. In some cases, the state tolling authority is a separate agency, apart from the DOT, and in other cases, enabling legislation only applies to municipal/local agency authorities or a specific transportation mode like airports or C H A P T E R 2 Literature Review

Literature Review 11   rail transit. Thus, one must conclude that without a detailed legal analysis of current enabling legislation, it is difficult to find a reliable total number of state DOTs that can use P3. Hence, the synthesis survey asked that specific question, and the remainder of the synthesis relies on answers provided by survey respondents. Technical and financial publications have discussed extensively the use of P3s and their char- acteristics. An extensive list covering these topics is provided in the Reference section at the end of the report. Relevant literature can be divided into the following categories: • Publications that describe various types of the P3 approach and its characteristics. These sources were published mainly in the early stages of P3 usage and had educational value. Also, many papers discuss critical success factors (CSF) and criteria that contribute to project success (Tiong et al. 1992; Li et al. 2005). • Papers that discuss the use of P3s in the context of a specific geographical region (Medda et al. 2012; Gurgun and Touran 2014). • Papers that present case studies of P3 projects and their degree of success and papers that can be used for benchmarking project performance with respect to cost, schedule, quality, and level of collaboration between public and private sectors (Haskins et al. 2002; Raisbeck et al. 2010). • Publications that provide a new methodology, paradigm, or model for studying a specific aspect of P3 delivery. These are mostly academic in nature. However, the methodologies introduced can be used in developing practical guidelines for preparing for a successful P3 experience. Examples include frameworks for establishing performance objectives (Liu et al. 2014), performance measurement metrics [key performance indicators (KPIs)] (Minchin et al. 2013), or a model for a risk informed approach in evaluating the residual value of the asset at the end of the concession period (Yuan et al. 2009; Yuan et al. 2015). Figure 3. P3 authorizations across the United States in 2018 (Source: FHWA; https://www.fhwa.dot.gov/ipd/p3/legislation/).

12 Performance Metrics for Public–Private Partnerships Key Performance Metrics in P3 Delivery Performance metrics in P3 projects typically address the DOT’s policy goals and the overall quality and performance requirements for the specific contracted work. These performance metrics typically include key indicators of travel time reliability, safety, overall project physical condition, asset availability, and other project elements. The condition of the asset at the time of handback is important to the public agency; to ensure satisfactory condition of the asset, the owner requires that the Contractor conduct inspections and evaluate the condition of the asset and make any necessary repairs and maintenance. The P3 project delivery process involves selecting the Contractor/Concessionaire, usually through a best-value procurement process. The Contractor will then complete the project and, depending on the type of P3, may be responsible for maintaining and operating the asset for a predefined period (usually 30 to 50 years) (FHWA 2014). At the completion of the conces- sion period, the asset will be handed back to the public agency according to the procedures established in the P3 contract. Once a project starts, monitoring project performance over the length of the project becomes vital for assessing and improving project performance and delivery decisions. The literature search was conducted to identify current suggestions for the selection of per- formance metrics in P3 projects. Many authors acknowledge that in order to have a successful P3 project, it is important to identify strategic objectives of the P3 project. According to Liu et al. (2014), strategic objectives are the foundation of performance measurement. P3 projects’ strategic objective is the achievement of best value, which emphasizes value for money and high performance standards (Akintoye et al. 2003; Liu et al. 2014). Rossi et al. (2004) proposed that performance measurement needed to include not only a measured output but also a commen- surate outcome, as shown in Figure 4. Liu et al. (2014) argued that the current system for gauging infrastructure project success is too focused on the inputs and outputs and very little on the outcome, which is in fact the best measure of performance. The FHWA’s P3 Procurement Guide (2019) supports this concept when it states that a P3 “enables the agency to focus on desired performance outcomes of a facility to maximize public benefit while letting the private entity determine the most efficient means to achieve those outcomes” (Smith et al. 2019). The guide goes on to say that by “estab- lishing requirements based on the performance of the facility or the project, the agency transfers risks to the Concessionaire while providing measurable metrics to gauge performance.” The Port of Miami tunnel provides a good example of how industry reacts when an agency concentrates its focus on performance outcomes rather than measurable outputs. This project was originally scoped to be a DB project delivery. When the agency conducted industry out- reach, it received a positive response from U.S. companies as proposed with a DB approach. However, the European firms that were contacted indicated that they would not pursue the project unless it was delivered as a P3 concession. Their logic was that a P3 gave them greater Figure 4. P3 Project Performance Measurement Model (adapted from Rossi et al. 2004).

Literature Review 13   flexibility to innovate and refine the tunnel’s design. It also provided a greater amount of finan- cial options when the design and construction were combined with the long-term operation of the tunnel. From the agency’s perspective, P3 delivery provided an opportunity for per- formance risk transfer because the European companies had experience with large-diameter, bored-tunnel construction, whereas the U.S. companies did not (Perez et al. 2016). Hence, the P3 approach allowed the agency to configure the procurement on its outcomes, leaving the competing P3 teams to determine the most cost-effective design and construction approach. Performance metrics allow project owners to assess project success during construction and operation. The FHWA’s P3 Procurement Guide (2019) suggests that when “developing perfor- mance metrics, the agency should determine whether the requirements can be clearly defined in the Project Agreement and whether and to what extent such requirements can be reasonably measured and enforced during the contract term. Sometimes a desired outcome or performance might make sense from a public policy perspective but in practice may be cost prohibitive or simply not practical for the agency to measure and enforce.” One common tool for performance measurement is a suite of various key performance indicators (KPIs). Several researchers noted that most KPIs are based on ex-post evaluation of cost and time performance and do not capture the complexities of a P3 project delivery system. In order to rectify this shortcoming, they have proposed various systems for performance measurement for P3 projects that consider many different factors, including project stakeholders and their objectives and motivations. A detailed conceptual listing of project KPIs is provided by Love et al. (2014) and Yuan et al. (2009). The focus of this synthesis is on metrics used to measure long-term performance of the con- structed facility. A common issue with developing these metrics encountered by owners that are new to the P3 arena is making the culture shift from measuring the quality of the constructed project to measuring how well it meets the key success factors established early in the P3 proj- ect development process. SHRP 2 Report S2-R07-RR-1: Performance Specifications for Rapid Highway Renewal presented a framework for performance specifications and guidelines for writing performance-based requirements for projects delivered by all delivery methods (Scott et al. 2014). The report issued the following cautionary thoughts on P3 performance metric development: Inexperienced P3 owners may be inclined to focus their procurements on performance criteria for the design and construction of the facility as opposed to criteria for how the facility will be operated and maintained. It is advisable to have a multidisciplinary team create performance requirements for the facility. Unlike method specification, a performance requirement promotes contractor innovation while focusing the agency resources on monitoring the performance outcomes in accordance with the pre- determined performance criteria. This approach, combined with financial penalties for failure to meet the performance targets, transfers performance risk to the private sector, giving the concessionaire “skin in the game” with respect to construction or operational performance outcomes. Furthermore, because the agency only specifies outcomes not methods, the private entity has flexibility to select materials and tech- niques that provide the most efficient solution to meeting the performance criteria. The agency should, however, recognize that this approach necessitates a high level of responsibility for the agency to identify all parameters critical for performance and to establish thresholds for both construction and operation performance requirements in the RFP documents (Scott et al. 2014). The SHRP 2 study differentiates between outcomes and methods. Other authors use the term “output” rather than “method” (Yuan et al. 2009; Love et al. 2014). The semantic concept is a bit abstract. So, the easiest way to bring clarity to the intent found in the literature is by referring to a simple example. Table 3 lists several common features of a typical P3 highway project and provides an output with the associated outcome to demonstrate the difference between the two concepts. This concept is an area where staff training is in order to help agency P3 project develop- ment teams develop both an understanding of the definitions of outcome versus output and the

14 Performance Metrics for Public–Private Partnerships skill set necessary to be able to apply each concept appropriately in the P3 solicitation and execu- tion processes. The major thrust of staff training is to shift the agency’s procurement culture from the DBB default to maximizing prescriptiveness to a true life-cycle mentality that focuses beyond the completion of construction and seeks to plan, design, and build toward an outcome rather than a construction quality control standard. Performance Measurement Frameworks Several frameworks have been developed by researchers for an effective performance mea- surement system. One framework models P3 processes and factors across the project’s life cycle, as shown in Figure 5. The model lists the areas in which key performance metrics must be End Product Output Outcome Concrete foundation Compressive strength > 3,000 psi Settlement < 0.5 in. over 20 years Fencing Fences and fence posts shall be plumb, and fence shall have no sags or deflections greater than 6 in. Fences shall be functional, visible, and accessible. Levee groundwater cut- off wall Reach a depth of at least 20 feet. No seepage during 100-year flood event. Culvert Clear obstructed culverts, when they are greater than 50% blocked. Self-flushing during service life. Table 3. Example outputs versus outcomes. Figure 5. P3 Process and Factors Performance Measurement Framework (Yuan et al. 2009).

Literature Review 15   developed. This synthesis will focus on performance metrics found in the operation, transfer management, and possibly post-transfer stages of the P3 project life cycle shown in Figure 5. Another example is the performance prism approach suggested by Neely et al. (2001) and proposed by Love et al. (2014), which provides a holistic approach to performance measure- ment by explicitly considering the multi-stakeholder environment present in P3 projects. Finally, Mladenovic et al. (2013) provide a comprehensive review of P3 publications and a list of KPIs used in P3 projects. The efficacy of the proposed systems has not been rigorously tested in practice as of this date. The content analysis of P3 concession agreements (Appendix C) found that, in general, P3 performance metrics can be categorized into the following types: • Administration, inspection, reporting • Roadway – General condition – Flexible pavements – Rigid pavements • Slopes, fencing, and vegetation • Litter, debris, and hazardous materials • Stormwater, drainage, and basins • Signage or pavement markings • Lane closures • Landscaping, sound, and privacy walls • Structures – General condition – Damage repair – Bridge decks • ITS and communications systems • Tolling • Electrical and signals • Storm and major damage • Incident response • Snow and ice control Handback Requirements and Enforcement In 2014, the Office of Innovative Program Delivery of FHWA published the report Model Public-Private Partnership Core Toll Concessions Contract Guide to provide guidance for public owners considering the use of P3s. This document provides detailed examples and model con- tract language about various aspects of P3 delivery. Of special interest to the current synthesis is the chapter on handback requirements. As mentioned earlier, the main emphasis in this study effort is to focus on the post-construction stage of P3 contracts. One of the important issues in P3 delivery, especially those including the operations and maintenance, is that the Concessionaire hands back the project at the end of the concession period in a state of good repair. This will ensure that the Concessionaire will build the project by following appropriate standards and with high quality; otherwise, it may be inclined to finish the project at minimum cost. A content analysis of agency handback criteria was completed and can be found in Appen- dix D. There is a knowledge gap regarding the handback requirements and methods of enforce- ment partly because very few highway projects using the P3 approach have reached the end of the concession period. Cui et al. (2018) reviewed highway projects delivered using P3s and could identify only three highway projects that have reached the end of their concession

16 Performance Metrics for Public–Private Partnerships periods. All of these projects were international projects, so the information regarding U.S. projects in this respect is minimal. Our search has not identified any additional highway projects that have reached the end of their concession period. Cui et al. (2018) identified three critical elements in the handback process: • Clear definition of handback requirements in the contract. • Clear and explicit definition of the inspection process during the concession period, including frequency and timing to ensure that the asset remains at an acceptable condition at the time of handback. • Type and extent of financial securities needed to ensure that meeting handback requirements is enforceable. Various financial instruments such as bonds, letters of credit, and project handback reserve fund have been suggested for this purpose. The overall conclusion of the study was that the three completed projects did not have all of the above requirements. For example, two out of three projects did not spell out handback requirements and handback procedures clearly in the contract. Despite this, all three projects were deemed successful from the owners’ point of view. These were very early P3 projects, and this was cited as a reason for not having all of the necessary elements of a prudent contract in place. It seemed that the most important factor for project success was the good working rela- tionship and trust that existed between the public and private sectors in these projects. A review of contract documents of U.S. P3 projects shows that the current contracts have become more robust in covering handback issues. Performance-Based Payment for Availability Payment Projects These metrics directly determine how much the Concessionaire will be paid in each periodic payment under the availability payment contracting approach. Because of the complexity of the O&M period in a major P3 project, most agencies have developed a system that is focused on a suite of KPIs that are regularly monitored and measured. These are periodically reviewed, and the impact of uncorrected deficiencies is quantified using a point system established in the concession agreement. The concession agreements reviewed in the content analysis found that a typical performance-based payment process involved five components: 1. Maximum availability payment for a given year during the concession period for which a periodic payment is being computed. 2. Measured performance metrics. 3. Deficiency severity classification. 4. Cure period for correcting noted deficiencies. 5. A scoring scheme that converted the number of deficiencies that were not corrected within the cure period to a payment adjustment factor. Table 2, as shown in Chapter 1, is an extract from the FDOT I-595 concession agreement showing the KPIs, called minimum performance requirements; a cure period; and a severity classification, called O&M violation classifications. In this example, the Concessionaire receives a quarterly payment (QP) equal to 25% of the current year’s maximum availability payment (MAP) less adjustments for failing to meet the KPIs by correcting any identified deficiencies within the designated cure period for the given KPI. FDOT used Equations 1 and 2 to calculate the QP: QP MAP QPAq y y q y= − 4 (1), , QPA QUA QVAq y q y q y= + (2), , ,

Literature Review 17   where MAPy = maximum availability payment in year y; QPq,y = quarterly payment for quarter q in year y; QPAq,y = quarterly payment adjustment for quarter q in year y; QUAq,y = quarterly unavailability adjustment for quarter q in year y; and QVAq,y = quarterly O&M violation adjustment for quarter q in year y. The unavailability adjustments and O&M violation adjustments vary based on the severity classification shown in Table 4. Without getting deeper into the mechanics of the computation than necessary to make the point, one can see that as a given violation becomes more severe, the reduction in the QP becomes greater. To provide an order of magnitude, the MAP on this project for the first year of O&M was $65.9 million. This would lead to a deduction of $183,056 ($65.9m divided by 360 from Table 4) for a single Class E O&M violation. The I-595 hourly unavailability violation calculation is not as straightforward as the O&M violation calculation. This factor is location specific, where each segment of the project has its own unique time and segment weighting factors. Thus, providing an example calculation would not provide a representative order of magnitude like the O&M calculation. Nevertheless, the metrics associated with both types need to be unambiguous and, more importantly, defensible. Their perceived effectiveness makes them a key success factor for the O&M concession period. Another example comes from the US-36 project in Colorado. That concession agreement uses a point-based scheme for assessing penalties for unavailability. The total points for a specific violation are assessed on severity, failure to cure, and/or appropriate reporting, and each point costs the Concessionaire $5,000 at the end of each month. Lessons Learned There were few lessons learned regarding performance metric development and efficacy found in the P3 literature. Nevertheless, there were a few discovered that can be applied in a broader sense to the performance metric process. These are listed as follows: • Each P3 agreement will become a precedent to the next (Virginia DOT). • It is important to establish an identification system that is standard for the entire main- tenance program to permit the tracking of deficiencies, corrective actions, and locations (PennDOT). Unavailability Factors O&M Violation Factors Availability Classification Hourly Unavailability Factor O&M Violation Classification O&M Violation Adjustment A 0.1 A MAPy/40,000 B 0.2 B MAPy/8,000 C 0.4 C MAPy/4,000 D 0.6 D MAPy/1,600 E 0.7 E MAPy/360 F 0.8 - - G 1.0 - - Hyphens = no data. Adapted from FDOT 2009. Table 4. FDOT I-595 concession agreement adjustment factors.

18 Performance Metrics for Public–Private Partnerships • ATCs must be vetted through the maintenance provider before final approval and inclusion in the concession agreement (PennDOT). • A P3 program governance structure supplemented with working groups covering quality, schedule, and commercial issues worked best (PennDOT). • Splitting payments into availability payments, which are subject to performance deductions, and milestone payments to provide a guaranteed cash flow enhance interest from industry (Michigan DOT) [applicable only to availability payment (AP) projects]. • Set maintenance standards that are higher than agency minimums because the payment scheme creates a strong motivation for the Concessionaire to achieve those standards (Texas DOT). • Review the performance standards from other states to ensure that requirements are reason- able and cure periods are appropriate and defendable (Colorado DOT). Summary The review of the cogent literature on P3 performance metrics leads one to conclude that this is truly a work in progress. The length of typical P3 concessions creates a situation where an authoritative, retrospective analysis of P3 metric effectiveness will not be feasible for at least a decade or more. This issue is exacerbated by the size and diversity of U.S. P3 projects and further complicated by the constraints of individual state-enabling legislation. Complexity is added to such an analysis by the number of potential variations available in the procurement process, not to mention the different types of both public and private financing that can be brought to a project. There is one conclusion that is well supported in the literature, and that is the need to design P3 performance metrics that measure outcomes, not merely outputs. Hence, it becomes neces- sary to differentiate between the two. Differentiating between an output and an outcome may be the factor that differentiates the ultimate effectiveness of a given metric. Therefore, future research on this topic is needed. Additionally, the remainder of the synthesis will use this concept gleaned from the literature as its guiding principle.

Next: Chapter 3 - State of the Practice »
Performance Metrics for Public–Private Partnerships Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Public–private partnerships (P3) allow public transportation agencies to attract private financing to deliver needed highway infrastructure and not have to wait until the required funding is fully in place via traditional state and federal sources.

The TRB National Cooperative Highway Research Program'sNCHRP Synthesis 563: Performance Metrics for Public–Private Partnerships documents key performance metrics used in various long-term P3 contracts for the delivery of highway projects, including services by Departments of Transportation (DOTs).

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!