Performance Metrics for Public–Private Partnerships (2021)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

19   This chapter includes the results of the highway agency questionnaire and the P3 concession agreement content analysis. The thrust of the questionnaire was threefold. First, it sought to identify the usage of each of the common variations of P3s. Second, it asked questions regard- ing the types of performance metrics in use and the types of projects on which they are being used, and, third, it collected respondent perceptions regarding the relative effectiveness of the types of metrics in use. Their results, reported in this chapter, come from the 16 respondents who indicated that they had P3 projects in planning, in procurement, in design/construction, and/or under operation listed in Table 5. Nine of the respondents indicated that they personally did not have sufficient experience with P3 performance metrics to be able to provide a reliable response to their effectiveness. The following sections will report the remaining results of the survey. The reader is cautioned not to try and add the numbers in the tables and graphs to see if they total to the numbers cited above, as most of the questions permitted the respondents to check all possible answers that applied to their agency and offered an “other” option. There are a couple of other limitations that apply to trying to interpret the data. First, several respondents with P3 authority are only in the planning stage of potential projects (WDOT and the Maryland State Highway Administration). Others, like the MDOT, CTDOT, MassDOT, and WSDOT, have limited their P3 experience to non-highway projects. Nebraska’s single P3 experience is a Build-Finance project, where avail- ability payments are being made over a four-year period to amortize the Contractor’s financing component. Finally, only Georgia, Louisiana, and Texas reported having four or more P3 proj- ects. Hence, the remainder of the respondents’ experience is truly limited, making it difficult, if not impossible, to reach any conclusive findings from the survey results. Nevertheless, the objective to benchmark the state of the practice in P3 has been achieved, and it can best be described as a work in progress. As will be seen, no seemingly negative responses were collected, and that outcome holds the promise of increased growth in P3 usage. P3 Delivery Method Usage The survey offered the full list of P3 variations shown in Table 1 in Chapter 1, combined with the financing/risk approach used, and asked respondents to indicate their experiences with each possible combination. Table 6 illustrates the results of the responses received, along with any qualifications received with the responses. The Remarks response row in the table was used to capture responses only in the planning/consideration stage and the NDOT Build-Finance project. Table 6 shows a distinct preference for using availability payments over revenue risk as the financing mechanism for P3 projects. For highway projects, only NCTA and VDOT have chosen C H A P T E R 3 State of the Practice

20 Performance Metrics for Public–Private Partnerships the revenue risk only approach, with CDOT and LADOTD added to the group with hybrid proj- ects that mix revenue risk and availability payments. Table 6 also shows DBF and DBFOM to be the most commonly used. Additionally, of the three DOTs with more than four P3 projects, only VDOT and TxDOT have utilized revenue risk for P3 projects. Respondents were also asked to provide their rationale for determining which features of work were selected for performance measurement. Figure 6 shows the major criteria to be new construction or reconstruction followed by availability of the asset. This leads one to infer that performance measurement is typically required in two primary areas. The first is related to technical performance, which is related to expected asset service life. An example might be the Delivery Method Financing/Risk Approach RemarksAvailability Payments No. of Projects Revenue Risk No. of Projects Hybrid¹ No. of Projects DBF NCTA 1–3 CTDOT 1–3 CTDOT – Toll plazas FDOT 4+ GDOT 4+ LADOTD 4+ MassDOT 1–3 MassDOT – Parking garage PennDOT 1–3 TxDOT 4+ WSDOT 1–3 WSDOT – Ferry terminal DBFO MassDOT 1-3 CTDOT 1–3 CTDOT– Rest areaconcession DBFM GDOT 4+ PennDOT 1–3 DBOM Caltrans 1–3 WSDOT 1–3 WSDOT – Ferry terminal TxDOT 1-3 DBFOM TxDOT 4+ NCTA 1–3 CDOT 1–3 Caltrans 1–3 MDOT 1–3 MDOT – Rail transit CDOT 1–3 VDOT 4+ LADOTD 1–3 GDOT 4+ ODOT 1–3 ODOT 1–3 FDOT 1–3 Other WDOT 1–3 WDOT 1–3 WDOT 1–3 In planning stage MDOT 1–3 In planning stage NDOT 1–3 Build-Finance CDOT 1–3 In planning stage ¹Hybrid is defined as any structure with both an availability payment and a revenue risk component payable over the term of the contract. Table 6. Survey respondent usage of common P3 delivery variations. Agencies (Acronyms) California (Caltrans) Georgia (GDOT) North Carolina Turnpike Authority (NCTA) Texas (TxDOT) Colorado (CDOT) Louisiana (LADOTD) Nebraska (NDOT) Virginia (VDOT) Connecticut (CTDOT) Maryland (MDOT) Ohio (ODOT) Washington State (WSDOT) Florida (FDOT) Massachusetts (MassDOT) Pennsylvania (PennDOT) Wyoming (WDOT) Table 5. Agencies with P3 projects in planning, in procurement, in design/construction, and/or under operation.

State of the Practice 21   measurement of pavement skid resistance deterioration during the concession period with cor- rective maintenance trigger points, whereas the second criterion measures the Concessionaire’s ability to provide the asset’s expected level of service throughout the concession period. An example would be lane closures due to unplanned corrective maintenance. Performance Metric Usage Figure 7 shows the response to whether post-construction performance metrics were used and the length of time they had been in use. Caltrans responded with greater than 10 years, but it had only a single P3 project, the Presidio Parkway. Likewise, Connecticut’s P3 concessions for rest area services have been in force for more than a decade. Finally, DBF projects do not have post-construction components with metrics. Therefore, Figure 7 essentially confirms that P3 performance metrics do exist. Where found, the majority have been in use for more than 5 years, which may lend some authority to the perceptions of effectiveness. Figure 8 breaks out the project elements on which post-construction performance metrics have been developed and the number of projects on which they have been used. The greatest use is in new/reconstructed flexible pavements and bridges. Among the remaining categories, it is interesting to note that performance metrics for new/reconstructed rigid pavements are found in only one-third of the responses compared with flexible pavement. That leads one to wonder whether that is due to the difference in expected service lives of the two pavement types or if that is due to perhaps having more flexible pavement P3 projects in this small sample. A similar disparity is observed when comparing new bridge construction to bridge rehabilitation. Again, this may be due to the attractiveness of P3s when applied to new bridges versus rehabilitating New construction or reconstruction Statutory requirement Availability of Asset Traffic volume Project site conditions Project length Project complexity Pavement type 8 2 6 1 2 0 2 0 0 1 2 3 4 5 6 7 8 9 What criteria are used to determine which features of the P3 will be measured for performance? Check all that apply. Figure 6. Performance metric selection criteria.

22 Performance Metrics for Public–Private Partnerships Figure 8. Project elements with performance metrics and number of projects. 2 2 2 6 4 Yes, for less than a year Yes, for 1 to 5 years Yes, for 5 to 10 years Yes, for more than 10 years No 0 1 2 3 4 5 6 7 Does your agency have performance metrics in use during the O&M period on any of your P3 projects and if so, how long have you been using them? Figure 7. Performance metric usage and length of time in use (n = 16).

State of the Practice 23   existing bridges or the risk profile inherent to latent defects in existing bridges, which is absent in new bridge construction. Two respondents offered other project work elements as follows: • Performance metrics are used for “fence-to-fence” maintenance activities, excluding snow and ice removal (NCTA). • Drainage systems; markings and delineators; guardrail/barriers; signs; electrical systems; vegetated areas; geotechnical; damage to project facilities; customer services; monitoring and clearance of debris; geotechnical cuts and embankments (ODOT). Performance Metric Effectiveness Effectiveness of performance metrics is of prime interest to both the agency and the Conces- sionaire. Table 7 shows the perceived effectiveness for performance metrics used by the seven respondents that provided data. It shows that for all project elements, the metrics in use were deemed to be effective. Only for flexible pavement and vegetation control was there more than one marginally effective rating. The importance of this finding cannot be understated. Future research to survey Concessionaires is needed to validate the agency perception found in Table 7. The interviews for the PennDOT Rapid Bridge Replacement Program P3 case example proj- ect included the Concessionaire who mentioned that PennDOT’s use of the National Bridge Inspection Standards (NBIS) as the basis for many of its performance metrics was an excellent example. The NBIS was developed specifically to quantify the condition of the nation’s bridges against a uniform set of metrics. Because both the agency and the Concessionaire are intimately familiar with its content, it qualifies as a clear, justifiable source from which to draw appropriate performance metrics for P3 bridge projects. Table 8 shows typical cure periods for the various work elements. No particular trend is observed in this table. Project Element Not Effective Marginally Effective Effective Total Flexible pavement 0 2 5 7 Rigid pavement 0 0 4 4 Bridges 0 1 6 7 Guard rails 0 1 6 7 Attenuators 0 1 4 5 Fence 0 1 4 5 Signs 0 1 5 7 Drainage systems 0 1 4 5 Barrier wall 0 1 4 5 Sound barriers 0 1 3 4 Embankments 0 1 5 6 Earth/rock cuts 0 1 2 4 Toll equipment/buildings 0 0 1 1 Lighting 0 1 5 6 ITS operations 0 0 1 1 Incident response 0 0 5 5 Vegetation control 0 2 3 5 Table 7. Perceptions of performance metric effectiveness.

24 Performance Metrics for Public–Private Partnerships Handback Criteria The end of the concession period generally results in the Concessionaire returning the asset to the agency. Concession agreements typically contain a set of handback criteria that describes the asset’s required condition at handback. As previously mentioned in Chapter 2, these criteria are outputs rather than outcomes measured during the concession period for payment purposes. The minimum performance requirements are shown in the Table 2 example and function as both minimum standards during operation and trigger points for corrective maintenance if measurements drop below specified minimums. Hence, it is logical to use those criteria to set the standard for handback. This is not always the case. These performance measurements represent the point at which performance is no longer deemed to be satisfactory and require correction to keep the asset available. Thus, theoretically, they represent a point where, if all requirements were at the mea- sured trigger point, the asset would require major rehabilitation or replacement, meaning that the asset has reached the end of its economic service life. Some of the concession agreements reviewed for this synthesis asked the Concessionaire to restore the asset to levels that exceeded the trigger points used during the O&M payment process. Table 9 is an example of handback cri- teria that differ from the O&M performance metrics. One can see that the agency uses residual life at handback as a way to articulate the desired outcome. The survey asked respondents to state whether or not they had handback metrics that were different from the O&M performance metrics. Figure 9 provides the responses to that question and indicates that 75% of the respondents do not have different handback criteria. The results shown in Figure 9 demonstrate another need for future research on the efficacy of handback criteria. The upshot is that there is very little, if any, experience with the effec- tiveness of handback criteria in the United States, and international experience is anecdotal (Cui et al. 2018). The length of typical concession periods will make a direct study of U.S. P3 highway projects impossible for at least a couple of decades. However, the World Bank Asset Deficiency Response Time < 24 hours 1-3 days > 3 days Flexible pavement 3 1 5 Rigid pavement 2 0 4 Bridges 2 1 7 Guard rails 2 3 2 Attenuators 2 3 1 Fence 1 1 3 Signs 2 3 4 Drainage systems 2 0 6 Barrier wall 1 0 6 Sound barriers 1 0 4 Embankments 2 0 6 Earth/rock cuts 2 0 5 Toll equipment/buildings 1 0 0 Lighting 2 2 4 ITS operations 1 0 2 Incident response 6 0 0 Vegetation control 2 0 6 Table 8. Response times for correction of performance deficiencies.

State of the Practice 25   6 2 Yes No Figure 9. Handback performance metrics versus O&M metrics (n = 8). Asset Asset Subsystem Handback Evaluation Tasks Handback Evaluation Criteria Residual Life at Handback Flexible Pave- ment Pavement section within the Project Right of Way. Flexible pavement condition survey shall be conducted in accordance with the Pavement Condition Survey Manual and Maintenance Operations Manual within 180 calendar days before the end of the Term. Complete all tests in the Handback Renewal Work Plan to demonstrate the achievement of the required life remaining at the end of the Term. For any continuous one-mile segment, have a Pavement Condition Survey Manual rating of 80 or higher for HOT Lane and General Purpose Lane pavement; No pot holes; Achieve standards in the Handback Renewal Work Plan to demonstrate the achievement of the required life remaining at the end of the Term. 10 Years Guard- rail Guardrail systems within the Project Right of Way. Final inspection in accordance with the Maintenance Operations Manual shall be conducted within 45 calendar days before the end of the Term. Complete all tests in the Handback Renewal Work Plan to demonstrate the achievement of the required life remaining at the end of the Term. All guardrail systems must be within 0.75 inches of plumb and grade; Surface materials are smooth, undamaged free of defects; Rails and terminal elements are not warped or deformed; Posts are installed square to the rail; Painted rail shall be free of surface defects; Achieve standards in the Handback Renewal Work Plan to demonstrate the achievement of the required life remaining at the end of the Term. 8 Years Table 9. FDOT I-595 Handback Requirements Extract (2009).

26 Performance Metrics for Public–Private Partnerships has used Build-Operate-Transfer (BOT) delivery to build toll roads in developing countries for several decades (World Bank Group 2018). These development agreements are generally only 5 to 15 years long and contain a robust set of handback criteria that has been tested. These data might be a good departure point for the suggested research. Incentive and Disincentive Schemes The questionnaire asked respondents whether they used incentive/disincentive (I/D) schemes in conjunction with their P3 performance metrics. They were asked to furnish the details if they did. Figure 10 shows that half of the respondents did not employ I/D schemes in their P3 proj- ects. Those who did use I/D declined to provide the details, citing a lack of personal knowledge of the mechanics of the I/D process. Performance Metric Implementation Respondents were asked to identify the major challenges that they faced in implementing P3 performance metrics. The purpose of the question was twofold. First, the answers identified barriers to implementation. Secondly, the major challenges identified described potential areas for future research to assist agencies in understanding the challenges and providing guidance on how to better address them. The top five can be divided into the following two categories: • Quantifying asset performance: “defining threshold values,” “condition assessment,” and “specification development.” • Program implementation: “organizational issues” and “balancing performance and value ($).” The advent of total asset management planning (TAMP) has generated a great deal of research on specific asset class performance assessment. The NBIS is a good example of a body of work completed for bridges. Similar past research is available for pavements, signage, safety features, and so forth. Thus, future research in quantifying P3 asset performance would focus on the effectiveness of the criteria already available. The second category deals with implementation. Some guidance regarding organizational staffing issues can be found in NCHRP Synthesis 518: Staffing for Alternative Contracting Methods (Tran et al. 2018). The FHWA Center for Innovative Finance Support has devel- oped a P3 Toolkit, which might be of assistance when addressing the value versus cost issue 3 1 4 Yes, on all P3 projects Yes, on some P3 Projects No 0 1 2 3 4 5 Does your agency employ incentive/disincentive schemes for P3 projects? Figure 10. Incentive/disincentive (I/D) schemes (n = 8).

State of the Practice 27   (FHWA 2019b). That resource contains a tool called “P3 Value” and offers a methodology for making value for money comparisons with other project delivery methods (DeCorla-Souza and Lee 2017). Summary The survey results (Appendix B) and the content analyses of concession agreement perfor- mance metrics and handback criteria (Appendices C and D) show that the benchmark for the state of the practice in P3 performance metrics is at an early stage of development. Hence, no substantive conclusions are drawn from the analyses in this chapter due to the small size of the available sample. That being said, there is a rich base of O&M performance metrics and handback criteria that have been developed by those agencies that do have multiple P3 project experience. The survey indicated that those DOTs believe that those criteria are effective, and one interview with a Concessionaire validated that survey finding anecdotally. That information is contained in Appendix C and represents the first time a consolidated listing of P3 performance criteria has been assembled. It provides a menu of options to act as the basis for developing performance metrics for future P3 projects. One of the lessons learned cited in Chapter 2 suggested that an agency “review the performance standards from other states to ensure that requirements are reasonable and cure periods are appropriate and defendable” (CDOT 2018). Appendix C repre- sents a starting point for implementing that particular lesson learned.