Evaluation of Project Delivery Methods (2009)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

TCRP G-08 – Project Delivery Methods Chapter 8 - Risk Based Approach CHAPTER 8 – TIER 3 – RISK-BASED APPROACH Introduction The Tier 3 optimal risk-based approach will leverage the current cutting-edge risk-based cost estimating methods that have emerged in the transit and highway agencies in the past few years (Touran 1994, Parsons 2004). The user should first complete Tier 1 and Tier 2. It is expected that most of the times the delivery method decision can be made by completing these two tiers. Even if a clear choice cannot be established after going through the first two tiers, then at least the first two tiers will short-list the number of viable choices. It is expected that by the time the decision-maker gets to Tier 3, they are only looking at two candidates for the delivery method. This is important because the effort involved in using Tier 3 (especially the quantitative approach) is considerably larger than either Tier 1 or Tier 2. The risk-based approach as proposed here consists of two phases. The first phase is a qualitative approach consisting of developing a risk allocation matrix that clearly portrays owner’s risk under competing delivery methods. By reviewing these risks, the owner (in our case mostly the transit agency) will have an opportunity to decide if a specific delivery method is superior. If this analysis cannot provide a definitive answer to the delivery selection question, then a quantitative approach should be considered. The quantitative approach emphasizes the effect of the Project Delivery Method (PDM) on project cost and schedule. As can be observed in Fig. 8.1, a two-phase approach is suggested for the risk-based PDM selection. The process depicted should be repeated for all the short-listed PDMs surviving the Tier 2 process. In the first phase, the PDM is selected mainly based on the Risk Allocation Matrix. This phase of Tier 3 is called the Qualitative approach. If after this phase still more than one choice remains equally viable, then a complete risk analysis would be required to quantify the effect of PDM on project’s cost and duration and finalize the PDM selection. This second process is called the Quantitative approach. Northeastern University The Research Report 161

TCRP G-08 – Project Delivery Methods Chapter 8 - Risk Based Approach Northeastern University The Research Report 162 Figure 8-1 - Overview of the Risk-based Qualitative and Quantitative Approaches Due to cost escalation on large transit projects, since 2002, the FTA has required that each “New Starts” project undergo a formal risk-based cost estimate. Specific requirements for these risk assessments are provided in FTA guidance documents8. A risk-based cost estimate generates a range of possible project costs rather than a single point estimate as seen in Figure 8.2. This distribution represents the combined effect of various risks that affect project cost. Using this distribution, the project owner would be able to estimate the probability of finishing the project within a specified budget. Alternatively, the owner can establish sufficient contingency budget to keep the probability of cost overrun or schedule delay below a specified threshold. Figure 8-2 - Distribution of Project Costs The same modeling method (and much of the same data) that is used to generate the cost and schedule risk analysis can be used to make more informed decisions and allocate risks appropriately, in essence, optimizing the project delivery and contracting decisions. One of the major findings of the structured interviews (conducted with transit agencies as part of this effort) was the apparent effect of a rigorous risk analysis on the project success. It was found that projects that paid more attention to risk analysis fared better in terms of achieving their budget 8 As an example, the current FTA guidance on risk assessment is PMO Operating Procedures No. 40, “Risk Management Products and Procedures, 2007, FTA, U.S. D.o.T.

TCRP G-08 – Project Delivery Methods Chapter 8 - Risk Based Approach and schedule goals. The following sections describe qualitative and quantitative phases in more details. Qualitative Approach The overview of the process is shown in Fig. 8.3. The risk-based approach is superimposed on the project development life-cycle. The most likely time to decide on the PDM is either at the end of the Conceptual Design or during the Preliminary Engineering phase. If the project goes into the Final Design, the agency will lose the opportunity to effectively use alternative delivery methods and will be limited to the traditional DBB approach. At the end of the Conceptual Design, the agency usually has not done a detailed risk analysis. If the PDM selection decision cannot be finalized by going through the first two tiers described in previous chapters, the agency would need to conduct a preliminary risk analysis in order to be able to make an informed decision regarding the choice of the PDM. Figure 8-3 - Overview of the Risk-based Approach The result of this preliminary risk analysis is a Risk Allocation Matrix. The risk allocation matrix has become an industry standard for legal teams when authoring alternative contracts for large infrastructure projects. As an example, a risk allocation matrix was a first step in creating the contract for the T-REX multimodal design-build project in Colorado. Table 8.1 shows a hypothetical risk allocation as envisaged in this guide. Risk factors are major events or conditions that can affect the project in a negative way (the events that can affect the project in a positive way are called opportunities and traditionally are far fewer than risks). Care should be taken to consider only the significant risks otherwise identifying and measuring all project risks would be a major effort. For each of these risk factors (that can be arranged according to their impact (rank) or their chronology) a main responsible party should be identified, given a certain project delivery method. As an example, the party responsible for Design defects in a DBB contract is the owner, whereas in the DB contract, the responsible party is the Constructor. To each risk factor, a rating will be assigned as to the effect of the PDM on the treatment of that risk factor, from the perspective of the Northeastern University The Research Report 163

TCRP G-08 – Project Delivery Methods Chapter 8 - Risk Based Approach owner agency. As an example, in Table 8.1, for the hypothetical project under study, the use of a DBB has a favorable effect for “Permits/approvals” risk from the agency’s point of view. It is felt that the agency is the best party to obtain these permits and that the agency can most effectively do this under a DBB approach. A rating of “+” is assigned for such a case. The same risk factor, under a DB delivery method is unfavorable from the agency’s point of view, because the agency feels that the DB constructor is not the best party to obtain various permits and approvals (such as environmental permits). A rating of “─” is assigned for such a case. As another example, the risk associated with “Design Defects” has a rating of “─” for the agency under the DBB arrangement because in this delivery method the agency is responsible for the accuracy of design. A DB approach on the other hand gains a “+” rating for the agency because it transfers this risk to the constructor. In summary, the ratings always evaluate a risk from the standpoint of the agency. If the choice of a PDM has no effect on a particular risk factor, then a rating of “0” will be assigned. In rating each risk factor, one can refer to the contents of Chapter 3 of this Guide, where advantages and disadvantages of various issues are documented. No attempt is made at this stage to quantify the impact of these risk factors (in terms of $ or project delay). After the matrix is developed and rated, the evaluation team can review the outcome and see if any PDM seems superior in terms of its capacity in dealing with these risk factors. For example, reviewing the matrix of Table 8.1 may lead one to believe that the DB is the better choice for the owner agency because of the number of favorable ratings that it has obtained. Table 8-1 - Example of Risk Allocation Matrix DBB DB Risk Factor Responsible Rating Responsible Rating Permits/Approval Owner + Constructor/Owner - Different Site Conditions Owner 0 Constructor/Owner + Design Defects Owner - Constructor + QA/QC Constructor/Owner 0 Constructor + Exchange Rate Risk Owner - Owner - Other risk factors Preparation of the risk allocation matrix and rating the risk factors can be accomplished in a reasonable time. If the outcome points to a clear PDM winner, then the decision is finalized and the results, along with justification will be documented. If after going through the proposed process, the choice is still not clear, then the process should be moved to a more detailed quantitative approach. Northeastern University The Research Report 164

TCRP G-08 – Project Delivery Methods Chapter 8 - Risk Based Approach It is emphasized that the ratings assigned to each risk factor is chosen from the owner agency’s point of view (and the potential benefits/loss to the owner as a result of a specific PDM). Quantitative Approach The quantitative approach should be attempted only if the qualitative approach does not result in a clear choice for the PDM. As shown in Fig. 8.3, the quantitative approach is suggested to be used at the conclusion of the P.E. phase, when the agency has conducted the FTA mandated probabilistic risk analysis on project cost and schedule. The risk analysis process is a major undertaking that will require hundreds of man-hours over the course of several weeks. The outcome of the risk analysis can also be used as input to the PDM selection decision (Fig. 8.4). The quantitative phase of Tier 3 would then be contingent on the availability of the complete risk analysis. If this risk analysis is not a requirement (for example in projects that do not apply for federal funding), then it is suggested that the PDM selection decision be made without this phase as the cost of this phase could be prohibitive. Figure 8-4 - Risk Analysis Outcome as an Input to PDM Selection The outcome of the probabilistic risk analysis required by the FTA consists of a distribution (range of possible values) for project cost and duration. Also, a list of the most important risk factors ranked according to their impact on budget or schedule is provided as part of the risk mitigation report. Usually, the number of these ranked risks is limited (for example, in several risk assessments conducted by the Project Management Oversight consultants on behalf of the FTA, the list of significant risks factors were between 10 to 15). This approach follows the logic of the Pareto’s law (also known as the 80-20 rule, the law of the vital few) which states that, for many events, 80% of the effects come from 20% of the causes. In the context of project risks, relatively few risks are responsible for most of the project cost or schedule overruns. The project cost distribution and the list of ranked risks will serve as inputs to the process of selecting the best PDM. For each ranked risk a distribution of risk costs is usually estimated. The highest ranked risks are those with large expected values and large ranges (an indication of high variability of the risk factor). The proposed process, called the quantitative approach in this work, will involve estimating the effect of each major risk factor on the agency’s budget, assuming a specific delivery method. The process starts by reviewing all the risk factors and select those risk factors where the choice of project Northeastern University The Research Report 165

TCRP G-08 – Project Delivery Methods Chapter 8 - Risk Based Approach delivery will affect their value. Only the risk factors that are sensitive to the project delivery method will be selected for further analysis. For each of these risk factors, the range of cost will be estimated under assumed project delivery method. This can best be accomplished by some of the same experts that were involved in the risk analysis. Fig. 8.5 provides an example of a hypothetical project where four major risk factors have been identified as the risk factors that are affected by the choice of project delivery method. These risk factors consist of permits, utility relocation, differing site conditions (DSC), and third party issues. Assume that the two choices for the PDM are DBB and DB. The cost of each risk is estimated using a triangular distribution, although many other distributions can be used depending on the nature of the risk factor. In a triangular distribution, the range of possible values is estimated with a lower bound (optimistic), an upper bound (pessimistic) and a most likely value. The triangular distribution is commonly used in probabilistic risk analysis because of its simplicity. The sum of these risk costs will give the distribution for the total risk costs. There are statistical methods to calculate this sum with relative ease. Comparison of distributions of these total risk costs will give the owner agency a valuable tool for assessing the effect of project delivery method on the project cost. A similar approach can be used to assess the effect of risks on project schedule. If the purpose is to examine the effect of delivery method on project duration, all the distributions depicted in Fig. 8.5 will have durations on the X-axis and the total effect will be the total impact on project schedule instead of cost. The quantitative approach is a powerful tool for comparing competing PDMs. It focuses on those differences between the PDMs that affect cost and schedule and provides a consistent way of evaluating each PDM vis-à-vis major risk factors affecting the project. This will allow the decision maker to document the reasons for the selection of a specific project delivery method. The drawback of this approach is its dependency on the availability of the expensive risk analysis results and the higher skill level required for pricing out each risk under various PDMs. However, the choice of the PDM is a natural outcome of a risk analysis exercise because one of the most important benefits of any risk analysis is risk allocation/mitigation. A properly selected PDM is an effective risk mitigation instrument that can help keep project costs low and project delays minimized. Northeastern University The Research Report 166

TCRP G-08 – Project Delivery Methods Chapter 8 - Risk Based Approach Figure 8-5 - Overview of the Quantitative Approach DB DBB Permits Pr ob . Pr ob . $ $ Utility Pro b. Pr ob . DSC $ $ Pr ob . Pr ob . $ $ Pr ob . Third Party Pro b. $ $ Pr ob . Pr ob . $ $ Total Effect on Cost Northeastern University The Research Report 167

TCRP G-08 – Project Delivery Methods Chapter 8 - Risk Based Approach Northeastern University The Research Report 168 Summary Tier 3 may be needed in cases when Tiers 1 and 2 cannot provide a clear best choice for the project delivery method. In such a case, Tier 3 can be used at two levels: qualitative and quantitative. Both approaches are based on a risk allocation exercise that will outlay major project risks to the agency under various delivery methods. In the qualitative approach, the decision-makers will base their final decision on the careful examination of each risk factor and after deliberating over the anticipated effect of each risk factor on project cost and schedule. This critical review can help the agency decide on the most appropriate delivery method. If this process still does not yield a final outcome, the agency can then proceed with the quantitative approach. In this approach, the cost and schedule effect of each risk factor will be estimated with an appropriate range, summed up, and used in comparing the total effect of risks under competing delivery methods. The agency can then select the delivery method that results in the most favorable outcome considering both cost and schedule.