Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
36 C H A P T E R 4 An agency requires a pre-award review process that is both thorough and expeditious to accrue the previously described benefits. A contractor once described waiting for the outcome of design submittal reviews on awarded DB contracts as âsprinting in place while holding your breathâ (Benton 2010). Regardless of the method used to deliver the project, the review of an ATC is a âmini-DB design reviewâ in that the ACM contractor cannot move forward in its bid process until it knows whether or not the proposed ATC under review will be acceptable. Thus, there exists a need for speed in the agency review process. Compounding the complexity of the ATC review process are specific requirements for assigning design responsibility appropriate to the specific project delivery method in use. Other issues involve validating promised benefits and determining how those benefits will be captured in the final contract in terms of alterations to the price and/or contract period of performance. This chapter will discuss the procedures in use by DOTs to review, evaluate, and approve ATCs. The process for evaluating ATCs has the following four major components: 1. The review/evaluation team 2. The preconstruction schedule milestone requirements 3. The management of ATC design information during review 4. Assignment of design responsibility/liability for approved ATCs 4.1 ATC Review Process This section will develop the basis for an agencyâs decision to use ATCs and extend that rationale into the ATC review and evaluation process. As with all forms of alternative con- tracting, there should be a tangible reason for including ATCs in the ACM projectâs pro- curement process. The ATCs add a new dimension to the procurement, and it is incumbent upon the agency to share that reason with the industry so that competing ACM contractors understand the rationale for the additional effort they may need to expend to respond to a given projectâs solicitation. An agency may consider including a step in the early project scope development process that results in a specific logic for using or not using ATCs in the procurement process on a project-by-project basis during its project delivery method selec- tion decision process. Past research on alternative project delivery has shown that the probability of receiving responsive proposals increases with the amount of information given to competitors to interpret the solicitation documents and understand exactly what is important to the owner of the Agency Review and Evaluation of ATCs
Agency Review and Evaluation of ATCs 37 project. While there is no guidance in the literature on developing ATC proposals, there is some guidance regarding the development of DB and CMGC proposals that would appear to apply to ATCs (West 2012). This guidance suggests that the agency answer the following questions (adapted for ATCs by the authors of this guidebook) and include the answers in its solicitation: â¢ What is the primary purpose for building the project? â¢ What benefits in terms of cost, time, and quality improvements does the agency expect from procuring the project with ATCs? â¢ What is the most important reason for selecting ATCs? â¢ On what specific aspects of the project does the agency want to encourage ATCs? â¢ On what specific aspects of the project does the agency intend to exclude ATCs? 4.1.1 Review Process Design and Team Selection Figure 11 shows a generic model of a typical ATC review process based on the outcomes of the survey and case study analysis performed for this research. The following are some of the key aspects that are shown in Figure 11 that bear further explanation: â¢ Assemble Integrated Design Review Team. The word âintegratedâ is assumed to mean that agency team members will come from both the design and construction groups. â¢ Primary Project Performance Requirements. The terms âprimaryâ and ârequirementsâ are intended to indicate the focus of the ATC review and how it facilitates the achieve- ment of critical success factors for the project. While an approved ATC will eventually need to be comprehensively reviewed in the context of the overall project, the develop- ment of the most important aspects is a key to achieving consensus on the applicability of proposed ATCs. â¢ Project Performance Attribute. This term was borrowed from the California DOT (Caltrans) value analysis methodology because it accurately reflects an agencyâs objectives during an ATC review. To be approved, a given ATC must be found to be âequal to or better thanâ the baseline design. The project performance attributes are the rationale for the design decisions shown in the baseline design documents. â¢ Performance Attribute Matrix (PAM). This is another California DOT term that merely refers to the collection of all evaluated attributes. The second important portion of Figure 11 to be understood is that the PAM is first used by the evaluation team to rate the baseline design. This is necessary to benchmark the point from which âequal to or better thanâ will be measured. One can see that there is a âCon- structability Rating,â which is a good example of why an integrated team is required for an ATC review. Assembling an ATC review team requires careful thought and extends beyond merely ensuring that members are technically qualified to determine the viability of the proposed concepts. It is also necessary to empanel members who can evaluate the impact of each ATC on environmental permits, ROW, utilities, maintenance of traffic, and third-party impacts, as well as the legal requirements to properly incorporate the ATC into the final ACM contract. Another factor to consider is the presence of observers from outside the agency. In Minnesota and Utah, enabling legislation for ACMs requires outside membership on selection panels. Additionally, the potential exists that the ATC evaluation panels might have representa- tives from the FHWA, state resource agencies, and local government observing the evaluation.
38 Guidebook for Implementing Alternative Technical Concepts in All Types of Highway Project Delivery Methods Determine Project Performance Attributes Determine Primary Project Performance Requirements Primary Project Performance Requirement Project Performance Attribute Performance Attribute Matrix (PAM) Rate Baseline Project with PAM Assemble Integrated Design Review Team Baseline Technical Rating Baseline Schedule Rating Baseline Constructability Rating Baseline Cost Rating Determine Relative Attribute Importance Baseline Performance Rating (PR) ATC-1 ATC-n ATC-1 Constructability Rating ATC-1 Technical Rating ATC-1 Schedule Rating ATC-1 Cost Rating (if reqâd) ATC-n Constructability Rating ATC-n Technical Rating ATC-n Schedule Rating ATC-n Cost Rating (if reqâd) ATC-1 PR ATC-n PR Rate ATC-1 Project with PAM Rate ATC-n Project with PAM ATC-1 PR > Baseline PR ATC RejectedATC Approved YES NO ATC-n PR > Baseline PR YES NO Integrated Design Team Ratings Figure 11. Generic ATC review process.
Agency Review and Evaluation of ATCs 39 To cover this eventuality, the Texas DOT includes the following clause in their solicitation (Texas DOT 2008): [Competitors] are advised that observers from Federal or other agencies, including representatives of local agencies and municipalities, may observe the . . . evaluation process. . . . FHWA has agreed to take reasonable steps to prevent this information from becoming a public record. Outside observers [excluding FHWA] . . . will be required to sign the Texas DOTâs standard confidentiality agreement. As cited in the Texas DOT clause, the issue of confidentiality will need to be addressed for any outside observers. The strength of the Texas DOTâs approach to this issue is that by stating the possibility that individuals that are external to Texas DOT may be present, the agency effectively puts all proposers on notice that it cannot guarantee complete confidentiality except through its nondisclosure agreement. Such notice puts the competitors in a position of need- ing to vet every potential ATC to ensure that proprietary information is not accidently leaked outside the ATC evaluation panel. The downside of this is that it might cause some competitors not to propose certain ATCs because of the external observersâ presence. 4.1.2 Preconstruction Milestone Development When an agency selects a project on which to implement the ATC method, the procurement schedule needs to be verified to ensure sufficient time is allowed for the development of ATCs by industry, as well as for review by the agency. Time frames are highly dependent on both the scale and type of project, as well as project delivery method, individual agency capabilities and resources, and specific guidance unique to each project. At this writing, two of the DOTs that have implemented DBB ATCs suggest that preliminary design information be provided to industry as early as is practical. The decision to implement either limited- or full-scope ATCs can be made at various stages of project development. This research showed that notifying the industry of the presence of ATCs prior to formal advertisement of a solicitation is an effective practice. The length of the period prior to letting ranged from 3 months for some DB projects to a year or more for DBB and P3 projects. The idea is to give interested competitors as much time as is practical to evaluate the projectâs scope and generate and assess CATCs that may enhance a given teamâs competitiveness. It is also wise to consider whether ATC projects require an extended proposal development period after advertising and before proposals are due. In the 2012 EDC ATC implementation webinars delivered to the design and construction industry, it was stated that the time available for development and review of ATCs was often the key determinant in a teamâs decision of whether to bid on an ACM project and expend the resources to generate ATCs. The use of design packages to analyze ATC potential can be used to inform the decision of whether addi- tional time will be permitted for proposal preparation before bid day and, if so, how much. The amount of time selected will also depend on whether CATCs are requested for a limited review prior to receipt of formal ATCs. Additionally, the agency should consider whether the number of proposed ATCs and the attendant one-on-one meetings should be limited. The procurement periods for DB and P3 projects should include sufficient time for the receipt, discussion, review, and decision-making process to provide compet- ing design-builders and concessionaires time to evaluate the competitive potential of their proposed ATCs. Table 4 in Chapter 2 illustrates the fact that an agency interested in implementing the ATC method can do so by degrees if desired. It can start by using limited-scope ATCs that mini- mally impact a projectâs delivery schedule and select areas of scope, like construction staging,
40 Guidebook for Implementing Alternative Technical Concepts in All Types of Highway Project Delivery Methods that avoid the issue of design liability for critical features of work, like bridges. Once comfort with the ATC process has been gained, an agency can progressively increase the amount of scope eligible for ATCs until the agency finds its optimum level of value. 4.2 ATC Evaluation Process This chapterâs introduction likened evaluating ATCs to a âmini-DB design review.â Other authors have used the analogy that evaluating ATCs is similar to receiving contractor value engineering change proposals before award and getting to keep all the savings (Hitt 2012). As mentioned in Section 4.1.1, the ATC evaluation factors are based on the project performance attributes developed by the integrated review team, which may be modeled after the Caltrans value analysis methodology. However, the reader should not interpret this to mean that the depth and breadth of the evaluation process normally used on a DB project must be transferred to evaluating all ATCs. Experience has shown that keeping the process as simple as possible allows an agency to quickly make approval/disapproval decisions. Additionally, it is prudent for the agency to be consistent in its procurements by using similar âboilerplateâ language to describe the ATC evaluation process. 4.2.1 ATC Evaluation Factors The factors used to evaluate ATCs are generally of five types, and Table 8 provides a couple of examples for each type: 1. Factors to determine whether the proposed ATC actually qualifies as an ATC under the agencyâs definition. 2. Factors measuring the âequal to or better thanâ benchmark. 3. Factors measuring the potential cost benefits, if applicable. 4. Factors measuring the potential schedule benefits, if applicable. 5. Factors meant to evaluate the agencyâs sense of the acceptability and desirability of an indi- vidual ATC. FACTOR TYPE FACTOR ATC Qualification Proposed ATC complies with current project design criteria. ATC does not require a design deviation. âEqual To or Better Thanâ Proposed ATC PAM matrix score is greater than or equal to the baseline PAM matrix score. Proposed ATC cost is less than the engineerâs estimate for the modified feature of work. Proposed ATC schedule is less than the period specified in the solicitation. Potential Cost Benefits (if permitted or required) Proposed ATC costs more but has a lower life-cycle cost. Proposed ATC cost savings are validated. Proposed ATC meets minimum net savings amount after redesign costs. Potential Schedule Benefits Proposed ATC reduces the period traffic is disrupted. Proposed ATC schedule savings are validated. Proposed ATC reduces the construction period for the modified feature of work. Acceptability and Desirability Proposed ATC changes to the baseline design criteria have been successfully used in another U.S. agency. Proposed ATC reduces internal agency oversight costs. PAM = Performance attribute matrix Table 8. Example ATC evaluation factors.
Agency Review and Evaluation of ATCs 41 4.2.2 ATC Evaluation Plan The ATC evaluation plan must be both fair and transparent. It should be reduced to the evalu- ation of specific information needed to determine whether a given ATC can be approved. This evaluation is not the same as a DB proposal evaluation, where the end result is typically a best- value award decision. The ATC evaluation is totally focused on determining whether the con- tractor can base its bid on the proposed ATC. Ultimate award will follow the prescribed process for the given ACM, and the actual change in cost will be contained in the proposed price. Hence, much of the information available on DB evaluation planning, including most of the guidance provided in an agencyâs DB policy manual, may not apply to ATCs. The evaluation plan will be focused on the factors that are under evaluation with the most attention paid to making the âequal to or better thanâ decision in a manner that can be logically justified. The simplest ATC evaluation plan consists of a pass-fail rating in each of the five factor categories shown in Table 8. Since the contractors are not competing on the ATC evaluationâs final score (if there is one), but on the competitive impact that an approved ATC has on their technical and price proposals, there is no need to differentiate between the competitors. To that end, it is useful to rate each ATC against a pre-established standard and approve those candi- dates that meet or exceed the standard and disapprove those that do not. While the idea expressed in the previous paragraph may seem like an over-simplification of the task at hand, an agency should question the need to spend any more time than absolutely necessary to satisfy its due diligence duties under 23 CFR 636.209(b) to approve or disapprove a given ATC. To help set the appropriate standards, the agency can seek project-specific answers for the following questions when developing rating standards for ATC evaluation: â¢ Is there an objective minimum design and safety standard required that can be identified? â¢ Is there a desired level above the minimum? â¢ Is the desired level different from minimum? â¢ What aspects are covered by a qualitative standard? â¢ How do aesthetics fit into the item in question, if at all? â¢ Is there a way to establish an efficiency of alignment, layout, or design? â¢ What owner/user/customer preferences should be considered when establishing the standard? â¢ How does the need to seek approval for a design deviation impact the ATCâs viability? Lastly, the completed evaluation plan should be formalized and pro- vided to industry during the pre-bid meeting and discussed as part of the explanation of how ATCs will operate on the given project. Local statutes and policies will govern how much of the plan must be included in the solicitation. The key point is to make sure that competing contractors understand the process and can submit responsive ATCs for consideration. The research found two approaches to the make-up of the ATC evaluation team and the overall project proposal evaluation team. Several DOTs, including Minnesota and Washington, specifically exclude ATC evaluation panel members from participating in the evaluation of the final proposals and selection of the winning proposer. The underlying rationale is that an ATC evaluation is essentially a âgo/no-goâ decision. The evaluated ATC is either equal or better than the baseline or it isnât. The evaluation of full proposals in most cases involves a best- value determination with approved ATCs incorporated into both the technical and price submit- tals where they may not be separately visible. Therefore, the knowledge that a given ATC was approved, but the proposer chose not to include it in the final proposal may influence the evalu- atorâs objectivity. There were several similar reasons discovered during the case study inter- views conducted in this research. Conversely, some DOTs stated that the risk was not worth the additional resources required to field two evaluation panels for every ATC project. Hence, it is suggested that agencies that intend to implement ATCs consider both sides of this issue. The appendices of this report contain samples of DOT ATC evaluation packages for DBB, DB, and P3 projects.
42 Guidebook for Implementing Alternative Technical Concepts in All Types of Highway Project Delivery Methods 4.3 ATC Submittal Documentation The required content of an ATC proposal is dependent on whether the ATC is a full-scope or limited-scope submittal. Once again, the issue must be viewed from a full-spectrum perspective. Obviously, full-scope ATCs will require a larger effort to prepare and submit than limited-scope ATCs. Therefore, to prevent wasting industry effort, a project with a full-scope ATC often provides a process where contrac- tors develop and present to an agency a CATC/preliminary ATC on which they receive a speedy decision about its relative attractiveness. This is usually a pass/fail process with an option to request additional information if a concept appears promising. On the other side, limited-scope ATCs can often omit the first step because the agency has designated the areas in which it will welcome ATCs. In either case, it is incumbent on an agency to restrict the information it requires in a full or final ATC submittal to only the items it needs to evaluate to make a decision. The ATC process is presumed to allow award of the contract without completed construction documents in all ACMs. A review (conducted in this research) of 92 ATC solicitation documents from 34 state DOTs found the following major topics were commonly required in an ATC submittal: â¢ Narrative explanation of the concept and its expected benefits â¢ Preliminary drawings or sketches illustrating the ATC â¢ A list of design deviations that would be required if the ATC were to be approved â¢ A discussion of the ATCâs impact on the following items: â Permits/environmental â ROW â Utilities â Schedule â Maintenance of traffic Other required submittal items that were not widely observed, but provide the complete picture for possible ATC submittal requirements are the following: â¢ Discussion of ATC impact on project risk profile â¢ Information on previous successful implementation of the proposed ATC in other jurisdictions â¢ Examples of specifications from other agencies that are being proposed as substitutes for the baseline specifications â¢ Estimate of potential incentive amounts for ATCs that promise time savings â¢ Requirement to conduct and pass pilot testing prior to construction of ATC feature of work â¢ Requirement to submit the ATC design to independent peer review â¢ Requirement to specifically identify any design deviations in the context of the RFP section that is being changed Many DOTs encourage proposers to develop CATCs for a fast, early decision on whether to follow through with a formal ATC proposal. The purpose of this approach is to encourage ATC submittals by providing a mechanism to filter out those proposals that have no hope of approval and thereby save proposers the time and expense of developing an ATC that is later disapproved. A typical DOTâs system is as follows (Missouri DOT 2013): CATCs will require minimal engineering and are intended for the contractors to present their ideas prior to investing time and resources into detailed engineering of their concept. The CATC submittal shall include at a minimum: a) Detailed narrative of the change being proposed. b) Estimate of cost savings. c) Any impacts on project time or other impacts to the project. The appendices of this report contain detailed descriptions of DOT ATC submittal packages for DBB, DB, and P3 projects.
Agency Review and Evaluation of ATCs 43 One unrecognized value of the CATC process is that the initial one-on-one confidential meet- ing with each proposer provides an opportunity to identify and resolve ambiguities in the base- line design criteria, as well as clarify the intent of the baseline design. This formal mechanism functions as a filter for formal requests for information (RFIs). More on this topic is contained in Section 4.5. 4.4 ATC Liquidated Damages and Incentives/Disincentives This section will discuss the issues of liquidated damages (LD) and other methods to protect an agency if an approved ATC does not meet the expectations indicated in a proposal. While the ATC process is highly collaborative, it does not release the contractor from performing in the manner detailed in the ATC proposal. The research study collected data about agency policies for promulgating LD to proposed ATC schedule savings. Roughly one-third of respondents to a survey replied that ATCs approved with a promise of time savings do not trigger LD if the revised completion is not met. However, another third reported that contract completion dates are revised and do trigger LD, and the remaining third reported that they sometimes impose LD if the completion date promised is not met. It would seem that if an ATC was accepted on the basis of schedule benefits, imposing LD would be appropriate if the promised schedule improvement was not realized. However, the virtually even three-way split among respondents choosing from the three possible options makes it impossible to provide definitive guidance on this topic. Respondents to a survey described in NCHRP Synthesis 455 ranked incentive/disincentive schemes as the least important factor in successfully implementing ATCs (Gransberg et al. 2014). None of the documents reviewed in the content analysis presented in NCHRP Synthesis 455 directly linked ATCs to incentive/disincentive schemes. A number of documents contained early completion bonuses, which were no doubt impacted by any approved ATCs, but since those clauses were outside the ATC clause, it can be concluded that implementing ATCs does not create a requirement to adjust typical incentive/disincentive schemes that an agency may be considering for a given project. 4.5 Clarifications and Modifications of the Baseline Scope for Approved ATCs An ACM projectâs risk profile is directly related to what is known and what is not known at the time the contract is let. In all projects, one âknown unknownâ (Ward and Chapman 2003) is whether the contrac- tor is accurately interpreting the solicitation documents provided by the owner. 4.5.1 Pre-Award Clarifications of Project Scope Most agencies require competing ACM contractors to submit formal written RFIs, and then agencies publicly respond to each contractor giving the same information to all competitors. The result is a situation where a contractor may be unwilling to ask a question or pursue a clarification to avoid losing its competitive edge. In this case, no formal RFI is sent, and the contractor will either bid the most conservative interpretation or add a contingency to manage the risk that its interpretation is wrong. Both cases lead to unnecessary price increases for the project (Ward and Chapman 2003). âFormal requests for information (RFIs), which are answered in a public manner for all competitors, create a situation where a contractor may be hesitant to ask a question or seek a clarification for fear of losing a competitive edge.â (Ward and Chapman 2003)
44 Guidebook for Implementing Alternative Technical Concepts in All Types of Highway Project Delivery Methods Confidential one-on-one ATC meetings provide an opportunity to seek clarifications from an agency without revealing a potentially attractive idea to the competition. Additionally, these meetings provide the agency with a critical analysis of the industryâs perspective of the project contract risks. The Washington State DOT ATC policy states âThe ATC process . . . allows a certain level of control by the agency over potential risks contemplated by proposersâ (Carpenter 2012). The California DOTâs policy states, âThe purpose of the one-on-one meetings with Proposers is for Department to discuss issues and clarifications regarding the RFP and Proposerâs ATCsâ (California DOT 2008). This idea is further explored by West (2012), who found the ATC pro- cess to be âvaluable to both the owner and the proposers because the owner was able to gain an idea of what to expect from the bids while the proposers were able to gain a clear under- standing of the ownerâs requirements. The process was completely confidential, enabling the proposers to retain any advantages established.â Finally, 26% of the 99 RFPs reviewed in NCHRP Synthesis 455 specifically allowed clarifica- tions to be sought during ATC one-on-one meetings. NCHRP Synthesis 455 concluded âthat the practice of publishing all RFIs may not be beneficial to the highway construction project processâ and suggested that âagencies consider creating a mechanism to gain clarification of solicitation documents in a confidential mannerâ (Gransberg et al. 2014). 4.5.2 Contractual Modifications to the Baseline Design Once the ACM contract has been awarded, it may be necessary to modify the baseline scope to make the ATC a legal part of the contract. The process essentially involves developing the necessary documentation to codify the approved changes to the original contract, regardless of project delivery method. As such, each agency will want to develop its own process that is fully compliant with local statues. Such issues as changes to the baseline schedule to capture promised time savings, specifically make proposed betterments contract requirements, and documenting changes to the risk profile that result from approved ATCs are the types of specifics that would be included in the final project scope at award. 4.6 Amendments to the Solicitation Triggered by ATC Submittals The cornerstone of ATC confidentiality is to protect the content of each contractorâs submittal from exposure to its competitors. Nevertheless, there are occasions when an ATC submittal contains some information that will require the solicitation to be amended for all competitors. This type of information is usually gained during confidential one-on-one meet- ings as contractors make use of the opportunity to confidentially seek clarifications on project aspects that may not be part of the proposed ATC under consideration at the meeting. Most agencies reserve the right to amend the solicitation for all competitors if a proposed ATC or the discussions associated with its approval expose an error, mistake, or ambiguity in the bidding documents. Since the line between an ATC betterment/change and an error requiring an amended solicitation is not clear, each agency must refer to its own local statues and case law when drafting an appropriate clause for an ATC project. The following examples are offered to provide a basic understanding of the definitions of those issues that may trigger an amendment: â¢ Error: The dimensions on one plan sheet for a given feature of work are not the same as the dimensions shown on another plan sheet in detail related to the same feature of work.
Agency Review and Evaluation of ATCs 45 â¢ Mistake: The quantities of work for a pay item do not correlate to the quantities taken off the plans by orders of magnitude indicating a potentially missing decimal point. â¢ Ambiguity: The specifications require a given material to comply with a standard that is either out of date or no longer exists. A second triggering of an amendment can occur when an agency finds it must seek outside permission before it can approve an ATC. Essentially, this penetrates the cloak of procurement confidentiality, making it difficult for the agency to protect the proprietary information con- tained in the ATC. For example, the Florida and Washington DOTs interpret their prevailing statutes to require a solicitation amendment if an ATC requires a design exception or design variation as defined by FHWA. Both agencies allow the contractor that submitted the ATC to withdraw it to avoid unwanted public disclosure in this event. Lesson Learned: Contractors proposing on ATC projects need to recognize that the ATC process, with its confidential one-on-one meetings, creates a potential for identifying a need to amend the solicitation if an unforeseen problem with the baseline is uncovered. A mechanism to amend the solicitation documents in this event should be included in the instructions to bidders/proposers to warn them of the possibility that confidentiality may be breached if an error, omission, or ambiguity is identified.