Implementing Alternative Technical Concepts in All Types of Highway Project Delivery Methods (2020)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

C-1 Appendix C: Case Study Protocol and Questionnaire The key step in Task 2.1 was developing a case study protocol for the case study interviews and data collection plan. The protocol included a research synopsis of objectives, projects, field procedures that detail the logistical aspects of the investigation, interview questions, and documentation to collect and a format for documenting and analyzing the individual case studies (Eisenhardt 1989; Yin 1994). In addition, a plan will be developed for cross case comparisons to determine similarities and differences between cases (Eisenhardt 1989; Miles and Huberman 1994). The case study protocol permits the research team to conduct case studies separately in different parts of the country, while maintaining the reliability of the case study results. Internal validity is addressed by attending to multiple sources of evidence and the use of multiple case studies improves the external validity of the project delivery and project control tools that may be identified as promoting ATC project success. Case Study Protocol Background Information The following information is summarized from the Interim Report: Alternative contracting methods (ACM), such as Design-build (DB), Construction Management/General Contractor (CMGC), and Public-Private Partnerships (P3), have been proven to accelerate the construction, reconstruction, and rehabilitation of aging, structurally deficient infrastructure because they generally permit early contractor involvement during the procurement and in some cases, ACMs also allow construction to begin before the design is 100 percent complete. ACMs also allow the DOT to shift some of the responsibility for completing the necessary preconstruction tasks to support the design after the award of the DB, CMGC, or P3 contract. This creates a different risk profile than when the project owner has full responsibility for design in a traditional design-bid-build (DBB) project. ATCs augment the ability to increase integration in both DBB and ACM projects. Therefore, the primary thrust of the case study data collection effort is to document the effectives practices found in each DOT for implementing ATCs in each project delivery method. The following are the objectives of the case study research data collection plan. • To identify and compare the advantages and disadvantages of each ACM with respect to determining applicability and appropriateness to support the DOT ATC selection decision process. • The following are the specific objectives taken from the Phase 2 workplan that will be addressed by the data collected in the case study task: 1. Develop a decision-making tool that will guide project sponsors in determining when to incorporate ATCs into the entire range of ACMs and DBB to determine the potential benefits based on project characteristics, agency goals, and context in which the project will be delivered and operated over the project’s life cycle. 2. Obtain input on the ATC Toolkit (including the analytical tools and decision-making tool) from stakeholders.

C-2 3. Pilot test the ATC Toolkit on projects proposed in two DOTs with varying degrees of ATC experience to check the reasonableness of results from the Toolkit and thereafter incorporate any lessons learned into the final Toolkit. Relevant Definitions Across the highway construction and engineering industry, terms relating to quality often have multiple meanings that in some cases overlap with one another and in others supersede each other. To prevent confusion among several vital terms important to this study, the following definitions have been provided. These definitions are in accordance with the most recent issuance of the TRB Circular Glossary of Highway Quality Assurance Terms E-C137 and the NCHRP Synthesis 376.  Design-bid-build (DBB): A project delivery system in which the design is completed either by in-house professional engineering staff or a design consultant before the construction contract is advertised. [The design-bid-build method is sometimes referred to as the traditional method.]  Design-build (DB): A project delivery system in which both the design and the construction of the project are simultaneously awarded to a single entity. [The main advantage of the design– build method is that it can decrease project delivery time.]  Construction manager/general contractor (CMGC) Also called Construction manager-at- risk (CMR): A project delivery system that entails a commitment by the construction manager to deliver the project within a guaranteed maximum price (GMP), in most cases. The construction manager acts as consultant to the owner in the development and design phases and as the equivalent of a general contractor during the construction phase.  Public-private partnership (P3): A government service or private business venture that is funded and operated through a partnership of government and one or more private-sector companies. Statement of Purpose The primary research objectives for this project are as follows: • Document and categorize current practices and applications for selecting a given ATC for the general population of all project delivery methods. • Explore how highway construction projects of all project delivery methods are effectively applying ATCs. • Identify benefits and limitations of the approaches. • Explore how to implement and apply project performance metrics and evaluation methods for all methods of project delivery specific to ATCs. • Produce a decision-making tool to that will guide project sponsors in determining when to use ATCs on the entire range of project delivery methods. • Produce an ATC Toolkit, including decision-making and ATC performance evaluation tools. Field Procedures Project Researchers  Doug Gransberg, G&A

C-3  Mike Loulakis CPS  Keith Molenaar, CU  Jorge Rueda, AU  Ghada Gad, CPP  Debra Brisk DRB  Steve DeWitt, TIS Case study delegation  Gransberg – Georgia, Missouri, and Ohio, DOT  Rueda – Alabama and Minnesota DOT  Molenaar – Colorado and Washington DOT  Gad – California and Utah DOT  Brisk – Minnesota DOT  DeWitt – North Carolina DOT Case study project selection criteria In the interim report, it was stated that a concerted effort will be made to select case study projects from transportation agencies that have mature experience with at least two different project delivery methods. The following selection rubric was used to select the DOTs found in the previous section. • Mature ACM program – Have completed more than 10 DB projects with ATCs. • ACM program authorized to use more than one ACM with ATCs used on at least one. • ATC program has been institutionalized by the development of standard guidance in the form of manuals, guidebooks, policy documents, etc. containing ATC selection and/or evaluation methodologies. • Project performance data is available for both ACM and DBB projects on a program basis. Case study informant selection Once a case study DOT has been selected, several members of the team directly associated with the agency’s ACM selection decision process and if possible, the agency’s project performance measurement process will be contacted and asked for an appointment for an interview. Potential interviewees include the following:  Agency-level ACM office directors or equivalent for centralized DOTs.  District engineers and staff responsible for selecting ACMs for decentralized DOTs.  Project-level project managers, construction managers, design managers, etc.

C-4 Case Study Basic Data and Research Delegation Tables # Case Study Name Location Organization Contact (Information) 1 2 3 4 5 6 # Case Study Name Contacted? Lead Investigator Interview Type Interview Date Follow-up Date 1 2 3 4 5 6 7 8 # Case Study Name Materials/Documents Received 1 2 3 4 5 6 7 8 Case study field procedures A case study structured interview is not merely an oral survey. This research instrument is used to not only answer the what and where questions, but more importantly the why and how questions that cannot be reliably collected through standard survey research. The primary input to the case studies will be gathered through structured interviews with agency personnel, contractors, and consultants that have been part of teams involved with complex projects. The structured interview questionnaires will be developed on lines prescribed by the US Government Accounting Office (GAO 1991). The GAO method states that structured interviews can be used where “information must be obtained from program participants or members of a comparison group… or when essentially the same information must be obtained from numerous people for a multiple case-study evaluation” (GAO 1991). Both these conditions apply to this project therefore, the tool is appropriate for the research. The process involves developing a questionnaire that was made available to each interviewee prior to the interview and then collecting responses in the same order using the same questions for each interviewee. The information can be gathered by both face-to-face and telephonic interviews. Time is given per the GAO method to ensure that the interviewee understands each question and that the data collector understands the answer. Additionally, interviewees are also allowed to digress as desired, which allows the researchers to collect potentially valuable information that was not

C-5 originally contemplated. The output is used to present the agencies’ perspective on various points analyzed in the subsequent tasks. The interview process generally follows the below set of steps: 1. Once an appointment for the interview has been made, send the questionnaire to the point of contact with instructions to review it prior to the interview. Also request that the interviewee obtain access to any relevant documents and have them available at the interview. 2. Commence the interview with a short explanation of the project, the desired information, and a statement that nothing will be published before the interviewee has had an opportunity to review the draft material and correct it as required. 3. Next, explain the questioning process. You will ask a question and then ask the respondents if they understand it. If not, further explanation will be provided. After the answer is given and recorded, you will read it back to the interviewee and give them an opportunity to refine it if required. 4. The process then proceeds question by questions until it is complete. 5. It is particularly important to allow the interviewee to digress if the tangent appears to be of interest to the research. The questionnaire is not in and of itself complete and generally applicable to all agencies. Therefore, the local variations and subtleties will be found in the digressions from the questions. 6. Complete the interview by recapping the major findings that you have drawn from the interview and ensure that you have correctly interpreted them. 7. Be sensitive to local semantics/jargon and ask for clarifications even if you think you understand. Don’t assume that your past experiences are in any way reflective of how things work in the interviewed agency. Data Analysis The data collected will be synthesized and evaluated to produce the following output:  Advantages and disadvantages to each system from the agency’s point of view.  Identification of trends and common finding between the literature review, survey and case studies.  Triangulate the common findings from these three sources of data to arrive at valid conclusions.  Case studies will be summarized individually in the lens of the ATC selection decision model.  Using literature review and previous phone survey information, compare key attributes of the baseline approach to key attributes in the ACM models.  Individual findings will be analyzed across the cases using pattern matching techniques.  Comparison to baseline DBB project delivery approaches. Draft Individual Case Study Report Outline The following is an outline of the case study summary that each data collector will produce. 1. Agency: 2. Locations: 3. Average annual construction program: $

C-6 4. Average annual ACM program: $ 5. See Table ACM Approx Total # Approx Annual # Average Project Value System for Selection? Primary Reason for Selecting DBB CMGC DB PDB P3 6. See Table ACM Procurement Method Used (check all) Primary Reason for Selecting Procurement Method QBS Best Value Low Bid Other DBB CMGC DB PDB P3 7. Detailed description of agency ACM selection process. Include copies of any documents that may be available for this process. 8. Detailed description of agency ACM performance evaluation process. Include copies of any documents that may be available for this process. 9. Short discussion on interviewees perceptions of the following: a. Impact of each ATC on internal staffing requirements. b. Impact of each ATC on cost/budget certainty. c. Impact of each ATC on schedule certainty. d. Impact of each ATC on final constructed quality. e. Major disadvantage of ATCs for each ACM. 10. Summary – Brief synopsis of the case study that includes statements on the following topics: a. Assessment of the agency’s ATC experience, organizational maturity, and potential to be used as a model for the tool set. b. Agency ATC selection process, if any. c. Agency ATC project evaluation process, if any. d. Agency ATC tools, if any. e. Key findings that will lend authority to the tool set under development. f. Other key findings. 11. Observations of the Researchers – This section is reserved for comments that are important to understanding the case study agency but are not covered in the above report format. Items like specific innovations, anecdotal negative experiences, opinions formed by the interviewers, etc. will go in this section.

C-7 Post Collection Analysis The case study output will be analyzed to determine candidate effective practices and best practices using a rigorous method based on importance index theory, as published by Gransberg et al. (2017) in Transportation Research Record No. 2630 entitled: ‘A Framework for Objectively Determining Alternative Contracting Method Best Practices.’ First, the researchers will augment the set of ATC effective practices that were identified in NCHRP Synthesis 455: Alternative Technical Concepts for Contract Delivery Methods and augment it with ATC related practices found during the Task 1.1 Benchmarking effort to produce a list of candidate practices. Secondly, the candidate ATC practices will be sorted into one of three categories based on a breakdown found in a report sponsored by the Connecticut Academy of Science and Technology on project deliverability (Lownes et al. 2012). The documents were classified into the following categories: 1. Organizational structure 2. Project delivery method selection process 3. Contracting techniques Each given practice will then be evaluated to determine if it qualified as a candidate for classification as an effective practice using the definition of having been “identified through high quality quantitative study.” The last stage will impose an additional condition that the practice had to have been observed as used by more than one state DOT by using the survey results from NCHRP and SPR research reports or similar documents, compiling a comprehensive list of all synthesis survey respondents that reported using the given practice. Once the list of candidates is identified in each category, they will be ranked using a rubric termed the “Importance Index” (II) (Assaf and Al-Hejji, 2006). The II is a combination of the frequency at which a specific practice was observed in the content analysis of the literature and its influence measured by the number of state DOTs that have adopted the practice. As such, the II holds that practices that are used frequently and are of high influence are more important that low frequency, low influence practices. This permits an objective ranking of candidate effective practices, which can then be used to infer the relative importance of adopting a specific ATC effective practice. The II is derived by first computing a Frequency Index (FI) and an Adoption Index (AI) based on Equations 1 and 2 to furnish input in the II calculation shown in Equation 3: Frequency Index (FI) (%) = ∑(n/ N)*100/Tn [Eqn 1] Where: n = Number of observations of a practice in a specific category N = Total observations of all practices in a specific category Tn = Total observations of all practices in all categories Adoption Index (AI) (%) = ∑(d/ D)*100/Td [Eqn 2] Where: d = Number of DOTs using a practice in a specific category D = Total DOTs using all practices in a specific category Td = Total DOTs using all practices in all categories

C-8 Importance Index (II) (%) = (FI * AI) [Eqn 3] The result is a list of ranked candidate practices within each category as well as a second list of all practices ranked as a total population. This is done because each of the categories describes a separate facet of ATC implementation and it is important to understand the relative importance within each category. The overall ranking informs the analyst regarding those practices which when combined enhance the effectiveness of the ATC program. To test the criterion proposed by Accardo (2009) regarding “high quality quantitative study,” a Research Index (RI) and a Verification Index (VI) are computed using Equations 4 and 5 based on the Assaf and Al-Hejji’s (2006) II theory. Research Index (RI) (%) = ∑(c/ C)*100/Tc [Eqn 4] Where: c = Number of literature citations reporting a practice in a specific category C = Total literature citations using all practices in a specific category Tc = Total literature citations using all practices in all categories Verification Index (VI) (%) = (RI * I) [Eqn 5] Lastly, a criterion will be developed based on the Task 1.1 benchmarking results to differentiate a best practice from an effective practice. The TRR paper used the criterion proposed by Michaelson and Stacks (2011) that consists of two objective criteria, which permit the analyst to identify a best practice from a practice that a given author believes to be sound. Their definition is: “A method or technique that has consistently shown results superior to those achieved with other means, and that is used as a benchmark.” The terms “superior to other means” and “used as a benchmark” furnish a means to distinguish a best practice from all other practices when evaluating the content of a particular document. As one would expect to be both consistently superior and a benchmark is a pretty lofty standard and will not often be attained. The team will pilot the case study protocol to ensure that it will provide the desired information. In the pilot test, case study participants will also be asked to comment on the protocol, covering topics such as ease of understanding, ease of response, amount of required pre-interview preparation required, etc. The protocol will then be modified as required. Once the team has scheduled case studies with the DOT, consultant, and contractor organizations, the case study data collection will commence. The primary objective of these case studies is to build solid theory for each delivery method. This theory will be developed by conducting several forms of data collection, including objective project risk model characteristic measurement, subjective interviews, and project performance metrics. The goal for case study selection is to generate a cross section of cases that permits the analysis of the advantages and disadvantages of the various ATC models across the various project characteristics. To ensure that this goal is met, the following criteria will be placed on the case study selection.

C-9 • Completed projects delivered using DBB, CMGC, DB, P3 and other ACMs. • Agency solicitation documents are available for content analysis. • Financial and schedule progress data available. • Design consultant and construction contractor available for interviews • Agency information – construction volume, outsourcing of design, seasonal issues, and so forth. • Alternative project delivery experience data – number, type, cost/schedule performance, project-specific legal issues, claims, protests, and so forth. The case study data collection plan will include the procedures for assembling the necessary input data to calculate metrics that can be used to identify potential effective practices for use in the Task 2.2 draft guidebook. During each case study interview, the DOT will be asked to furnish quantitative data on the case study project. The focus will be on projects where ATCs resulted in documented benefits as well as projects where the ATC process was not considered successful. It is hoped that the outcomes can also be mapped to create the tools for the Task 2.6 implementation toolkit in addition to furnishing definitive guidance that can be provided in the final guidebook. Upon completion of the case studies, the team will reduce and analyze the case study data to identify trends and disconnects, gaps in the body of knowledge, needs for contract clause guidance, examples of successful practices, lessons learned and candidate best practices Figure C.1 is a flow chart showing the logic used in the research methodology for gleaning both best and effective practices from the case study data.

C-10 Figure C. 1. Best Practice Determination Flow Chart NCHRP Project Survey Output DOT ACM Policy Documents Academic ACM Literature Identify ACM Effective Practice Candidates Content Analysis Content Analysis Content Analysis ACM Practices Reported in Surveys ACM Practices Formally Adopted ACM Practices Evaluated by Research Candidate Set of Effective Practices Compute Frequency Index Compute Adoption Index Compute Research Index Compute Importance Index Compute Verification Index Compute Average Composite Rank Ranked Best Practices Candidates Rank by Importance Index Ranked Effective Practices by Category Ranked Effective Practices by Population Rank by Verification Index Ranked Effective Practices by Category Ranked Effective Practices by Population Meets “Best” Criteria? Effective Practice List Best Practice ListYES NO

C-11 Case Study Interview Questionnaire Case study DOTs and rationale State Level PDM Rationale Remarks California Program DB/CMGC/ P3 Recent DB ATC program Georgia Program DB/P3 P3 ATC program Minnesota Program DB/ CMGC Pre-approved Element (PAE) program (risk management tool) Missouri Program DBB/DB DBB ATC program + Conceptual ATC (CATC) Washington Program DB Long standing DB ATC program Single Project Case Studies Alabama Project DBB DBB ATC for the first time Colorado Project DB Alternative Configuration Concept (ACC) North Carolina Project P3 P3 ATC Ohio Project DB Geotechnical DB ATC (risk management tool) Rhode Island Project DB Limited scope utility ATC DB Unable to arrange interview with project team members Utah Project CMGC Proposed Technical Concept (PTC) CMGC NOTE: Program level case studies will also conduct a project-level case study.

C-12 Case Study Project Structured Interview INTRODUCTION/BACKGROUND: The purpose of this questionnaire is to identify state highway agency effective policies and procedures for delivering construction projects using alternative technical concepts (ATCs). The objective of this research is to produce a practical guidebook in AASHTO standard format that presents effective practices for establishing and implementing ATCs in project delivery methods. Its specific focus is on the specific policies and contractual content used during procurements that include ATCs as well as the benefits and/or costs associated with the use of ATCs. DEFINITIONS: The following definitions are used in conjunction with this questionnaire: • Alternative technical concepts (ATC): A procedure where the designers and/or contractors are asked to furnish alternative design solutions for features of work designated by the agency in its procurement documents. The proposed changes must be equal to or better than the baseline design contained in the solicitation documents. • ATC project: A project delivered using any project delivery method that includes ATCs as part of the pre-award process. • Design-bid-build (DBB): A project delivery method where the design is completed either by in-house professional engineering staff or a design consultant before the construction contract is advertised. Also called the “traditional method.” • Construction Manager/General Contractor (CMGC): A project delivery method where the contractor is selected during design and furnishes preconstruction services. Also called CM-at-Risk. • Design-build (DB): A project delivery method where both the design and the construction of the project are simultaneously awarded to a single entity. • Public-Private Partnership (P3): A project delivery methods where private funding is involved. There are many variations. • Best Value: An award method that utilizes cost and other factors to select the winning bidders. Examples are cost-plus-time bidding, qualifications, design approach, etc. • One-on-One meetings: Confidential meetings between the agency and individual entities that are competing for the same project whose purpose is to propose, review, and approve/disapprove ATCs before the entity’s bid or proposal is submitted. • Open records act: A statutory requirement to disclose all information pertaining to public project upon request by a member of the public. Also termed “Sunshine Law.”

C-13 I. Agency and Interviewee General Information: 1. DOT: 2. On average, how many projects using the ATC process does your agency deliver each year? In the past year, approximately how many projects allowed for the submission of ATCs? On average, how many contractors submit ATCs for each project? On average, how many ATCs were submitted by each proposer? What percentage of ATCs are approved? Rough estimate is acceptable. 3. What project delivery methods does your organization use? For which of them does your organization use ATC provisions? Check all that apply. Project Delivery Method Typical Project ATC Project DBB CM-at-Risk or CMGC DB Other, please specify Procurement Method Low Bid Best Qualified Best Value Other, please specify II. Agency ATC Practices – General Information 4. Does your agency have a manual or solicitation document that specifically describes the procedures to be used with projects using the ATC process? Yes No 5. How does your agency define an ATC? 6. Which of the following reasons motivates the use of ATC provisions in contracts awarded by your agency? Check all that apply. Which of the below is the single most significant reason for the use of ATC provisions? (Interviewer circle the check box) Reduce project duration Increase quality Encourage innovation Project size Increase agency control over budget Reduce preconstruction costs Reduce risk related to contractors poor performance Optimize use of agency resources Third party issues (permits, utilities, etc.) Low complexity of scope of work High complexity of scope of work Reduce agency staffing requirements Transfer design liability to contractor The confidentiality of pre-proposal communications with contractors The greater integration between the agency and the contractors The ability to identify issues in the solicitation documents before awarding the contract Other

C-14 7. Which of the below policies or procedures apply to a project delivery method that uses ATCs? (Check all that apply) DBB CMGC DB Doesn’t apply The scope of what can be submitted as an ATC is limited. The features of work where changes from ATCs may be proposed is specified. Confidential one-on-one meetings are held. The contractor may choose to include or not include any of its approved ATCs in its proposal. ATCs submitted for approval must include an estimate/statement of cost impact. ATCs submitted for approval must include an estimate/statement of schedule impact. ATCs submitted for approval must include a statement of quality impact. Cost estimates from ATCs are reviewed by an independent cost estimator. Stipends are paid and permit the agency to use ATCs proposed by entities other than the winner. Stipends are paid and do NOT permit the agency to use ATCs proposed by entities other than the winner. ATCs are reviewed and approved by the project evaluation, selection, or award panel. ATCs are reviewed and approved by personnel other than the project evaluation, selection, and/or award panel. If required, the agency can refer an ATC to a third party for technical review. The number of ATCs submitted by a single entity is limited. If the ATC is a design change, the contractor must prove that it has been reviewed by an engineer licensed in the agency’s state. Design concepts/standards/specifications from other states/agencies are permissible. ATCs can be used to propose changes to the general provisions to the contract. ATCs can be used to propose changes to the special provisions to the contract. Use of formal risk allocation techniques to draft contract provisions regarding ATCs 8. How does your local open records act impact the use of ATCs by your agency? No impact, we are able to adequately protect the confidentiality of the content of all ATCs submitted regardless if they come from a winning or losing proposer. Minimal impact, we are able to adequately protect the confidentiality of the content of all ATCs submitted throughout the procurement process and only the contents of ATCs that are included in the winning proposal are exposed to the public record after award. Minimal impact, we are able to adequately protect the confidentiality of the content of all ATCs submitted throughout the procurement process and only the contents of ATCs that are included in the winning proposal plus those from losing proposers who accept a stipend are exposed to the public record after award. Major impact, all ATCs submitted as parts of the procurement are exposed to the public record after award. Major impact, all ATCs submitted as parts of the procurement are at risk of being exposed to the public record before award. Our open records act makes it functionally impossible to reasonably consider ATCs in a manner that is fair and equitable to our industry partners. 9. Have you ever received a Freedom of Information Act (FOIA) request from a contractor in order to obtain information about approved ATCs submitted by other bidders? Yes No Don’t know If yes; how did you respond to the FOIA request(s)?

C-15 10. Rate the following factors based on their importance to the success of the ATC during the procurement process. Factor Essential Important Not Important No Opinion Confidential one-on-one meetings Confidentiality of pre-proposal communications between agency and contractors on matters other than ATCs Ability to safeguard ATCs containing proprietary content Ability to guarantee ATC confidentially Offering a stipend to use ideas from losing entities. Ability to grant design criteria variances Excluding ATCs that would exceed permitting constraints Allowing ATCs that would exceed permitting constraints Ability to measure the benefits upon project completion Independent technical review of ATC designs Independent review of ATC cost estimates Agency buy-in to the ATC process Industry buy-in to the ATC process Incentives/disincentive schemes 11. How do you assign design liability to ATCs? Assignment DBB CMGC DB Agency retains liability Liability is shifted to contractor Liability is shifted to the contractor’s design consultant Other; please specify. III. Case Study Project - General Information: 12. Case Study Project Title: 13. Project Delivery Method: 14. Short Scope Description: 15. Expected contract duration: 16. Actual project duration: 17. Award amount: 18. Actual cost at project completion: 19. How was this project funded? State funds Federal funds State and Federal funds 20. General Composition of Project Scope: Road Construction Bridge Construction/Replacement

C-16 Road Repair Bridge Repair Road Routine Maintenance Bridge Routine Maintenance Hazardous waste treatment, mitigation, removal Landscaping Erosion Control/Stormwater Mitigation Environmental Mitigation Bike lanes, sidewalks, transportation enhancement ADA-Related Improvements Traffic Signalization Drainage Maintenance Rest Area Improvements Pavement Markings Roadway Safety Improvement/Maintenance Other: {explain} A. Does your agency limit the use of ATC provisions to this type project? Yes No 21. Number of proposers and ATCs How many contractors submitted proposals on this project? How Many contractors submitted proposals with approved ATCs? How many proposals were submitted for review? How many proposals were approved? 22. Do you believe the number of proposals submitted for this project was impacted by the use ATC provisions? Yes No If yes, explain: IV. Case Study – Procurement Process 23. How did you advertise and award the case study contract? Advertise/Award Method Invitation for Bids (IFB), full open competition, low bid IFB, competition restricted to prequalified entities, low bid 1-step full open competition RFQ, QBS, no price competition 1-step full open competition RFP, includes qualifications, technical, and price 2-step full open competition, RFQ/RFP 1-step competition restricted to prequalified entities RFQ, QBS, no price competition 1-step competition restricted to prequalified entities RFP, includes qualifications, technical, and price 2-step competition restricted to prequalified entities, RFQ/RFP Sole source Other, please specify 24. Refer to question 23. Is this advertise/award approach the usual approach used when allowing ATCs with other project delivery methods? Yes No Don’t know If no, explain why it is different: 25. Refer to question 23. Is this advertise/award approach the usual approach used in non- ATC contracts executed under the same project delivery method? Yes No Don’t know If no, explain why it is different: 26. Did you develop a shortlist for this project

C-17 Yes No Don’t know If yes; how many contractors were in the shortlist: 27. What is the disposition of an approved ATC? Check all that apply. Approved ATCs remain confidential through award of final contract. Approved ATCs of winning contractor are revealed upon award. Approved ATCs from losing contractors are revealed upon award. Approved ATCs from losing contractors remain confidential after award. Approved ATC triggers an addendum to the solicitation to all competitors Other please describe: 28. Have you had a protest of an awarded contract related to the ATC process? Yes No If yes, what was the basis of the protest? Please describe: 29. Refer to question 28. What was the result of the protest? Protest was upheld Protest was overturned Protest was dropped Comments? 30. Please check all of the applicable features that describe the process you use for you confidential one-on-one meetings that involve ATCs. One-on-one Meeting Features One or more one-on-one meetings are required for all competing contractors One or more one-on-one meetings are optional for all competing contractors– agency’s decision One or more one-on-one meetings are optional for all competing contractors– contractor’s decision Agency members at the meeting are from the evaluation panel Agency members at the meeting are not necessarily from the evaluation panel – relevant agency’s staff One-on-one meetings are held before submitting the Initial ATC One-on-one meetings are held after submitting the Initial ATC One-on-one meetings are held at the moment of submitting the Initial ATC V. Case Study – Payment Provisions 31. What type of compensation method was used in this project? Was the compensation method the same for the ATC and non-ATC work? ATC Non-ATC Other: Unit Price Lump Sum Cost-Reimbursable Other: 32. Refer to question 31. Is this the usual compensation approach used when allowing ATCs with other project delivery methods? Yes No Don’t know If no, explain why it is different:

C-18 33. Do cost estimates for proposed ATCs include a deductive amount for the contract scope that is being replaced? Yes, always Yes, sometimes* No VI. Case Study – ATC Impact on other Contract Provisions 34. Do your ATCs that promise time savings trigger liquidated damages if the revised completion is not met? Yes, always Yes, sometimes No 35. Have you adjusted your typical incentive/disincentive provisions for this project due to the use of ATC provisions? Yes No Don’t know If yes, explain why and how: 36. Is the QA/QC scheme that you used for this contract different from the one used on non- ATC project executed under the same project delivery method? Yes No Don’t know If yes, describe the differences: VII. Case Study – ATC Submission and Evaluation 37. What information must be submitted by contractors with their ATCs? (If possible, the interviewer should copy this information from the solicitation documents) a) Description: A description and conceptual drawings as needed of the configuration of the ATC or other appropriate descriptive information, including, if appropriate, product details and a traffic operational analysis; b) Usage: The locations where and an explanation of how the ATC would be used on the Project; c) Deviations: References to requirements of the RFP which are inconsistent with the proposed ATC, an explanation of the nature of the deviations from the requirements and a request for approval of such deviations along with suggested changes to the requirements of the RFP which would allow the alternative proposal; d) Analysis: An analysis justifying use of the ATC and why the deviation, if any, from the requirements of the RFP should be allowed; e) Impacts: A preliminary analysis of potential impacts on vehicular traffic (both during and after construction), environmental impacts, community impacts, safety, and impacts on the cost of repair, maintenance, and operation; f) Risks: A description of added risks to the Department or third parties associated with implementation of the ATC; g) Quality: A description of how the ATC is equal or better in quality and performance than the requirements of the RFP; h) Operations: Any changes in operation requirements associated with the ATC, including ease of operations; i) Maintenance: Any changes in maintenance requirements associated with the ATC, including ease of maintenance; j) Anticipated Life: Any changes in the anticipated life of the item comprising the ATC;

C-19 k) Engineer’s Certification: The Firm’s Engineer of Record shall provide a written certification signed and sealed confirming the renderings being submitted can be designed and constructed as depicted. 38. How are ATCs evaluated by the review panel to determine if they are “equal to or better” than the original design/requirements included in the solicitation documents? (Check all that apply) ATC Evaluation Approach Based only on the professional judgement and expert opinions of ATC reviewers Based on a standard criteria used in all ATC projects Base on a criteria developed specifically for this project Evaluated on an objective manner – using a formal methodology Using a weighted criteria Other: VIII. Case Study - ATC’s Cost and Time Implications 39. Have you assessed the cost implications or savings on this project due to the use of ATCs? Yes No If yes, how much did the project cost increase/decrease: 40. Have you assessed the schedule implications or savings on this project due to the use of ATCs? Yes No If yes, how much did the project schedule increase/decrease: IX. Complementary Information 41. What has been the biggest challenge with implementing ATCs in your agency? 42. What would you do differently in the next contract with ATCs? 43. In your opinion, have ATCs impacted positively contracting procedures in your agency? Yes No If yes, explain how: 44. Do you have anything else you would like to share regarding the procurement procedures on your projects using the ATC process? Major Takeaways and Interviewer’s Observations/Comments: