Performance-Based Construction Contractor Prequalification (2009)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

SUMMARY PERFORMANCE-BASED CONSTRUCTION CONTRACTOR PREQUALIFICATION Performance-based contractor prequalification goes beyond the financial prequalification provided by the surety industry when it issues a bond in conjunction with a public trans- portation project and includes a contractor’s performance record in the prequalification process. A contractor with a marginal track record but the same level of financial assets will receive roughly the same bonding capacity, and hence the same opportunity to bid, as another contractor with a record of exemplary performance. This creates a situation in which the agency is subsidizing marginal performance, which in turn reduces the incentive for top performers to continue to superior performance. Therefore, a contractor prequalifi- cation system that directly rewards good past performance and encourages poor perform- ers to improve is needed to rectify this unintended consequence. The objective of this synthesis is to identify and synthesize current contractor perfor- mance-based prequalification practices based on construction quality, timely performance, safety record, and other criteria. Its focus is on traditional design-bid-build projects, but because the evaluation of qualifications is an important aspect of alternative project delivery methods, it also looks at state departments of transportation (DOT’s) experience in design- build and construction manager/general contractor projects as well. Finally, it indentifies those systems that have effectively been implemented in a manner that furnishes an incen- tive for good contractor performance, while influencing marginal contractors to improve their performance to remain competitive in the industry, and adds value to the completed construction project. In addition to a rigorous literature review, the synthesis is based on new data from a sur- vey, a set of structured interviews, case studies, and two content analyses. A general survey on performance-based contractor prequalification provided 52 responses from 41 U.S. state DOTs and seven Canadian provincial ministries of transportation. A content analysis of administrative prequalification documents from 43 U.S. states was conducted. Another content analysis of request for qualifications documents from 107 sets of performance- based prequalification request for qualifications (62 transportation projects and 45 non- transportation projects) from 27 states and two Canadian provinces also was conducted. For further verification, structured interviews with 10 construction contractors from nine states and one Canadian province were conducted to obtain the industry perspective on this subject. Finally, four case studies from different states were conducted to furnish spe- cific information on performance-based contractor prequalification systems that had been implemented. The synthesis’ conclusions covered the gamut of contractor prequalification issues from the current administrative forms to post-project evaluation. They are as follows: The survey results and the prequalification form content analysis found that many • factors used in performance-based prequalification are contained in the current administrative prequalification process. Therefore, the transition from a system of administrative prequalification only to performance-based prequalification could be accomplished smoothly.

2 Bonding did not carry the weight in both administrative and performance-based • prequalification processes as the other factors listed in this research. Three of the anal- yses looked at how often bonding, sureties, and insurances were required and found that they occurred on average less than 10% of the time. This is probably because agencies require these instruments to be submitted at project award. These findings contradict the “conventional wisdom” found in the literature that bonding capacity can be substituted for prequalification. In fact, the majority of U.S. DOTs and Canadian ministries of transportation (23 of 41 and 6 of 7, respectively) stated that they do not believe that a performance bond is an adequate substitute for prequalification. Thus, the synthesis concludes that the “soft” factors related to managerial competence and past performance are more important to the prequalification process than the “hard” aspects related to bonding and financial status. Additionally, bonding and financial factors are best restricted to general administrative prequalification and later as proj- ect-specific prequalification factors applied at the time of contract award (see conclu- sion 5 for details). A rigorous post-project contractor performance evaluation system can replace many • of the commonly used minor performance-based prequalification factors and thereby simplify the process. Many of the current performance-based contractor prequalification programs result in • an adjustment to a contractor’s bidding capacity. These seek to create a disincentive to marginal work by reducing the total amount of work a contractor can compete for if its performance is not satisfactory. Ontario has replaced performance bonds with a rigorous form of performance-based contractor prequalification that saves the prov- ince about $70 million Canadian dollars (CDN) annually on bond costs. This leads to the conclusion that a system could be devised to reward contractors with good perfor- mance records by adjusting or reducing the percentage of the contract amount that they have to bond. This adjustment would give these contractors an incremental financial edge over marginal contractors that would be required to bid the costs of fully bonding the project. The next conclusion is a synthesis of the entire project that found that performance-• based contractor prequalification essentially consists of a three-tiered system. The first tier is primarily financial and mirrors the current administrative prequalification systems found across the nation in a simplified form. The second tier is performance based and includes post-project contractor evaluations, and the final tier consists of project-specific prequalification. The last tier primarily will apply to projects being delivered by alternative methods such as design-build and construction manager and general contractor, but also could be used on traditional design-bid-build projects for which specific contractor technical qualifications and past project experience are keys to project success. The final tier (i.e., short-listing) needs to be included as an option in the prequalification system to ensure that an agency’s prequalification program can be applied uniformly to all its projects without regard to project delivery method. There are few actual barriers to implementing performance-based contractor prequali-• fication. Many highway agencies have some form of contractor performance evalu- ation in their system. Many have some form of performance-based prequalification as well. The perspective of the contractors interviewed for this report was receptive to implementing this system, because it reduces the number of marginally qualified contractors against which they must compete. Thus, it appears that both the highway agencies and their industry partners will benefit from this change if some of the minor administrative hurdles and perceived barriers to implementation found in chapter four can be eliminated or surmounted.

3 A content analysis of administrative prequalification docu- ments from 43 U.S. states was conducted. Another content analysis of request for qualifications (RFQ) documents from 107 sets of performance-based prequalification RFQs (62 transportation projects and 45 nontransportation proj- ects) from 27 states and two Canadian provinces was also conducted. For further verification, structured interviews with 10 construction contractors from nine states and one Canadian province were conducted to obtain the industry perspective on this subject. Finally, four case studies from different states were conducted to furnish specific informa- tion on performance-based contractor prequalification sys- tems that had been implemented. PERFORMANCE-BASED PREQUALIFICATION BACKGROUND NCHRP has produced two research reports in recent years that deal with this topic. The first is NCHRP Web Docu- ment 38: Quality-Based Performance Rating of Contrac- tors for Prequalification and Bidding Purposes (Minchin and Smith 2001). This is one of the more recent research studies of this issue. The second is NCHRP Report 561: Best Value Procurement Methods for Highway Construc- tion Projects (Scott et al. 2006). Although the latter report focused on project delivery methods, it contained a large amount of information on project-specific contractor prequalification. Review of NCHRP Web Document 38 The Minchin and Smith 2001 report details the development of a numerical model to measure contractor past performance and integrate it into the bidding scheme through the calcu- lation of a “contractor quality factor.” This factor is deter- mined based on a numerical analysis of past performance. It is implemented through the A+C bidding scheme autho- rized in FHWA Special Experimental Program 14 (FHWA 1998b). Table 1 shows how this system is implemented by subtracting a value associated with the “contractor quality factor” (i.e., the “C” factor in A+C) from the actual bid price to create a “modified bid” upon which the award decision is made by awarding it to the lowest modified bid. In this example, the high bidder has the low modified bid as a result of receiving a high quality score. CHAPTER ONE INTRODUCTION INTRODUCTION The subject of construction contractor prequalification based on performance is one that evokes two opposing reactions within the transportation industry. On one side, the argument revolves around ensuring that only competent constructors with a track record of quality work are given contracts to do more work with precious public funds. The other side’s argument revolves around the inherent subjectivity of the process in which the definition of “qualified” does not spring from the inherently objective comparison of bid price. It also argues that a potential “Catch-22” exists whenever a con- tractor must have specific experience to be deemed qualified and, as a result, will never be able to become qualified for specific types or classes of work. All state departments of transportation (DOTs) have some form of prequalification in the sense that administrative procedures must be satisfied before a contractor can be included on the agency’s bidders list. This report will call such a procedure “administrative prequalification” to differentiate it from the subject of this synthesis, performance-based prequalification. SYNTHESIS OBJECTIVE The objective of this synthesis is to identify and synthe- size current contractor performance-based prequalification practices based on construction quality, timely performance, safety record, and other criteria. Its focus is on traditional design-bid-build (DBB) projects, but because the evaluation of qualifications is such an important aspect of alternative project delivery methods, it also looks at DOT experience in design-build (DB) and construction manager/general contractor (CM/GC) projects as well. Finally, it aims to find those systems that have been implemented effectively in a manner that furnishes an incentive for good contractor per- formance, while influencing marginal contractors to improve their performance to remain competitive in the industry, and that adds value to the completed construction project. In addition to a rigorous literature review, the synthesis is based on new data from a survey, a set of structured inter- views, four case studies, and two content analyses. A general survey on performance-based contractor prequalification provided 52 responses from 41 U.S. state DOTs and seven Canadian provincial ministries of transportation (MOTs).

4 best value metric is calculated” (Gransberg and Molenaar 2003). The proliferation of award algorithms leads to con- fusion in the industry. Confusion about how to win leads to reduced competition and increases the likelihood of bid protest (Parvin 2000). The issue is noted at this early point in the synthesis as a warning about the potential problems of implementing performance-based contractor prequalifi- cation systems that might create a false sense of objectivity through formulaic methodologies such as the one proposed in the NCHRP Web Document 38. Review of NCHRP Report 561 The second major report was NCHRP Report 561: Best Value Procurement Methods for Highway Construction Projects (Scott et al. 2006). Although the focus of this study was much broader looking at numerous ways to award a highway proj- ect on a basis of other than price alone, it delved thoroughly into the use of qualifications and past performance, includ- ing both administrative and performance-based prequalifi- cation as one part of the procurement practice as evidenced by the following quotation: Best-value procurement methods allow various elements to be considered in selecting a contractor on the basis of performance. Objective elements include contractor experience with similar projects, completion within schedule, compliance with material and workmanship requirements, timeliness and accuracy of submittals, and record of safety. Subjective elements include effective management of subcontractors, proactive measures to mitigate impacts to adjacent properties and businesses, training and employee development programs, corporate commitment to achieving customer satisfaction, and client TABLE 1 ExAMPLE OF A+C BIDDING USING THE CONTRACTOR QUALITY FACTOR Contractor Bid Amount Quality Points $/Quality Point “C” Factor Modified Bid A $2,175,000 91 $10,000 $910,000 $1,265,000 B $2,200,000 88 $10,000 $880,000 $1,320,000 C $2,225,000 97 $10,000 $970,000 $1,255,000 Source: Minchin and Smith (2001). The issue with this system is essentially the derivation of the value of a well-qualified contractor to a given project. NCHRP Web Document 38 from which Table 1 is taken does not reveal how the value of a quality point was established in this example, and a search of the literature for a rigorous method to develop this value came up empty as well. This leads to the inference that the number was set arbitrarily using a measure of professional judgment as to how impor- tant this factor was in relation to the estimated cost of the example project. In Table 1, the value is set at $10,000 per quality point. If this value had been set at $8,300 per quality point (as shown in Table 2), Contractor A, the low bidder based on price alone, would have won even though they were less qualified than Contractor C. Thus, the system is shown to be sensitive to the value placed on the quality points. A 17% change in this value gives the project to the low bid- der and essentially wastes the time and effort by both the DOT and the industry competitors to prepare, submit, and evaluate qualifications in this procurement. Nevertheless, a method similar to this one has been used successfully by at least one DOT, Arizona (Arizona DOT 1997), to procure a large DB project. TABLE 2 SENSITIVITY ANALYSIS OF MINCHIN AND SMITH’S CONTRACTOR QUALITY FACTOR Contractor Bid Amount Modified Bid At $8,300/pt Bid % from low bid Modified Bid % Difference from low bidder @ $10,000/pt Modified Bid % Difference from low bidder @ $8,300/pt A $2,175,000 $1,419,700 — — — B $2,200,000 $1,469,600 1.15% 4.35% 3.51% C $2,225,000 $1,419,900 2.30% −0.79%* 0.01% *A negative value indicates the project will be awarded to this contractor rather than the low bidder. This issue of sensitivity to arbitrarily assigned numbers and weights is detailed in a 2003 paper on best value award algorithms, which is the formula by which DOTs calculate a number similar to the “modified bid” in Table 1. This study evaluated four DOT best value award algorithms and one used by FHWA and found that “when given the same set of input, the effect of the various selection methods is pronounced … every responsive bidder [in Table 3] can be selected as the winning proposal depending on how the

5 using performance-based qualification factors in the selec- tion process, the DOT can filter out unqualified contractors, thereby increasing the probability that the project will be completed successfully (Gransberg and Ellicott 1996). How- ever, the key to successful “public sector application of qual- ifications parameters in a bid is the use of these parameters in the selection process, and their application must be justifi- able and defensible” (Scott et al. 2006). This speaks to the concern expressed by Parvin (2000) with regard to reducing the probability of bid protest by making the performance- based prequalification system transparent and easy to under- stand. A case study of how the Minnesota DOT defended its prequalification method for a DB project is presented in chapter five and validates the idea that prequalification parameters must be both “justifiable and defensible” as cited by Scott et al. (2006). Therefore, in light of the above discussion, this report will proceed with the following two guiding principles for evalu- ating potential performance-based contractor prequalifica- tion systems: The specific elements of a performance-based contrac-• tor prequalification system add value to the project in terms of reducing performance risk. The elements of performance-based contractor prequal-• ification system are justifiable and defensible. relations. These elements not only affect the ultimate performance and overall cost of the completed facility, but also contribute to the efficient execution of the work. Efficiency is very important to contracting authorities that are interested in a high level of public acceptance. It is also recognized that, because of constrained staffing and budgets, it is not possible for state agencies to “inspect” quality into the work. Therefore, a procurement process is needed that considers value-related elements in awarding contracts (Scott et al. 2006). [Emphasis added.] TABLE 3 ExAMPLE OF BEST VALUE SELECTION WITH FIVE TYPICAL AGENCIES Firm Technical Score Time Price Proposal6 Indiana DOT Low Bid, Fully Qualified1 Arizona DOT Best Value with Quality Credit2 South Caro- lina DOT Low Composite Score3 Washington State DOT High Best Value Score4 FHWA Best Value5 A 92 450 $11,880,000 $11,880,000 $10,573,200 129,130 77.44 *63.10 B 86 460 10,950,000 10,950,000 10,074,000 *127,326 *78.54 62.73 C 76 500 9,850,000 9,850,000 *9,554,500 129,605 77.16 59.14 D 74 500 9,760,000 *9,760,000 9,564,800 NR 75.82 57.99 E 68 500 *9,700,000 NR 9,700,000 NR 70.10 53.54 Source: Gransberg and Molenaar (2003). NR = not responsive. *Winning proposal. 1Fully qualified: Technical score > 70. 2Arizona DOT quality credit calculated similar to Table 1.1 “C” factor. 3See equation (2) in reference; Technical Score < 75. 4See equation (5) in reference. 5See equation (6) in reference. 6Winning proposal in low bid selection. The term “value-related elements” will be an underlying foundation of this synthesis. As will be discussed later in this report, the members of the construction industry are not opposed to performance-based prequalification as long as the effort they must invest to participate adds value to the process and does not become a tedious recitation of facts and information that does not help the DOT differentiate the best qualified contractors in the pool. That finding from the contractor structured interviews intersects with the above quote and validates the idea that, for performance-based prequalification to be generally accepted by both the high- way agency and industry, the process must demonstrate that it adds value to the completed project by reducing the risks associated with awarding the project to a marginally quali- fied constructor. State agencies often use general past performance and experience criteria in their administrative prequalification procedures to admit a contractor to the state bidders list. By

6 This form essentially shifts administrative prequalification to a higher level by adding in performance-based prequalifi- cation factors that apply to all projects regardless of type. The disadvantage to this form lies in its broadness, not providing a mechanism to tie qualifications to specific project require- ments. Thus, it has an increased potential that a contractor can prequalify but still be marginally qualified to perform a given type or class of work, thus reducing the value added to the procurement process. Project-specific performance-based prequalification eliminates this issue by making each factor project related and filtering out those contractors whose past experience, while generally acceptable, does not show them to be well qualified for a given project’s technical or other requirements. Thus, the major disadvantage of this form is the additional administrative burden that is placed on DOTs to develop a unique set of project-specific prequalification criteria and on the industry to submit a unique statement of qualifications (SOQ) for every project. Theoretically, the above two forms can be overlapped to form a hybrid combination of both. Figure 1 illustrates this prequalification universe graphically. Thus, it is possible to divide performance-based prequalification into two phases: (1) phase one would entail a general prequalification based on factors that would be applicable to all projects; and (2) phase two would include one or more project-specific factors. This approach would reduce the administrative effort by using an annual general prequalification based on performance that admits a contractor to the bidders list and then follows up with an RFQ for specific projects for which the owner believes that specific factors are necessary to move a set of contractors from the overall bidders list to a project-specific short-list. In other words, the short-list would consist of the most qualified contractors for a specific project drawn from the pool of qualified contractors based on project-specific qualification factors. An example of this approach could be a seismic retrofit project for which past experience with the retrofit technology specified for the job would greatly facili- tate timely completion and enhance quality assurance. Public works authorities in New South Wales (NSW), Australia, use a version of the hybrid system that they call the “two envelope” method (New South Wales Depart- ment of Commerce 2007). In this system, contractors are prequalified on a performance basis. If necessary, project- specific qualification requirements are then published for a given project. Two envelopes are submitted at the bid tender, with the first containing the project-specific SOQ and the second containing the bid proposal. Envelope 1 is opened and all contractors that meet the minimum project-specific qualification requirements are assigned to what amounts to a “short-list.” Those that do not make the short-list have their second envelope returned unopened. Then the second enve- lope for each contractor on the short-list is opened and the project is awarded to the lowest bidder. This is similar to the system used by the Indiana DOT called “low bid/fully KEY DEFINITIONS The report will use a number of terms-of-art in a precise sense throughout its entirety. To gain a full understanding of the meaning of this study, readers must understand the specific definition of each of these terms. The report first explains the difference between admin- istrative prequalification (the information submitted for inclusion on an agency-approved bidders list) and perfor- mance-based prequalification (the focus of this synthesis). Project-specific prequalification is also important in this report. The definitions for the three terms are as follows: Administrative prequalification: A set of procedures • and accompanying forms and documentation that must be submitted for inclusion on an agency’s approved bidders list. These forms may include an evaluation of financial statements, the dollar amount of work remain- ing under contract, available equipment and personnel, and previous work experience. This information may be provided on a project-by-project basis or on a speci- fied periodic basis, such as annually. Performance-based prequalification: A set of proce-• dures and backup documents that must be followed by a construction contractor to qualify to submit a bid on a construction project based on quality, past perfor- mance, safety, specialized technical capability, proj- ect-specific work experience, key personnel, and other factors. This information may be provided on a project- by-project basis or on a specified periodic basis. The project could be delivered using traditional DBB or alternative project delivery methods such as DB, CM/ GC, or any other method. Project-specific prequalification: Contractor prequali-• fication requirements that exist only for a single project. These requirements normally address project techni- cal and procurement factors that are essential for the success of the given project. They may include criteria that require the contractor to have had past experience building a certain technology (i.e., seismic retrofit, intelligent transportation systems, etc.) or a given proj- ect delivery method such as DB. The requirements may extend to cover specific experience for key project per- sonnel and specific types of plant and equipment. Performance-based prequalification can take two forms. First, it can be a general prequalification that allows a given contractor to bid on all projects. This form is the major focus of this synthesis. In its second form, it is a project-specific prequalification in that the process applies only to a given project and the contractor may resubmit its qualifications to bid on a different project. Each form has its advantages and disadvantages. The major advantage of general performance- based prequalification is the reduction in administrative effort expended both by the agency and the construction industry.

7 relating to issuance of bonds for public works projects. It also may include other factors such as demonstrated ability to perform a certain type of work. Whether by prequalification or other methods, public owners are increasingly exploring ways to include non-price factors, both qualitative and quantitative, in the procurement process to motivate contractors not only to improve their performance during construction, but equally as important, to build value into the end products of construction (Scott et al. 2006). Once again, the idea of using prequalification to add value to the construction process is expressed. Additionally, the idea of using performance-based prequalification as a means to “motivate” contractors to “improve their performance during construction” is articulated, and tying these two ideas together becomes the objective for accruing the benefits of enhanced construction quality and reduced administrative burden. The NSW prequalification manual, which calls the process the “scheme,” describes these benefits as follows: Allows the NSW Government as a major buyer of con-• struction-related services to more effectively implement continuous improvement initiatives in the construction industry to achieve better project outcomes; and Results in • significantly reduced tender assessment times and simplified contract administration because prequal- ified tenderers [bidders] have already demonstrated an understanding of and compliance with NSW Government construction industry benchmarks, with management procedures and systems requirements; and In line with the NSW Government’s direction to do • business with the best of the private sector, the Scheme provides for incentives for good performance and also for the application of restrictions or sanctions in the event of poor performance as measured against the respective scheme requirements (New South Wales Department of Commerce 2007). [Emphasis added.] Thus, it appears that the benefits of performance-based contractor prequalification have been recorded both in the United States and overseas. Next, the motivations for imple- menting these programs will be explored in the literature. Motivations for Developing and Implementing Performance-Based Prequalification NCHRP Web Document 38 (Minchin and Smith 2001) essentially categorized the motivations for implementing a performance-based contractor prequalification program in two areas. The first had to do with “frustrations” felt by both owners and construction contractors, which are described as follows: Public owners generally treat low-quality construction • work no differently than high-quality construction work. qualified” to award DB contracts (IDOT 1998). The pre- viously described A+C bidding described in NCHRP Web Document 38 (Minchin and Smith 2001) also falls into this hybrid category in the performance-based prequalification universe. General Performance-Based Prequalification Financial factors• Insurance factors• Bond factors• Past timely • completion factors Safety record • factors Others as • appropriate Hybrid Prequalification Two Envelope • Method A+C Bidding• Project-Specific Performance-Based Prequalification Technical factors• Past project factors• Key personnel • factors Plant/equipment • factors Others as • appropriate FIGURE 1 Performance-based contractor prequalification universe. LITERATURE REVIEW The literature on this subject is both extensive and rich, showing the relative importance of this area in the transpor- tation industry. It is obvious from the literature review that all authors without exception believe that the qualifications of a given contractor can have a marked impact on the success of the projects it builds. In fact, one paper states that the most qualified contractor “correlates to the lowest administrative burden” for the agency (Molenaar and Songer 1998), imply- ing that a well-qualified contractor requires less oversight and can be trusted to comply with contract requirements such as contractor quality control (QC). Another author goes on to justify prequalification by saying that “because of con- strained staffing and budgets, it is not possible for state agen- cies to ‘inspect’ quality into the work” (Scott et al. 2006). The same author provides a succinct definition and motiva- tion for establishing a thoughtful prequalification process. Prequalification in its simplest form is an assessment of financial responsibility, which often mirrors what sureties look for in making their underwriting decisions

8 these project delivery methods return quality equal to or better than that obtained by the traditional methods (Ernzen and Feeney 2002). Thus, the motivation for implementing performance- based contractor prequalification is twofold. First, it fur- nishes a vehicle to reward good performance and, second, it satisfies a need to ensure that a better qualified contractor with a record of good performance is entrusted with a greater deal of autonomy in the quality management process. Thus, in both cases, the agency is properly discharging its respon- sibility to the traveling public to deliver a quality project with the public dollar. To accomplish this purpose, the program must have all the necessary components to collect contractor performance data, reduce that data in a meaningful manner, and utilize the performance output in the prequalification decision-making system. Components of a Performance-Based Prequalification Program NCHRP Synthesis Report 190 (Thomas and Smith 1994) found that DOTs rely on the following four strategies to qualify construction contractors to bid: Prequalification: contractor must be qualified before it • can submit a bid Postqualification: only the lowest responsive bidder is • required to submit qualifications to prove that it is also a “responsible” bidder Performance bonding: reliance on the surety industry • to identify qualified contractors Contractor licensing: state-sponsored program to • ensure that only qualified contractors can bid based on licensing requirements This report focuses on the first strategy of prequalifica- tion. The literature review found that most performance- based contractor prequalification programs consisted of the same set of components (Russell et al. 1992; Al-Gobali and Bubshait 1996; Minchin and Smith 2001; Ernzen and Feeney 2002; Hancher and Lambert 2002; McLawhorn 2002; Scott et al. 2006; Norman-Eady 2007): A questionnaire and application furnished by the con-• tractor that detailed the following information: Financial data – Available equipment and plant – Construction experience for a specified period – Names and backgrounds of key personnel – Classes and types of work for which qualification – was requested A formula or algorithm to convert financial data into • a rated capacity that establishes the maximum amount of work a given contractor can be awarded in a given period Public owners indirectly reward poor workmanship by • not penalizing poor workmanship, thus giving a bid- ding edge to those contractors that consistently per- form poorly. Administrative prequalification merely establishes • a benchmark for financial capacity, not technical capability. Reliance on performance bonding does not protect the • public owner from marginally competent contractors that have a strong financial foundation. Many of these frustrations spring from the public agen- cy’s requirements to ensure free and open competition and to avoid unnecessary delays to much needed transporta- tion projects resulting from bid protests. Prequalification is inherently a reduction in the level of free and open competi- tion. Therefore, these programs must be well-designed and avoid an appearance of being arbitrary. The Delaware Code furnishes that state’s DOT with the authority to prequalify construction contractors (Delaware Code 2001) and cites 10 specific reasons why a contractor can be found unquali- fied to bid. Two of these reasons, “inadequate experience to undertake the project” and “documented failure to per- form on prior public or private construction contracts,” fall into the performance-based prequalification realm. Neither, however, would apply to a marginally qualified contractor that had not been directly penalized for poor workmanship as expressed by NCHRP Web Document 38. The NCHRP report also details a second more timely motivation for implementing performance-based contractor prequalification. This motivation deals with the movement toward alternative project delivery methods and a greater reliance on contractor QC. The report describes this motiva- tion in the following terms: Changes in regulations regarding use of contractor quality testing in quality assurance decisions and continuing reduction in DOT personnel will increase the need for “quality driven” contractors in public transportation construction projects. This change, coupled with more departments adopting performance-based and performance-related specifications, places more need on contractors to know and use quality management in their field operations management. With more contractors providing the quality control function, the DOTs’ role would change to a quality assurance role. As one part of the quality assurance process, there is a need for comprehensive methods to evaluate a contractor’s eligibility to engage in work from a quality perspective (Minchin and Smith 2001). Another author expressed the same sentiments in a paper focused on contractor-led quality control: As state highway agencies move further in this direction, it is incumbent on them to first plan carefully during the procurement phase to ensure that they choose qualified teams. They must then draft contracts and specifications that put sufficient checks and balances in place so that

9 Several authors conducted research to determine the rela- tive importance of the various components listed above. One early study involved a survey of construction professionals from both owner and contractor organizations and asked them to rank order 20 prequalification factors by importance. An intersection of the two groups found that both rated “financial stability,” “past project performance,” and “personnel avail- ability and experience” as the “key decision variables relevant for a generic contractor prequalification knowledge base” (Russell et al. 1992). The NCHRP Web Document 38 study found that “project management/control skills,” “personnel experience,” “quality of final project,” and “experience with project type” were the most important for a similar group of survey respondents on essentially the same topic (Minchin and Smith 2001). When the two studies are combined, the two input components (contractor’s questionnaire/application and contractor project performance evaluation) are covered by these topics, thus validating those program components based on independent research. The remainder of the information gleaned from the lit- erature will be reported in the subsequent chapters as it A contractor project performance evaluation system • that usually revolves around ratings assigned on a spe- cific standard form A formula or algorithm to adjust the rated capacity • based on the accumulated record of project perfor- mance evaluations An appeals process for a contractor that believes it has • been unfairly or improperly rated ! FIGURE 2 Kentucky Department of Highways performance-based contractor prequalification process (Hancher and Lambert 2002). Figure 2, taken from a paper by Hancher and Lambert (2002), details the Kentucky Department of Highways qual- ity-based contractor prequalification process. This program is indeed a performance-based system as it contains all of the previously cited components. From looking at the flow chart, it is evident that this program connects the calculation of total maximum financial capacity and performance evalu- ation by developing an “annual eligibility rating,” which is used to adjust the contractor’s capacity to what is termed its “maximum eligibility amount.” This value equals the amount of work a contractor may be awarded in the given year, and as a result, “lower-quality work will reduce the allowable work volume, whereas high-quality work will increase the allowable work volume” (Hancher and Lambert 2002).

10 deals with the details of various performance-based con- tractor prequalification programs and their effectiveness. At this point, the literature review has demonstrated both the motivation for and the benefits of implementing perfor- mance-based contractor prequalification. It also furnished the framework against which the remainder of this synthesis report will be structured. SYNTHESIS METHODOLOGY This report is the result of an intersection between a compre- hensive literature review, a national survey of both U.S. and Canadian public highway agencies, structured interviews with construction contractors, and formal content analyses of DOT administrative prequalification forms and project- specific RFQs. Additionally, case studies of DOT experience with performance-based prequalification were conducted to validate the results of the research. This methodology allowed the authors to not only collect information on per- formance-based contractor prequalification policies and pro- cedures across the nation by means of the standard survey but also confirm those findings through a rigorous analysis of both types of prequalification documents. The literature allows the findings from the other research instruments to be put in a global context to identify trends and similarities and capture state-of-the-art practices in the more general topic of applying qualifications evaluation techniques used in alter- native project delivery methods for traditional DBB projects. The triangulation of these three methods allows emerging and commonly used practices in this area to be identified. Before describing the details of the research methodology, the relative importance of the various research instruments must be understood. As performance-based prequalification is still relatively new to much of the U.S. transportation indus- try and only a handful of states have significant experience, this study went beyond the typical synthesis literature review and survey to conduct two content analyses of administrative and performance-based prequalification documents. These analyses developed lines of converging information with the literature review and the survey responses by furnishing a quantitative analysis of how DOTs are actually applying prequalification to the project delivery process. This quan- titative analysis provided valuable insight into the various ways in which public agencies are using this tool on both their traditional and alternative delivery projects. This study gives the greatest weight to the output from the general survey of highway agencies and the contractor structured interviews. The administrative prequalification form and RFQ content analysis were supporting lines of information. Finally, the case studies validated the conclusion, as appropriate, because they provided examples of how U.S. and Canadian highway agencies have implemented performance-based contractor prequalification, and they validated the recommendations for future research made in the final chapter. Research Instruments The synthesis employed the following major research instruments: Content analysis of U.S. state administrative prequali-• fication forms Content analysis of U.S. and Canadian project-specific • RFQs Survey of U.S. state and Canadian province transpor-• tation agencies Structured interviews of U.S. and Canadian construc-• tion contractors Case study analysis of U.S. and Canadian performance-• based contractor prequalification programs The structure and content of each of the instruments was developed to integrate with all other instruments, allowing the researcher to map the output of each instrument to the corresponding outputs and to identify data trends. Formal Content Analyses Two of the research instruments used in this synthesis consisted of content analyses of state DOT administrative prequalification documents and project-specific prequalifi- cation RFQ documents. These content analyses consisted of gathering and reviewing solicitation documents and searching for the requirements for qualifications that were outlined in the documents. The first formal content analy- sis furnishes quantitative measurements of current DOT requirements for prequalification factors. These measure- ments are calculated by counting the number of times that prequalification terms of interest are required by contrac- tors for admission to the state bidders list. This type of analysis can be used to develop “valid inferences from a message, written or visual, using a set of procedures” (Neuendorf 2002). The primary approach is to develop a set of standard categories into which words that appear in the text of a written document, in this case a prequalifi- cation form, can be placed; the method then utilizes the frequency of their appearance as a means to infer the con- tent of the document (Weber 1985). Thus, in this study, the content analysis consisted of two stages. First, all instances in which the words associated with contractor performance were found in each document, and their context, were recorded. Second, that context was used to determine, if possible, whether the information was important to the agency in a given context. This allowed an inference to be made regarding the given owner’s approach to administra- tive prequalification. When the results are accumulated for the entire population, trends can be identified and reported. This method was then repeated with other terms, such as past experience and financial capacity that were common to prequalification systems, and the context was recorded and then analyzed.

11 Project Management Plans• Safety Record/Plan• A matrix was developed from the content analysis output containing key prequalification concepts and practices. As the project literature review and survey progressed, further review was necessary for topics that had not been identified in the original content analysis. The information gathered was reduced to general categories that are detailed in chapter two of this report. The content analysis output was further combined with the results of the survey for this project and the literature that was reviewed to create the synthesis. Project-Specific Prequalification RFQ Content Analysis In addition to the analysis of administrative prequalification documents, the study also sought to identify state or prov- ince project-specific performance-based prequalification policies and guidelines that were currently available. This effort was done to provide a comparison between general performance-based prequalification and project-specific prequalification. Consequently, 107 sets of performance- based prequalification RFQs (62 transportation projects and 45 nontransportation projects) from 25 states, the District of Columbia, and two Canadian provinces were assembled (see the geographic distribution of these projects in Figure 4). The process described for the administrative prequalifica- tion document content analysis was used to derive the con- tent of each of those documents. The intersection between the output from this content analysis and the administra- tive prequalification documents permitted the researcher to determine a relative level of importance for similar factors in the two different systems. This process validated the con- clusions and recommendations for future research listed in the final chapter. General Survey of Public Highway Agencies In addition to the content analysis, a survey was issued to the state construction engineers in U.S. DOTs and Cana- dian MOTs (see Appendix A for details). A total of 45 complete and seven partial responses were received. Sur- vey responses were received from 41 states (see Figure 5). Responses from seven provincial MOTs are also included. This analysis separated the U.S. and Canadian responses to account for the difference in the construction contract- ing regulatory environment that exists in both countries and also to highlight potential innovative Canadian prequalifi- cation practices, keeping them from being lost in the total survey population. This process was repeated for the formal content analy- sis of the project-specific RFQ documents. The output from the two content analyses can then be compared to determine how prequalification policy is being implemented in the project-specific solicitation documents. The output also can be compared with the responses from the survey and struc- tured interviews, which will be discussed in detail later in this chapter, to map respondents’ output against their respec- tive state policy and solicitation documents. The use of these instruments in conjunction with the comprehensive review of the literature allows the researcher to not only maintain a high level of technical rigor in the research but also follow Yin’s three principles in the process of research data collec- tion: using multiple sources, creating a database, and main- taining a chain of evidence (Yin 2004). During this effort, the researcher was careful to remem- ber that single sources provide limited data based on “one specific source” and that such a limitation can create dif- ficulty when drawing results, in addition to a lack of “trust- worthiness and accuracy” (Yin 2004). Multiple sources alleviate lack of trust, increase viability, and frequently pro- vide supplementary realms of thought and research, which strengthens results. Administrative Prequalification Content Analysis The prequalification forms from 43 state DOTs were col- lected and analyzed (see Figure 3). The primary method was to categorize the various requirements listed on each form into a standard set that then could be used to compare the results of this content analysis with the results of the project-specific RFQ prequalification content analysis. The categories were taken from NCHRP Report 561 (Scott et al. 2006), because this report had completed a content analysis of 50 case study projects and displayed the results in a given format. Selecting this framework allowed the researcher to intersect the synthesis content analyses with the literature and draw conclusions. The NCHRP Report 561 categories are shown here. The first three factors were lumped under the single category of “prequalification” in NCHRP Report 561. Because this effort necessarily requires more detail, the categories are broken out for this analysis. Financial/Insurance• Plant/Equipment• Legal Record• Past Project Performance• Key Personnel Experience• Subcontractor Information•

12 ! FIGURE 3 Administrative prequalification content analysis (shading indicates states whose forms were analyzed). ! FIGURE 4 Project-specific requests for qualifications (RFQ) content analysis distribution.

13 “information must be obtained from program participants or members of a comparison group … or when essentially the same information must be obtained from numerous people for a multiple case-study evaluation” (Litkowski 1991). Both of these conditions apply to this synthesis; therefore, the tool is appropriate for the research. The process involved (1) developing a questionnaire that was given to each interviewee during the interview and (2) collecting responses in the same order (the same questions were used for each interviewee). The information was gath- ered using face-to-face and telephonic interviews. Time was given per the GAO method to ensure that the interviewee understood each question and that the data collector under- stood the answer. Additionally, interviewers and intervie- wees were allowed to digress as desired, which allowed the researcher to collect potentially valuable information that was not originally contemplated. The output of the analy- ! FIGURE 5 General survey responses—Respondents shown in shaded maps. Contractor Structured Interviews The ability to gauge the perception of the construction indus- try on this particular topic was essential for this study. Indus- try members include the business owners who are directly affected by any change to the requirements for bidding on public works contracts. Additionally, the construction indus- try has an enormous amount of political clout and can influ- ence public highway agencies’ ability to make changes of this nature. Nine U.S. and one Canadian contractor were interviewed (see Table 4 for details). They represented both large and small companies as well as local, regional, and national operations. The regional and national contractors had experience with more than one highway agency. The structured interview outlines were developed along simi- lar lines to the method prescribed by the U.S. Government Accounting Office (GAO) (Litkowski 1991). The GAO method states that structured interviews can be used when

14 with performance-based prequalification. Three agencies (Florida, Michigan, and Ontario) volunteered to furnish case study information. In addition to the agency case study interviews, the researcher synopsized a case study of a legal challenge to project-specific performance-based prequali- fication that was successfully defended by the Minnesota DOT and found in the literature. The case studies were collected using Yin’s methodol- ogy and the previously stated three principles of case study research data collection (Yin 2004). Therefore, the infor- mation gleaned from the case studies is coupled with infor- mation collected in the survey and the literature review to validate any conclusion drawn from the case studies. The case study information was gathered by both face-to-face and telephonic interviews. Protocol to Develop Conclusions and Recommendations for Future Research The major factor in developing a conclusion was the inter- section of trends found in two or more research instruments. The intersection of more than two lines of converging infor- mation adds authority to the given conclusion. Additionally, greater authority was ascribed to information developed from the general survey of highway agencies and the con- tractor structured interviews. The administrative prequali- fication form and RFQ content analysis were considered to be supporting lines of information. Finally, the case studies were used to validate the conclusion as appropriate, because they provided examples of how U.S. and Canadian highway agencies have implemented performance-based contractor prequalification. Recommendations for future research were developed based on the effective practices described in the literature and confirmed as effective by one of the research instru- ments but generally not widely used. Gaps in the body of knowledge found in this study were used to define the areas in which more research would be valuable. The conclusions and recommendations for future research are combined to form the performance-based contractor prequalification framework discussed in chapter six. sis is used to present the contractor’s perspective on various points discussed in the subsequent chapters. TABLE 4 CONTRACTOR STRUCTURED INTERVIEW DETAILS Location Type Work Annual Volume Experi- ence with Perfor- mance- Based Prequalifi- cation Support Perfor- mance- Based Prequalifi- cation Alberta Regional General Contractor <$250 million Yes Yes California National General Contractor >$500 million Yes Yes Colorado National General Contractor >$500 million Yes Yes Florida National General Contractor >$500 million Yes Yes Idaho National General Contractor >$500 million Yes Yes Michigan Local Microsur- facing Contractor <$20 million Yes Yes Missouri Regional Chip Seal Contractor <$100 million Yes Yes Nebraska National General Contractor >$500 million Yes Yes Oklahoma Local Pav- ing Contractor <$100 million Yes Yes Texas Regional General Contractor <$100 million Yes Yes Case Studies The surveys contained a question asking whether respon- dents would furnish case studies based on their experiences

15 CHAPTER TWO CONTRACTOR PREQUALIFICATION POLICIES AND PROCEDURES Additionally, the survey responses were deemed to be the most current and up-to-date information on the subject. The third analysis of survey responses pertains to the factors used in highway agency performance-based prequal- ification. This is the subject of this study and the most impor- tant analysis in the synthesis project. The output from this analysis can be compared with the output from the previous two analyses to determine those factors that are important on both administrative and performance-based prequalifica- tion programs. Additionally, the factors found in this analy- sis that were not found in the previous two analyses indicate those factors that are unique to performance-based contrac- tor prequalification. The results of the two survey output analyses are reported by splitting the U.S. and Canadian responses as separate pop- ulations. This is done for three reasons. The regulatory and legal environments in which highway construction is pro- cured are somewhat different in each country. Therefore, it is appropriate to keep the two nations separate in the analyses to ensure consistency in the results. Second, seven Canadian provinces and 41 U.S. states responded. If the two were com- bined as a single population, the Canadian responses would be statistically swamped by the U.S. responses and lost to the analyst. Finally, as will be seen in the chapter five case study analysis of the Ontario MOT’s prequalification program, it is important to view the Canadian results separately to identify potential Canadian practices that could be imported for use by U.S. DOTs. The final analysis consists of another content analysis of 107 sets of performance-based prequalification RFQs from 27 states and two Canadian provinces. The RFQs come from 62 transportation projects and 45 nontransportation (primar- ily buildings) projects. The nontransportation RFQs were included to allow the researcher to sample a parallel industry in search of potential ideas and trends that could be used in transportation projects. The output from this analysis comes from project-specific performance-based contractor prequal- ification and is used for comparison with the survey output for general performance-based prequalification to identify those factors that are unique to specific project concerns rather than to overall qualification. The survey also showed that, of the 41 U.S. respondents, 23 have authority to deliver projects using DB and 11 utilize public-private partnerships. INTRODUCTION This chapter will discuss four sets of analyses. The first is the formal content analysis that was done on the contractor administrative prequalification forms from 43 U.S. states. These forms represent the current minimum requirements for a contractor to be allowed to bid on projects in each state. Taking the population of forms as a single body of information, they show the various contractor qualification factors that are deemed important for bidding on transpor- tation construction projects. Although many of these forms include some performance-based factors, they are termed “administrative prequalification” and are taken together as a set to represent the status quo for prequalification. They effectively represent a benchmark against which the survey responses can be measured. Not all 50 states have a form- based system. The states of Alaska, Hawaii, Louisiana, and Mississippi require registration with the state and bonds in lieu of administrative prequalification, while Georgia, Maryland, and Minnesota require only bonds and have no separate registration. The second analysis comes from the general survey of highway agencies with regard to each agency’s administra- tive prequalification. In theory, the survey answers would be exactly the same as those found on the forms. This was not the case for several reasons. First, the prequalification-related terms-of-art are not standard across the nation. Although a concerted effort was made to use the most common terms and provide definitions when developing the survey ques- tionnaire, it was functionally impossible to address all pos- sible terms. Therefore, there is a limited mismatch between the survey and the terms used on each state’s prequalifica- tion forms and that issue probably accounts for some of the difference. Next, the survey responses reflected the current status of prequalification in each agency, and therefore, it is possible that a difference exists because the form found on the Internet at the DOT website may not have been the most current. This was true in a couple of cases in which the respondents indicated that their system was currently under modification and that their answers reflected current thinking. Finally, the questionnaire listed all the factors that were found across the United States, and it is possible that a respondent mistakenly checked a factor that was not on the agency’s current form. Nevertheless, the preponderance of the survey responses and the form output were in agreement.

16 TABLE 5 USE OF PROJECT-SPECIFIC ADMINISTRATIVE PREQUALIFICATION CRITERIA Project-Specific Prequali- fication Criteria No. of Responses (of 14) Monetary size 10 Technical complexity 6 Delivery method 6 Technical content 5 Traffic control issues 2 Environmental issues 1 3rd party issues 1 Quality assurance requirements 1 Location (urban vs. rural) 1 An additional five are using CM/GC project delivery. All of these alternative delivery methods utilize project-specific contractor prequalification. Therefore, to completely synthe- size the topic, project-specific prequalification requirements must be understood within the context of the general perfor- mance-based contractor prequalification program. PREQUALIFICATION REQUIREMENTS Before getting into the details of the analyses, it is instruc- tive to understand the requirements for prequalification as found by the survey. The survey asked four questions regard- ing agency requirements for administrative prequalification. Figure 6 shows the results of those questions. It shows that 35 states have prequalification requirements. Surprisingly, six states did not answer these questions, which may indicate that they either do not have a requirement or did not under- stand the question. FIGURE 6 U.S. state administrative contractor prequalification requirements. FIGURE 7 U.S. state performance-based contractor prequalification requirements.

17 Comparing the differences in the prequalification require- ments for the two systems in Figures 6 and 7, the trend toward providing project-specific reasons to use prequalification is found for those agencies that differentiate among their prequalification requirements on a project-by-project basis. TABLE 6 USE OF PROJECT-SPECIFIC PERFORMANCE-BASED PREQUALIFICATION CRITERIA Project-Specific Prequali- fication Criteria No. Responses (of 14) Delivery method 6 Technical complexity 5 Technical content 5 Monetary size 3 Traffic control issues 2 Location (urban vs. rural) 1 Environmental issues 1 QA requirements 1 3rd party issues 0 Contractor Administrative Prequalification Form Content Analysis A review of 43 prequalification application forms shows that contractor administrative prequalification is widely used by Next, the survey sought to determine whether or not administrative prequalification criteria were applied uni- formly across all projects. It found a 60% to 40% split between those that use the same criteria regardless of proj- ect class and those that have project class-specific criteria. Table 5 shows that those that differentiate by project class use project monetary size as the major factor, followed by technical complexity, project delivery method, and tech- nical content. The remaining options in the survey were found to be trivial. Looking at this result, it seems that for those agencies that do differentiate by project class in their prequalification process, the major factors seem to relate to qualifying contractors that have the competence to deal with specific technical issues on individual projects, the appropri- ate financial capacity for a given type of project, and that also have experience with a given project delivery method. Finally, the fact that 40% of the respondents choose to dif- ferentiate in the above manner indicates that they perceive some value in adding this next level of complexity to their contractor prequalification systems. The answers for the same questions as applied to perfor- mance-based contractor prequalification are summarized in Figure 7 and Table 6. The trend is reversed from the previous analysis with more agencies requiring performance-based prequalification for selected projects than generally for all projects. Additionally, twice as many agencies in this group use project-specific prequalification criteria than those that use the same criteria for all projects. The same top four prequalification criteria are cited, but in a different order of importance, which may portray a shift in philosophy from administrative to performance-based prequalification. FIGURE 8 Major administrative contractor prequalification form factors.

18 Financial criteria1. a. Type of business b. Financial statement Managerial criteria2. a. Key personnel experience b. Major violations c. Business connections d. Project failures Performance criteria3. a. Work classifications b. Construction experience c. Major project experience d. Available equipment Bonding, sureties, and insurance4. Financial Criteria The first major factor in financial criteria is “Type of Busi- ness,” and this determines whether the applicant is publicly or privately owned. This may seem benign, but it is actu- ally determining who can legally authorize a bid, looking for apparent conflicts of interest, and seeking to identify poten- tial legal issues with the business formation. The next major factor is “Financial Statements,” which covers a broad range of possible information. Some states ask for a simple account of last year’s receipts, whereas many other states require a full accounting of the company’s financial status performed by a certified public accountant. Managerial Criteria Managerial criteria contain four of the above major factors. “Key Personnel Experience” basically looks to see whether the people working directly on the projects have the req- uisite experiences to complete the projects for which their firms will be prequalified. The “Major Violations” deter- mines whether key principals in the organization have been previously disbarred from bidding and/or convicted of construction fraud, felony, or other crime. One can see that both deal with specific issues regarding the people who actually will be involved in completing the contracts for which the firm is prequalified. The “Business Connec- tions” factor is seeking to identify any conflicts of interest with subsidiaries and material suppliers. For example, con- struction companies may own trucking companies, asphalt plants, and quarries with which the state already may have a separate supply or service contract. These other contracts state DOTs. Typically, the practice is based on a standard form that asks generic questions such as follows: What kind of work classification are you requesting? • Do you have any experience with this type of work?• What is your company’s bonding limit? • All of these questions fall into some form of contractor administrative prequalification. This practice has worked for many years and has been a useful tool for all DOTs. Through the research survey, some trends were found in the data. After close review of each state’s administrative prequali- fication process, it was evident that certain key aspects held more importance toward prequalifying a construction con- tractor. As demonstrated in Figure 8, 10 factors of contractor administrative prequalification are specified in at least 22 of the 43 prequalification forms reviewed. These factors are as follows: Type of Business • Financial Statements • Work Classification • Business Connections • Construction Experience • Project Failures • Major Violations • Major Project Experience • Key Experience • Available Equipment • These 10 factors cover a broad range of information on a given construction company and show a distinct trend. One can see in Figure 8, that three general categories of informa- tion are sought: financial information, managerial informa- tion, and experience/performance information. Of the top five factors, “financial statement” deals directly with a con- tractor’s financial status; “project failures” and “key experi- ence” are related to the company’s managerial criteria; and “work classifications” and “major project experience” are related to the company’s performance criteria. Thus, it can be concluded that administrative prequalification consists of a combination of (1) current financial condition (that is, a contractor’s financial strength), (2) construction experience, and managerial ability (that is, a contractor’s technical com- petence), and (3) past performance. It seeks to identify those contractors with a record of defaulting on contracts, presum- ably to eliminate them from the agency-qualified bidders list as not “responsible” (i.e., a bad risk for future work). Criteria Categories All prequalification factors can be grouped in four catego- ries that are used as prequalification criteria. They are listed here for each of the 10 major factors categorized:

19 and performance criteria carries three times more weight than the sum of financial criteria and bonding, sureties, and insurance criteria, 76% to 24%, respectively. These two criteria deal with the same aspects used in the per- formance-based contractor prequalification programs dis- cussed in the literature review. This analysis allows one to map the results of the prequalification form content analy- sis directly to the results of the general survey responses for administrative prequalification, which is discussed in the next section. Administrative Prequalification Survey Analysis The general survey of U.S. states broke the prequalification process into two categories: administrative and performance based. The survey contained a series of questions regarding the specifics of each respondent’s administrative prequali- fication program. In short, administrative prequalification was defined as the process used to qualify a contractor to bid on a given agency’s construction projects. Figure 10 ranks the results of the responses for that survey and shows a clear stratification of administrative prequalification factors, of which the top eight factors will be deemed major. ! FIGURE 9 Administrative prequalification form criteria breakdown. These eight major factors can be broken into the same criteria as that used for the state prequalification form con- tent analysis: Performance Criteria1. a. Major project experience b. Available equipment c. Performance evaluations might appear to give that contractor an advantage. “Project Failures” requires the applicant to list all projects that they did not complete, usually by being held in default of the contract. Sometimes this factor extends to defaulted proj- ects that key individuals were involved with while working with other companies. Performance Criteria Performance criteria include factors that relate to the past project performance of the applicant and the ability of the applicant to perform on any upcoming projects. “Work Clas- sification” determines what type of construction a company is qualified to compete for and complete. This question is asked many different ways, however. Most states merely ask the applicant to complete a checklist of classes of construc- tion work. Others ask the applicant to list the type of work by state specification number. The “Construction Experience” question usually requests how long the applicant has been working in the construction field as a prime contractor and in some cases as a subcontractor. “Major Project Experience” allows the contractor to elaborate on key successes they may have had in the past or simply to list all the projects that they have completed within a given number of years. Some states extend the applicable experience to projects completed for other governmental agencies and private owners. Proj- ects cited in this section usually were constrained to those completed in the past three years. Most states asked the applicants to declare current ongoing workload at the time of their application. “Available Equipment” typically covers the construction plant and equipment the applicant owns and is available for use on agency projects. In some cases, it also includes equipment that is available to rent or lease. All four of these factors come together to give performance criteria major importance in the prequalification process. Bonding, Sureties, and Insurance Criteria Finally, bonding, sureties, and insurance criteria were shown to have a lower impact than would be expected. This may be for several reasons, but the chief one is that contractors are required to furnish bonds and insurance on a project-by- project basis. Thus, agencies that did not ask for this type of information on their forms may be assuming that they will receive these instruments individually as each project is awarded. When the 10 major factors are grouped into the four criteria along with the minor factors listed on the question- naire in Appendix A, the results shown in Figure 9 can be developed and the trend regarding the weights of each type criteria can be identified. The sum of managerial criteria

20 Finally, “Capacity Factor” is calculated as a function of the two previous factors to determine the company’s financial limitations for future projects. The last two factors in the managerial criteria category were discussed in the section “Contractor Administra- tive Prequalification Form Content Analysis,” but they are not used in the same context in this analysis. “Past Illegal Behavior” determines whether a contractor has a criminal record as well as looks for any construction-related legal issues such as fraudulent claims. “Technical Ability” seeks to determine whether the management team is experienced enough to effectively manage a construction project of a given magnitude in the requested work classification. Once again, bonding, sureties, and insurance did not include spe- cific criteria that could be classified as major factors. The minor factors in this category were statements from bond- ing, sureties, and insurance companies, and as shown in Fig- ure 10, those factors were reported to be used only by a small number of respondents. By grouping all the factors for administrative prequali- fication into the same four criteria as the prequalification form content analysis, Figure 11 shows a breakdown of how the factors fit into each of the four criteria. All of the major and minor factors are listed in the questionnaire contained in Appendix A. Once again, the sum of the managerial and per- formance criteria is greater than that of the other two catego- ries. Financial criteria carry more weight in this area than previously shown with the prequalification forms. Neverthe- less, when the results of the two analyses are taken together in FIGURE 10 Surveyed administrative prequalification factors ranked. Financial Criteria2. a. Financial capability b. Financial analysis c. Capacity factor Managerial Criteria3. a. Past illegal behavior b. Technical ability Bonding, Sureties, and Insurance4. The first three major performance criteria have been dis- cussed with the exception of “Performance Evaluations.” These evaluations are used by 27 of the 41 states reply- ing in the survey. This process rates how well a contrac- tor performed in various evaluated aspects on past projects. This is an important feature of performance-based con- tractor prequalification and will be discussed in detail in chapter three. The three major financial criteria are noticeably differ- ent than those factors found in the content analysis of the prequalification forms. However, these factors look at the contractor’s past, present, and future financial ability to complete a major construction project. “Financial Capabil- ity” describes the company’s finances at the present, and “Financial Analysis” dissects what a company has done in the past and calculates a factor of how they have fared.

21 Performance-Based Prequalification Survey Analysis After the administrative prequalification questions, the sur- vey asked a series of questions about performance-based prequalification. This was defined as the process required by a construction contractor to qualify to submit a bid on a construction project. Prequalification was based on quality, past performance, safety, specialized technical capability, project-specific work experience, key personnel, and other factors. The questionnaire listed 17 possible factors and gave the respondents the opportunity to write in factors of their own. Figure 12 shows the major factors of the performance- based prequalification. Major factors were defined as those selected by 50% or more of the respondents, which indicates that they play an important role in contractor performance- based prequalification. The major factors broken down into the four previously described criteria categories are as follows: Performance Criteria1. a. Major project experience b. Available equipment c. Quality and workmanship the context of performance-based prequalification, one can conclude that the existing agency systems for administra- tive prequalification already contain a significant amount of performance-based information. This leads to the inference that those agencies that want to move from administrative to performance-based prequalification will not have difficulty, because both the agency and the construction industry are familiar with submitting and evaluating information regard- ing past performance and current levels of experience and technical competence. FIGURE 11 Surveyed administrative prequalification criteria breakdown. FIGURE 12 Ranking of survey performance-based prequalification major factors.

22 Two of the above major factors have not yet been defined. Technical ability can best be described as the contractor’s expertise and experience in constructing technically com- plex projects. Managerial ability covers the contractor’s ability to manage the in-house resources, subcontractors, and project control tasks such as scheduling to the degree required to successfully complete a construction project. Again, bonding, sureties, and insurance criteria did not have any major factors. Some minor factors were related to bond- ing, sureties, and insurance, but as shown in Figure 13, those factors equated to only 12% of the total criteria breakdown. FIGURE 13 Surveyed performance-based prequalification criteria breakdown. Figure 13 shows that “Financial Criteria” and “Bond- ing, Sureties, Insurance” encompass only 25% of the per- formance-based prequalification process, whereas the sum of “Performance Criteria” and “Managerial Criteria” make up 75% of the performance-based prequalification process. This means that the performance-based prequalification pro- cess is heavily weighted to factors that relate to the agency’s experience with how well the contractor and its key person- nel have performed on past projects. Taking the combined results of the administrative prequalification form content analysis, the survey results for contractor administrative prequalification, and the survey results for contractor performance-based prequalification leads to a distinct trend. All three prequalification analy- ses assigned the most weight to factors in the managerial and performance criteria categories over the financial and bonding, surety, and insurance criteria categories. Some of the major factors were different in each analysis, but each system predominately seeks to measure the ability of a con- tractor to perform and manage a construction project using different approaches. The administrative prequalification form content analysis and the administrative prequalification survey both found Financial Criteria2. a. Financial capability Managerial Criteria3. a. Technical ability b. Past illegal behavior c. Key personnel experience d. Managerial ability Bonding, Sureties, and Insurance4. The three major performance criteria listed above con- sist of “Major Project Experience,” “Available Equipment,” and “Quality and Workmanship.” Of these three, two have been described in this chapter. Quality and workmanship was not a factor in the administrative prequalification pro- cess, but it is an important aspect in a performance-based contractor prequalification system. Quality and workman- ship incorporate the contractor’s ability to build what the highway agency specified as the final product. It speaks to the contractor’s ability to conform to contract require- ments as described by the plans and specifications. This factor is the crux of using contractor project performance evaluations in the prequalification process and is discussed in detail in chapter three. Financial criteria play a role in the performance-based prequalification process, but this role is less significant than the role they played in administrative prequalification. Only one major factor deals with financial criteria in the perfor- mance-based prequalification process, which is “Financial Capability.” In administrative prequalification, financial capability was defined as the condition of the contractor’s present finances. Financial capability in the performance- based prequalification context seeks to determine whether the contractor has the ability to execute a construction project without serious cash flow constraints. It can also be defined as the contractor’s fiscal responsibility in dealing with the state, its subcontractors, and the public. Managerial criteria play a large role in the performance- based prequalification process. Following are the four major factors: Past illegal behavior 1. Key personnel experience2. Technical ability 3. Managerial ability 4.

23 prequalification relying more heavily on financial criteria and bonding criteria, and requiring only basic managerial and performance criteria information to determine whether the contractor has the basic skills to handle a typical project. The performance-based prequalification relies more heav- ily on managerial and performance criteria to determine whether a contractor’s past record demonstrates its ability to adequately perform on a typical project. FIGURE 15 Canadian survey response breakdown for performance-based prequalification criteria. Comparing the U.S. and Canadian responses leads to the idea that administrative prequalification could be focused on the contractor’s financial health and fundamental business condition to measure its capacity to successfully enter into construction contracts rather than on contractor past per- formance. Performance-based prequalification then could be focused on a contractor’s record in successfully furnish- ing the technical skills, experience, and quality that would enhance the probability that it could finish a given project on schedule, within budget, and to the requisite level of constructed quality. This would create a two-step prequali- fication system that could measure the synergy between a construction company’s underlying financial strength and its commitment to satisfying its customer, the highway agency, through its actual performance. REQUEST FOR QUALIFICATIONS CONTENT ANALYSIS Project-specific performance-based contractor prequalifica- tion is commonly used in best-value project delivery methods such as DBB best-value (sometimes called A+C bidding), CM/GC, and DB (Scott et al. 2006). The major thrust in this form of prequalification is to ensure that the contractor and its key personnel have both the necessary specific technical experience and a track record of success building a specific type of project. Contractors often are required to pass the agency’s administrative prequalification process in addi- tion to submitting their project-specific qualifications for the given project. The content analysis of RFQs furnishes the factors related to the past, present, and future aspects of a contractor’s experience using a checklist approach. For instance, if the contractor has previously completed a given classification of work, it is found to be qualified without regard to the quality of that work. Both analyses showed that administrative prequalification is essentially an inventory of experience without regard for the quality of performance related to that experience. Thus, it would appear that a con- tractor with sufficient financial assets and marginal experi- ence or performance would be considered fully qualified. Nevertheless, the survey responses for contractor perfor- mance-based prequalification measured the same aspects of a contractor’s situation with a different set of priorities. This system adds the contractor’s evaluated performance record on top of the foundation established in the other two analy- ses. For instance, key personnel experience is a factor used to determine technical ability and managerial ability. Finally, although contractor bonding and insurance is included in all three of these analyses, it carries the least weight of all the factors. Analysis of Canadian Survey Responses When comparing the prequalification systems used in Can- ada with that of the systems portrayed by the U.S. survey respondents, some differences are noticed. For instance, by comparing the results in Figure 14 with those of its American counterpart (shown in Figure 11), the 28% change between these two figures demonstrates that the Canadians place more importance on the “Bonding, Sureties, Insurance” cri- teria. The Canadians also place less importance on manage- rial criteria. Nevertheless, all seven survey respondents use administrative prequalification. ! FIGURE 14 Canadian survey response breakdown for administrative prequalification criteria. Figure 15 shows that the breakdown of Canadian survey responses for performance-based prequalification criteria is similar to the U.S. breakdown shown in Figure 13. The Canadians place greater importance on the managerial crite- ria, however. This follows the pattern that would be expected in a dual prequalification process, with the administrative

24 FIGURE 17 Nontransportation project RFQ criteria breakdown. TABLE 7 REQUESTS FOR QUALIFICATION PREQUALIFICATION CONTENT ANALYSIS RESULTS Prequalification Factor Transportation Projects Non-Transportation Projects # RFQs % RFQs # RFQs % RFQs Past Performance 62 100.0% 45 100.0% Key Personnel Experience 56 90.3% 36 80.0% Management Plans 41 66.1% 34 75.6% Quality Plans 36 58.1% 24 53.3% Subcontracting Plan 34 54.8% 20 44.4% Current Workload 16 25.8% 16 35.6% Safety Record 30 48.4% 16 35.6% Financial Information 59 95.2% 45 100.0% Bonding, Sureties, Insurance 60 96.8% 45 100.0% CONTRACTOR PERSPECTIVE ON PERFORMANCE- BASED PREQUALIFICATION The contractors were asked to identify the prequalification factors that were used by the agencies with which they worked. Figure 18 illustrates the output from that analysis and is bro- ken down in the same manner as previous analyses. This constitutes one of Yin’s “lines of converging information” and validates the previous results by once again showing the study with a broader base of DOT experience than did the general survey. RFQs were found from 27 states and two Canadian provinces, whereas the survey found that only 12 states and four provinces used the broader form of perfor- mance-based contractor prequalification, which is the focus of this synthesis. Thus, the objective of this analysis is to look for trends in project-specific prequalification that can be applied to overall performance-based prequalification. The RFQ content analysis followed the same pattern as that used in the administrative prequalification form content analysis. Because of the inherent difference in the salient purposes of the two types of documents, not all the factors were identical. The RFQ prequalification factors were broken down into the nine major categories shown in Table 7, which can then be grouped into the four types of criteria used in the previous analyses (see Figures 16 and 17). Additionally, projects were split between transportation and nontranspor- tation to determine whether different prequalification fac- tors are used and to continue the search for applicable trends outside the transportation sector. FIGURE 16 Transportation project RFQ criteria breakdown. Once again the combination of performance and mana- gerial criteria outweighs the financial and bonding, sure- ties, and insurance criteria by a factor of roughly 2 to 1 in both types of project-specific prequalification projects. The transportation project financial and bonding criteria increased in weight from the performance-based prequali- fication criteria shown in the survey from 25% to 34% of the total weight, respectively. This is probably for the same reason cited in the performance-based prequalifica- tion survey analysis—that is, the agency expected to get bonding, surety, and insurance information with each proj- ect. Because the RFQs are project-specific, they are asking for this information as part of the evaluation of contractor qualifications for a given project.

25 preponderance of weight being assigned to performance and managerial criteria. Contractors were asked to identify those factors that they believed were most effective in encouraging well-qualified contractors to bid on projects for which per- formance-based contractor prequalification was in effect. Of 10 responses, the majority (one local, two regional, and four national) cited integrating past project performance evalua- tions into the prequalification process as the most effective practice. This was followed by key personnel experience and financial capability, both of which were each cited by two local, one regional, and two national contractors. FIGURE 18 Performance-based prequalification criteria breakdown from contractor interviews. Figure 18 shows that the contractor perspective is in line with the agency perspective on this subject. Some of the com- ments that were captured outside the structured interview questionnaire were enlightening. Essentially, all the inter- viewed contractors supported the idea of performance-based prequalification. Three contractors (one local, one regional, and one national) indicated that a well-qualified contractor with a proactive QC program cannot fairly compete against a marginally qualified contractor with a poor track record of quality performance. The well-qualified contractor can- not build the project as “cheaply” as the marginal contractor because the agency ultimately will accept substandard work because of the pressures to open the transportation facility to traffic as soon as possible. The interviewed contractors believe that marginally qualified contractors bid the project knowing that, ultimately, they will not have to achieve the specified level of quality. This perception is confirmed in the literature—for example, one study found that highway agencies “indirectly reward poor workmanship by not penal- izing poor workmanship, thus giving a bidding edge to those contractors who consistently perform poorly” (Minchin and Smith 2001). Therefore, they strongly support a system that will remove this disparity from the construction project delivery environment. SUMMARY From the previous discussion, it is evident that the five dif- ferent analyses yielded the same general trend. That trend is that managerial and past performance criteria carry greater weight in the contractor prequalification process than do financial and bonding criteria. The general survey asked whether the ability to furnish a performance bond constituted sufficient verification of a contractor’s qualification to suc- cessfully complete a construction project. The fact that four states only require bonds with no administrative prequali- fication form submission indicates that for those agencies the answer is yes. Additionally 17 respondents to the survey also answered yes to the same question. Thus, those agen- cies have a culture that is willing to trust the surety industry with regard to contractor qualification. Additionally, seven more respondents take that trust to a new level by being willing to accept a performance bond for less than 100% of the contract amount. When asked whether the agencies they deal with believe that the ability to bond a project con- stitutes sufficient evidence of contractor qualification, one local, one regional, and four national contractors answered yes. Remembering that these administrative systems have been in place for decades leads one to the conclusion that bonding capacity is definitely a tried-and-true indicator of contractor capability. This thought tends to conflict with the output from the majority of the other states, which clearly showed that past performance was important. Thus, those agencies with a culture of relying on the surety for de facto prequalification of construction contractors may indeed need to address this issue as a potential barrier to the implementa- tion of performance-based contractor prequalification.

26 CHAPTER THREE CONTRACTOR PERFORMANCE EVALUATIONS INTRODUCTION The motivation for evaluating contractor performance and then integrating those evaluations into the performance- based contractor prequalification system is simple. It pro- vides a tangible means to reward good contractors and a disincentive for marginal contractors. Other countries have taken performance evaluation to the next level and created performance contracts that serve as good models for North American highway agencies. The motivation for the evalua- tion of contractor performance in New Zealand is explained as follows: The concept of performance-based contracts originated from a consideration of four factors, namely, (a) the increasing lack of personnel within the national road departments …; (b) the frequency of claims …; (c) the need to focus more on customers’ satisfaction by seeking to identify the outcomes, products, or services that the road users expect to be delivered, and by monitoring and paying for those services on the basis of customer- based performance indicators; and (d) the need to shift greater responsibility to contractors throughout the entire contract period as well as to stimulate and profit from their innovative capabilities (Cabana, et al. 1999). [Emphasis added.] These four factors exist in the United States as well and support the use of contractor performance evaluation. A U.S. research study reports that “there are a number of issues associated with this criterion [past performance]… Careful consideration could therefore be given to a decision to use such a process to ensure that appropriate questions are asked and that the results are both fair to the contractor and useful to the owner” (Scott et al. 2006). In Canada, the Ontario Ministry of Transportation (MTO) implemented a contractor performance evaluation system in 2001 (Minchin and Smith 2001). This system is discussed in detail as a case study in chapter five. To summarize, MTO uses its program of performance appraisals and infraction reports for each project to establish an overall performance rating, which is maintained on a three-year moving average basis for con- tractor prequalification. Poor performance is penalized through the infraction system, which effectively reduces the amount of MTO work a marginal contractor can bid on at any given point in time (MTO 2004). Indeed, the evaluation system has proven itself to work so well that MTO no lon- ger requires prequalified contractors to furnish performance bonds, saving the province a significant amount of money each year (Minchin and Smith 2001). CONTRACTOR PERFORMANCE EVALUATION ISSUES Implementing a post-project contractor performance system can be controversial if the process has not been thoughtfully developed. One author describes the overall problem of con- tractor past performance evaluation like this: [Owners] must carefully consider how to implement it such that it is accurate and unbiased and could evaluate the pros and cons when making the decision to use past performance in the evaluation. Owners … may decide to address past performance by asking for evaluations for similar projects completed by the contractor in the recent past, often asking for specific data relating to schedule, cost, and claims performance on those specific projects. The use of these metrics can be controversial due to concerns relating to due process because the contractors do not have the opportunity to object to negative ratings and because of concerns regarding the validity of the information obtained (Scott et al. 2006). [Emphasis added.] Taking the reports by Minchin and Smith (2001) and Scott et al. (2006) together, essentially four major issues with post-project contractor performance evaluation must be addressed in a successful system: Accuracy of the rating• Fairness of the evaluation system• Consistency between raters and from project to • project Effective life of a given evaluation• Evaluation Accuracy The first issue deals with the agency’s ability to field an eval- uation system that accurately measures contractor perfor- mance. Being responsive to this issue will involve ensuring that measurable standards are established and published for those factors to be evaluated. Most highway agencies have some form of quality management program that has estab- lished standards that both agency and contractor personnel understand. These standards can be used in the evaluation system as long as no conflict develops that would be det-

27 than satisfactory performance records. Therefore, as the post-project evaluation scheme furnishes the input to that process, it is important that the methods used for evaluation are deemed to be fair and as objective as possible. The fairness issue basically deals with ensuring that the evaluation system is transparent and furnishing a mechanism for contractors to appeal a negative rating. Transparency means that the evaluation system and all its components are published in advance of the evalua- tion and that the agency applies them exactly as they are published (Molenaar and Johnson 2003). Furnishing an appeals process demonstrates to the contracting industry that the agency is open to challenges of its evaluation sys- tem by creating due process before a contractor is penal- ized by a negative rating (Scott et al. 2006). These two aspects will greatly ameliorate negative perceptions of the possible harmful impacts of a new contractor performance evaluation system (Molenaar and Johnson 2003). In fact, to enhance the appearance of fairness, the Kentucky Depart- ment of Highways (DOH) allows the contractors to rate the DOH’s performance on the same projects. These con- tractor ratings are used by the DOH to “determine qual- ity improvements needed, personnel training needed, and topics for discussion at the annual meetings with the con- tractor associations and for evaluations of personnel and other uses as deemed appropriate” (Hancher and Lambert 2002). Finally, one of the regional contractors that was interviewed indicated that the most important feature of a DOT contractor evaluation program is fairness, validating the ideas found in the literature. A last aspect of fairness is the ability of a contractor to appeal a negative rating that it deems to be inappropri- ate and get it changed or removed from the record. One research report speaks to the idea that fairness demands that a contractor have recourse to “due process” (Scott et al. 2006). Although a DOT does not want to create a special formal disputes resolution system for performance evalu- ations, it does need to provide a mechanism in its evalu- ation framework through which a contractor can protest what it believes to be an unfair assessment of its perfor- mance. This mechanism can be as simple as allowing the contractor to add rebuttal comments to the evaluation form and charging the chain of command above the evaluator to investigate and determine whether the contractor’s pro- test has merit before entering the final evaluation into the system. Seven of the 10 (one local, two regional, and four national) contractors stated that the systems they operated under had an appeal process. Additionally, 22 of the 31 survey respondents indicated that they used a post-proj- ect performance evaluation with an appeals process. That means that roughly 20% of the systems do not furnish a mechanism for contractor feedback. The contractor inter- view results are roughly in the same orders of magnitude and confirm that fact. rimental to either program. Quality is but one factor and will be discussed in detail later in this chapter. Other fac- tors, such as those listed in the survey, can be evaluated if a measurable standard can be set. Many of the factors found in the literature on contractor evaluation programs are inher- ently subjective. One excellent example is contractor efforts to “mitigate cost and time overruns” (FDOT 2005). FDOT recognizes the subjectivity of this evaluation factor in its program and seeks to reduce it as much as possible by estab- lishing a scoring scheme that correlates with the issuance of a “deficiency letter” when the agency believes the contractor is not acting to mitigate time and cost overruns. The FDOT system is as follows: Mitigate Cost and Time Overruns [12 points total] — The contractor takes the initiative and works diligently to avoid cost or time increases and to mitigate the effects of changed conditions whenever they do occur. Requests for additional money or time are well documented (complete and accurate), fair, and submitted timely. 12 points—The contractor worked diligently to avoid • cost and time increases or to mitigate the effects of changed conditions. All requests for additional money or time were in good faith, accurate, timely, and well documented. If additional documentation is requested, it was promptly provided. No more than one (1) deficiency letter by the CEI noting contractor’s failure to mitigate cost and time impacts. 9 points—No more than two (2) deficiency letters • by the CEI noting contractor’s failure to mitigate cost and time impacts. 6 points—No more than three (3) deficiency letters • by the CEI noting contractor’s failure to mitigate cost and time impacts. 4 points—No more than four (4) deficiency letters • by the CEI noting contractor’s failure to mitigate cost and time impacts. 0 points—Five (5) or more deficiency letters by the • CEI noting contractor’s failure to mitigate cost and time impacts (FDOT 2005). The important factor to note in the FDOT system is the requirement for its personnel to notify the contractor when they believe it is not acting in the state’s best interest by issu- ing a deficiency letter. Thus, this step creates an opportunity for communication between the agency and its contractor regarding what the agency expected and how the contrac- tor can correct its behavior if future cost and time overrun issues arise. Evaluation Fairness One paper describes fairness in contractor selection as giv- ing all bidders “the same information” and treating them “equally without any discrimination” (Palaneeswaran and Kumaraswamy 2000). Performance-based contractor prequalification seeks to discriminate between contractors with satisfactory records of performance and those with less

28 for poor performance. The survey found that 78% of those agencies with a contractor evaluation scheme kept the evaluations active for three to five years. Of the case study agencies (see chapter five), Michigan and Ontario kept the evaluations active for three years and Florida kept them in the record for longer than three years. Thus, it would appear that an active lifetime of at least three years seems to be appropriate. The second subissue deals with impact of a single negative evaluation. The survey asked whether a negative evaluation would automatically disqualify a contractor from bidding on future work. Only one agency (Quebec) answered that it would. None of the interviewed contractors indicated that the scheme with which they had experience followed that approach. This is a drastic adverse procurement action. In the litigious U.S. construction industry, disqualification and debarment is a serious step that likely results in legal action. Thus, it is safe to conclude that an effective contractor evalu- ation scheme need not disqualify the contractor for a single bad performance and that the life span of a given evaluation ought to be on the order of three years. A final subissue pertains to the way evaluations are con- ducted on multiyear projects as opposed to single-season projects. A large proportion of typical transportation con- struction projects are awarded and constructed in a single season. Therefore, a contractor that may have had initial performance issues in the early stages of a project should be allowed to “recover” and not be penalized if it has taken appropriate steps to correct early deficiencies. This would argue for a system of interim evaluations that pro- vides a conduit for issue resolution before the final rating. It also suggests that multiyear projects might have more than a single final project contractor performance evalu- ation to ensure that the agency is kept properly informed of current contractor performance based on the most recent information. U.S. CONTRACTOR PERFORMANCE EVALUATION SURVEY OUTCOMES The analysis of the responses to the survey questions about post-project performance evaluation led to a number of dis- coveries. First, post-project performance evaluation factors can be grouped into three categories: project performance, managerial, and project closeout. The first two categories relate to the previous analyses and the third one is new to this feature of the process. Also, the survey respondents’ perception of how implementing performance-based prequalification with a rigorous contractor evaluation proj- ect indicates that most project aspects will improve or not change, although other aspects would be negatively affected by the system. Evaluation Consistency The agency’s ability to achieve consistency from project to project and between different evaluators on a single contrac- tor was cited in one of the contractor structured interviews as its major concern. This response came from a national con- tractor with experience with performance-based prequalifi- cation and contractor evaluation in several different states. The Kentucky DOH guidelines for preparing contractor per- formance evaluations states: “Evaluations shall be performed in an objective, consistent, and well-documented manner” (Hancher et al. 2001) [emphasis added]. Therefore, both the owner and the contracting industry see achieving consis- tency as a desirable goal in contractor evaluation schemes. Highway agencies have long experience in implementing methods that achieve consistency in their quality manage- ment programs. For example, one paper addresses the crux of this issue by stating: “[quality assurance] action could be aimed not only at attaining consistency between what the agency wants and what it specifies but also at main- taining consistency between what the agency wants and receives” (Pathomvanich et al. 2002) [emphasis added]. This paper on quality assurance (QA) associates the verbs “attaining” and “maintaining” with the process of achiev- ing total consistency. By borrowing this idea from the QA world, highway agencies can apply the same to actions to achieve consistency in contractor evaluations. The first step is to attain consistency. Reviewing the Kentucky DOH’s use of contractor evaluations of the DOH, one sees a strong element of striving to attain consistency. Attaining consistency in the DOH program includes per- sonnel training and personnel evaluation to ensure that the evaluators understand and apply the evaluation scheme as it is designed and published. Next, using the contractor input to identify quality improvements and topics for discussion with the industry are aspects of maintaining consistency after it has been attained. It can be concluded that an effec- tive contractor evaluation scheme must be a living program in much the same manner that the agency implements its QA program. Effective Life of an Evaluation The final issue deals with how long a given evaluation remains as an active component of a contractor’s perfor- mance record. This issue can be broken down into three subissues. First, as stated in this synthesis’ objective, an evaluation system should encourage a marginal contrac- tor to improve its performance. To achieve this end, the life of a negative evaluation should be finite and eventu- ally be dropped from the active performance record. If the evaluation had an infinite life, the incentive to improve performance would never outweigh the penalty assessed

29 Project Performance Factors As shown by rank in Figure 19, the project performance fac- tors category is composed of seven factors that deal specifi- cally with how the contractor is performing the work on a project: Timely project completion • Timely and complete submittals • Environmental compliance • Proper maintenance of traffic • Impacts to the traveling public • Disadvantaged business enterprise (DBE) utilization• Level of effort displayed on the job • Impacts to the traveling public describe the contractor’s ability to maintain safe and steady traffic flow through their Post-Project Performance Evaluations The respondents were asked to identify the factors used by their state organization for post-project contractor per- formance evaluation (a complete listing of these factors is shown in Appendix B). When these factors are graphed (see Figure 19) and ranked by number of respondents who use each factor, a hierarchy of importance is established. This hierarchy can be broken into three distinct groups: Project performance factors using the same definition • as previous analyses Managerial performance factors using the same defini-• tion as previous analyses Project closeout performance factors, which are defined as • those evaluated factors that relate to the manner in which the job is completed and any postconstruction issues ! FIGURE 19 Surveyed post-project performance evaluation factors.

30 Responsiveness to warranty call-backs reflects the contrac- tor’s willingness to stand behind the work they have com- pleted in a manner that ensures quality and workmanship as discussed in the previous chapter. Evaluation Factor Breakdown As shown in Figure 20, the breakdown of the post-project performance evaluation process is roughly equal for perfor- mance and managerial factors. When these factors are com- pared with the criteria discussed in the previous chapter’s performance-based prequalification factor analysis (shown in Figure 10), it is clear that the factors used in the contractor performance evaluation system are essentially the same as those listed in the performance-based prequalification survey output. This leads to the conclusion that the contractor perfor- mance evaluation system must nest within the performance- based contractor prequalification program to be effective. FIGURE 20 Breakdown of post-project performance evaluation criteria. CONTRACTOR EVALUATION USAGE Next, the manner in which contractor performance evalua- tions were utilized by the surveyed state agencies was ana- lyzed. The majority (64%) used post-project performance evaluations as a factor in their administrative prequalifica- tion process, as shown in Figure 21. Another 24% use it for performance-based prequalification, and some responses indicated that it was used for both. A few agencies used this for the release of retainage or final payment. Additionally, 6% of the respondents conduct contractor evaluations, but do not feed them back into their process in any manner. When the 3% that did not know how they were used is added to the 6% that do not employ them, nearly one-tenth of the sample does not exploit the potential benefits of this powerful con- tract communication tool. Overall, this graph illustrates the point that contractor performance evaluations are used to feed the contractor prequalification process. project. DBE utilization describes the ability of the contrac- tor to accomplish stated goals for qualified DBE participa- tion in the construction project. The level of effort displayed is defined by the agency representative on the project and describes the effort displayed by the contractor and its employees to maintain constant and effective work flow. Managerial Performance Factors Managerial performance factors are composed of three fac- tors that are the heart of contractor management ability, including the willingness of the contractor’s personnel to work proactively with agency officials and representatives. The factors in this category are as follows: Coordination and cooperation with agency• Conformance with contract documents• QA program effectiveness• Safety program effectiveness• Coordination and cooperation with property owners• Coordination and cooperation with third-party • stakeholders Mitigation of time overruns• Mitigation of cost overruns• These factors encapsulate the contractors’ quality man- agement program effectiveness and evaluate how well the contractor is working to the letter of the contract. QA program effectiveness ties directly to quality and work- manship, one of the major factors on performance-based prequalification. Safety program effectiveness deals with the contractor’s ability to prevent accidents among both the workforce and the public. The next two factors deal with how well the contractor works with individuals not affiliated with the state agency such as business owners, city leaders, other government agencies, and utilities. Finally, mitigating time and cost overruns measures the contractor’s willing- ness to keep the impact of change orders and other contract modifications to their lowest reasonable value throughout the project. In some states, like Florida, the initial reason- ableness of contractor change order proposals is evaluated using these factors. Project Closeout Performance Factors Project closeout performance is a separate category and includes the following two factors: Timely punchlist completion• Responsiveness to warranty call-backs• These factors are generally addressed toward the end of the project. Timely punchlist completion deals specifically with project closeout activities and working with the agency to correct any minor deficiencies or final construction items.

31 Project-Specific Performance Contractor Quality Control Evaluation Although the experience with using contractor QC perfor- mance as part of the prequalification performance is limited, this issue has been researched in one area and some guidance on the topic can be gained. That area is in the use of project- specific prequalification requirements for DB transportation projects. To effectively transfer design liability to the design- builder, an agency must also transfer many of the traditional QA responsibilities. This leads to a concern that the “fox may be guarding the hen house” (Gransberg and Molenaar 2008). Most DB projects require that competing contractors submit an SOQ for evaluation by the agency. Thus, the opportunity to eliminate marginally qualified contractors from competi- tion exists at the outset of each DB project procurement pro- cess. A study by Ernzen and Feeney (2002) of the Arizona DOT’s DB program (appropriately titled, “Contractor-Led Quality Control and Quality Assurance Plus Design-Build: Who Is Watching The Quality?”) addressed this concern directly by comparing project QA test data on a DB project for which the design-builder had been assigned the responsi- bility for QA. To conduct this assessment, the design-builder was provided with data from a similar project delivered by traditional DBB means. It found the following: Analysis of the data shows that despite a highly compressed schedule, the quality of the material on the project exceeded the project specifications and was similar to the quality of work completed for the state under traditional contracting methods with an Arizona DOT-operated QA program (Ernzen and Feeny 2002). That DB project quality was roughly equal to the quality found on the DBB projects was confirmed by a 2006 FHWA report. The FHWA Design-Build Effectiveness Study (2006) reports actual results that confirm this belief as summarized in the following quotation: On average, the [DOT] managers of design-build projects surveyed in the study estimated that design-build project delivery reduced the overall duration of their projects by 14 percent, reduced the total cost of the projects by 3 percent, and maintained the same level of quality as compared to design-bid-build project delivery. [Emphasis added.] Therefore, taking the two studies together, it is evident that linking contractor past performance to the need for viable and trustworthy contractor QC programs is possible, even in a DB environment in which the agency must give up much of the traditional control it has in DBB project delivery. Fortunately, the literature contains examples of two long- standing contractor evaluation systems described in the next sections. These examples can be used as models for agencies that wish to implement this type of system. FIGURE 21 Breakdown of post-project performance evaluation use. The use of contractor project performance evaluations encompasses the three major aspects of project success: cost, time, and quality. Of the three, all highway agencies have formal quality management programs that have been used for extended periods of time. Also, of the three, final project quality is the aspect whose impact is felt beyond the end of construction. Therefore, contractor project per- formance evaluation programs must necessarily be syn- chronized with agency quality management programs and, in cases in which they are used, must furnish a means to directly evaluate the performance and effectiveness of con- tractor QC programs. PERFORMANCE-BASED PREQUALIFICATION AND CONTRACTOR QUALITY CONTROL One aspect of performance-based contractor prequalifica- tion that cannot be overlooked is its impact on the agency’s quality management program. Ensuring the quality of the final product is ultimately a major reason for implementing performance-based contractor prequalification. The FHWA allows state DOTs to use contractor test results in project acceptance decisions (FHWA 2004) and that authority has increased the emphasis on contractor QC programs. A contractor’s performance in properly implementing its approved contractor QC plan certainly could be evaluated as part of the prequalification process. Many factors will independently influence the outcome of any construction project. However, performance can be influenced by the system employed to ensure quality (TRB 1979). The sur- vey showed that nearly 70% of the respondents use con- tractor test results as part of their quality management programs. Only seven of those respondents indicated that contractor test results of performance in that area were car- ried into their prequalification processes. Thus, this is an area in which performance-based prequalification could be implemented.

32 project cost and time problems on behalf of the owner. Thus, there is no incentive to form partnerships on a project. Nevertheless, CONQUAS 21 appears to be a great model for the construction quality evaluation piece of performance- based contractor prequalification. Nothing would prevent a highway agency from adding the other evaluation categories for schedule, budget, and willingness to cooperate with the owner and third-party stakeholders on a given project. This combination is found in the contractor evaluation system used by many federal agencies. Federal Construction Contractor Appraisal Support System The FHWA’s Federal Lands Highway Divisions use the Department of Defense Construction Contractor Appraisal Support System (CCASS) (FHWA 1998a, 2005). An NCHRP study reports the following: The federal government and a number of state agencies have for many years maintained a database of contractor evaluations on past projects and often use this resource as a means to measure the contractor’s track record. Despite certain drawbacks, this appears to be the best means of assessing past performance as it allows contractors the opportunity to appeal negative ratings (Scott et al. 2006). The CCASS evaluation system has been used for decades and not only records actual contractor performance but also enables federal agencies to make a decision on the “respon- sibility” of bidders in a DBB project (FHWA 1998a; DOD 2007). In this system, a low bidder with several unsatisfactory ratings can be found to be “not responsible” and not awarded the contract. CCASS requires that the agency evaluate the contractor’s performance in the five areas shown in Table 8. The way in which its quality of work evaluation is imple- mented in relation to contractor QC in the FHWA system is described as follows: The first of these elements, Quality of Work, essentially overlaps the contractor’s inspection system requirements under FAR [Federal Acquisition Regulation] Clause 52.246-12, Inspection of Construction [contractor QC plan]. That is, a contractor which fails to maintain an effective quality control inspection system will generally warrant an unsatisfactory rating in the Quality of Work category. Deficient contractors must be clearly notified of the deficiencies and provided an opportunity to correct them. Evaluations may be shared with other contracting agencies and private entities. Evaluations may be used, in part, for determinations of responsibility prior to award of sealed bid contracts or in evaluating past performance as a part of source selection for a negotiated contract (FHWA 1998a). As indicated in the previous quote, the FHWA requires that the contractor be notified if the agency believes it is not per- forming at a satisfactory level. This kind of mandated com- Singapore Construction Quality Assessment System The Singapore Building and Construction Authority (2005) uses a system called CONQUAS 21. It involves a rigorous and seemingly objective evaluation of contractor construc- tion quality. The evaluation output is maintained in a data- base and is used to rate contractors for prequalification. The limiting aspect of CONQUAS 21 is that it is focused totally on construction quality and does not attempt to evaluate other aspects of contractor performance. It maintains its objectivity by using a go/no-go evaluation for a published set of construction quality standards for various technical features of work. For instance, structural steel is broken out into the following assemblies: Main Member/Partially Assembled Component• Metal Decking• Erection Tolerances• Corrosion and Fire Protection• Welding Test Reports• Contractor QC testing and owner QA efforts are used to determine whether or not each rated component met the published quality standards. A score is merely the sum of the “go” responses divided by the total number of evaluated items. These are plugged into a weighted criteria scheme that gives those members that have the greatest impact on over- all structural quality more weight than those that are ancil- lary features. For instance, main bridge structural members would be assigned more weight in the scheme than the guard rails. After all the major features are rated, an overall project performance rating is developed using a weighting scheme that uses the same logic as that used for the individual fea- tures of work. This system has several obvious advantages. First, the measurement of quality against a published standard removes much of the subjectivity from the evaluation. Hence, it could be less controversial to implement. Next, the focus on construction quality using a rigorous system to evaluate contractor QC program effectiveness creates an incentive to maximize quality to improve the contractor evaluation and enhance a contractor’s prequalification rating. Because Singapore uses these ratings to assemble bidders lists, this system creates an incentive for achieving acceptable quality the first time. CONQUAS 21 also has several disadvantages. The fact that it evaluates only quality creates an environment in which budget and schedule are no longer valued in the prequalifica- tion process. Thus, a contractor with an excellent construc- tion quality record but poor management and project control abilities receives the same rating as a contractor with an equal quality record but a history of finishing jobs on time and to budget. Second, there appears to be no place to insert evalu- ation factors for proactive contractors that strive to mitigate

33 Inadequate or incomplete quality control • documentation Failure to identify and correct deficient work• Inadequate materials and shop drawings submittals• Incorporation of unspecified materials (FHWA 2005)• CCASS evaluations are filed and remain in the contrac- tor’s record for six years. They are used to determine respon- sibility on DBB contracts and as part of the prequalification process on DB and other types of negotiated contracts. Given the above discussion, using some form of evaluation of a given contractor’s actual QC performance has a positive impact on final project quality. Additionally, that evaluation may be part of a performance-based contractor prequalifica- tion program. PERCEIVED IMPACT OF PERFORMANCE-BASED PREQUALIFICATION ON PROJECTS To measure the potential impact of performance-based con- tractor prequalification on future transportation projects, the survey asked the following: “Regardless of your experi- ence with contractor prequalification, in your opinion what impact would performance-based contractor prequalifica- tion have on the following project aspects?” munications between the owner and the contractor gives the contractor the ability to correct the defect found by the owner and, if necessary, to refute or clarify those perceived defects. In fact, the CCASS process requires that the agency forward all its ratings to the evaluated contractor and gives that entity 30 days to comment on the rating (DOD 2007). The agency must then review the contractor’s comments and determine whether or not to adjust the final rating. That this long-lived system is effective is evidenced by the fact that a recent sur- vey of both federal agency evaluators and their evaluated con- tractors found that 92% of the government respondents and 98% of the contractors believed that it is an “effective tool for improving government-contractor communication” (“Con- tractor Performance Assessment Reporting System” 2007). An example of how one of the Federal Lands Highway Divi- sions applies the rating system for quality is as follows: Quality of work reflects the contractor’s management of the quality control program, as well as the work performed. Questions, which could be addressed, are as follows: Has a quality product been provided? If not, specifically describe the deficiency in quality and the shortcomings in the contractor’s quality control system responsible for it, for example: Inadequate control• Failure to perform necessary testing• Failure to implement [the mandated] inspection process• TABLE 8 DEPARTMENT OF DEFENSE CONSTRUCTION CONTRACTOR APPRAISAL SUPPORT SYSTEM (CCASS) RATING AREAS 1. Quality Control Quality of workmanship• Adequacy of contractor QC plan• Implementation of contractor QC plan• Quality of QC documentation• Storage of materials• Adequacy of submittals• Adequacy of QC testing• Adequacy of as-builts• 4. Effectiveness of Management Cooperation and responsiveness• Management of resources/personnel• Coordination and control of subcontractors• Adequacy of site clean-up• Effectiveness of job-site supervision• Compliance with laws and regulations• Professional conduct• Review/resolution of subcontractor issues• Implementation of subcontracting plan (DBE)• 2. Timely Performance Adequacy of initial progress schedule• Adherence to approved schedule• Resolution of delays• Submission of required documentation• Completion of punchlist• Submission of updated and revised progress schedules• Warranty response• 5. Compliance with Labor Standards Correction of noted deficiencies• Payrolls properly completed and submitted• Compliance with labor laws and regulations with specific • attention to Davis-Bacon Act and EEO requirements 3. Compliance with Safety Standards Adequacy of safety plan• Implementation of safety plan• Correction of noted deficiencies• Possible Ratings: Outstanding, Above Average, Satisfactory, Marginal, Unsatisfactory in each area plus an overall rating for project. Source: Construction Contractor Appraisal (2007).

34 This particular analysis is integral to the idea of identi- fying potential barriers to implement the proposed system. DOT respondents would not be opposed to implementing per- formance-based contractor prequalification. Table 9 shows that the majority believe it will generally improve many, if not most, project aspects. This intersects with the information found in the literature review as evidenced by the following quotation: “The process of short-listing to only qualified bid- ders who have a proven track record with budget and schedule performance could, and does, enhance performance” (Mole- naar and Songer 1998). ANALYSIS OF CANADIAN SURVEY RESPONSES The Canadian results can be compared with the U.S. results to see how the two countries approach the same issue. Only four of the seven respondents used a post-project contractor evaluation system. Figure 22 uses the same labels for project performance, managerial performance, and project close- out performance as the U.S. results shown in Figure 19. The The survey then provided a list of project aspects and asked the respondant to choose whether implementation would make that project aspect “Better,” “No Change,” or “Worse.” It also allowed the respondents to select “No Opin- ion.” The results are shown in Table 9. Table 9 shows that the majority of respondents felt that implementing performance-based contractor prequali- fication would not have a negative impact on any project aspect except the “Number of Bidders.” For this particular aspect, “worse” indicates that there would be fewer bid- ders if everyone was prequalified based on their previous project performance. This confirms the very essence of performance-based prequalification: that marginal con- tractors will be encouraged to improve their performance to protect their unrestricted ability to bid. Therefore, this response can be interpreted as a vote for performance- based prequalification. Few respondents believed that implementing this process would make any of the other project aspects worse, as seen by 2% selecting “worse” in only three categories. TABLE 9 IMPACT OF PERFORMANCE-BASED CONTRACTOR PREQUALIFICATION ON PROJECT ASPECTS Project Aspect Better No Change Worse No Opinion Number of bidders 5% 37% 37% 21% Material quality 37% 41% 0% 22% Workmanship quality 59% 15% 0% 26% Safety 46% 22% 0% 32% Maintenance of traffic 34% 37% 0% 29% Level/amount of agency inspection required 20% 54% 2% 24% Timely project completion 44% 29% 0% 27% Timely construction submittal completion 39% 37% 0% 24% Timely punchlist completion 41% 34% 0% 25% Personnel experience 29% 42% 0% 29% Personnel competence 37% 37% 2% 24% Number of contractor-initiated change order requests 17% 54% 0% 29% Number of claims/disputes 24% 47% 0% 29% Responsiveness on warranty call-backs 24% 35% 0% 41% Achievement of DBE goals 32% 42% 2% 24% Environmental compliance 27% 44% 0% 29% Contractor cooperation with agency 44% 32% 0% 24% Contractor cooperation with property owners 24% 49% 0% 27% Contractor cooperation with third-party stakeholders 24% 49% 0% 27% Contractor cooperation with public concerns 29% 44% 0% 27% Note: Shading indicates predominate opinion.

35 the United States is probably a viable proposition and could be accomplished without a negative impact on U.S. projects. Overall, the results of the Canadian side of the survey reinforce some of the findings from the U.S. analysis. The Canadians place a greater emphasis on the business side of administrative prequalification and more of a management and production emphasis on the performance-based prequal- ification process. FIGURE 23 Canada breakdown of post-project performance evaluation criteria. ! Canadians gave more weight to “Safety Program Effective- ness” and “Responsiveness to Warranty Call-backs.” With these two notable exceptions, the Canadian post-project eval- uation system is roughly the same as that found in the United States. In Figure 23, the Canadian results are grouped into the three categories in the same manner as the U.S. results. FIGURE 22 Canada-surveyed post-project performance factors. Comparing Figure 23 with Figure 20 shows that results from the two countries are almost identical. Therefore, it appears that the post-project performance evaluation approach in Canada is not remarkably different than the one used in the United States. This validates the list of factors used in the survey as accounting for the major elements of post-project performance evaluation and lends authority to the conclusions drawn in this chapter. Table 10 outlines the perceived impact of performance- based prequalification. The table shows that the majority of Canadian respondents believe that implementing perfor- mance-based contractor prequalification would not negatively affect any project aspect other than the number of bidders. This is consistent with what the U.S. analysis found and shows that, even though the legal and regulatory environment is dif- ferent in Canada, the impact of implementing this program would be roughly the same. It also leads to the inference that importing certain features of the Canadian system for use in

36 two (one regional and one national) had experience with agencies that carried contractor QA performance into their prequalification system and one of those was the Canadian contractor. One of the interviewees, a national contractor, stated that the two programs need to be complementary rather than conflicting. In other words, a contractor with a proactive quality management program should not be penalized on its evaluations if it reports and corrects defi- ciencies as they are identified. The bottom line is that the contractors see the post-project performance evaluation as an essential part of the performance-based prequalifica- tion system. Their major concerns center around fairness and consistency in the ratings. These validate both the information found in the literature and the results of the agency survey. FIGURE 24 Contractor breakdown of post-project performance evaluation factors. SUMMARY The major factors of both performance-based contractor prequalification and of contractor project performance eval- uations were discussed in this and the previous chapter. It is clear that most of the survey respondents feel that imple- menting performance-based contractor prequalification with a rigorous system of project performance evaluation would have a positive impact on those project aspects listed in the survey. The contractors echo the owners’ sentiments. The rationale driving this perception is the essential con- nection between a performance evaluation of current work feeding the prequalification for future work. When project performance evaluation affects a contractor’s ability to bid on future work, it encourages the contractor to improve its performance on its ongoing projects. Molenaar and Johnson (2003) would predict that this ultimately will translate into higher-quality roads and bridges. TABLE 10 CANADA IMPACT OF PERFORMANCE-BASED CONTRACTOR PREQUALIFICATION ON PROJECT ASPECTS Project Aspect Better No Change Worse Number of bidders 14% 29% 43% Material quality 86% 14% 0% Workmanship quality 100% 0% 0% Safety 57% 29% 14% Maintenance of traffic 57% 29% 14% Level/amount of agency inspection required 29% 57% 14% Timely project completion 57% 43% 0% Timely construction submittal completion 29% 43% 0% Timely punchlist completion 29% 43% 0% Personnel experience 43% 57% 0% Personnel competence 71% 29% 0% Number of contractor-initiated change order requests 29% 57% 0% Number of claims/disputes 43% 57% 0% Responsiveness on warranty call- backs 86% 14% 0% Achievement of DBE goals 0% 14% 0% Environmental compliance 71% 29% 0% Contractor cooperation with agency 86% 0% 14% Contractor cooperation with prop- erty owners 43% 57% 0% Contractor cooperation with third- party stakeholders 57% 43% 0% Contractor cooperation with public concerns 43% 43% 14% Note: No opinion responses not shown. Shading indicates the predominate opinion of respondents. CONTRACTOR PERSPECTIVE ON POST-PROJECT PERFORMANCE EVALUATION The contractors were given the same list of evaluation fac- tors as given to the agency survey respondents. Figure 24 is a breakdown of that output and can be compared with Fig- ure 20, which shows the agency responses. The two figures are roughly the same, with contractors shifting the empha- sis to managerial factors and adding a bit more weight to the closeout factors. The contractor output again validates the agency results. Regarding the use of contractor test results, of the 10 responding contractors, one local, one regional, and five national contractors had completed proj- ects for which this program was in effect. However, only

37 CHAPTER FOUR BARRIERS TO IMPLEMENTATION istra 2008). However, neither the literature nor the survey contained information regarding the required staffing lev- els. This will be a significant issue for small states and does require future research to provide meaningful guidance. Evaluator Qualifications Next, the agency will need to ensure that its evaluators are indeed qualified to evaluate the subject contractors. In most cases, contractors are evaluated by the agency construction personnel who administer the evaluated contract. Implemen- tation will require that an ongoing training program for the evaluators be developed and fielded to ensure consistency between evaluators and different types of projects. Addition- ally, this kind of program will be necessary to demonstrate the agency’s commitment to fairness and the reduction of as much subjectivity in the process as possible. Agencies that currently use this type of system—such as the FHWA (FHWA 2005) and the Florida DOT (Sadler 2007)—have found that a review of all contractor evaluations at a level above the evaluator is also required to make the program as consistent as possible. This issue was raised in the contractor structured interviews, in which two local, two regional, and four national contractors indicated that their major concern was the agencies’ ability to consistently rate them from proj- ect to project. Evaluation Administrative Rules The administrative rules of the process need to be transpar- ent and logically derived (Parvin 2000). Determining the frequency of evaluations is important. The trend found in the literature (FHWA 2005; DOD 2007; MTO 2007; Sadler 2007) appears to support at least one interim evaluation before the final evaluation. Florida DOT furnishes evalua- tions on a monthly basis. The theme is to notify the contrac- tor when it is not performing well and give it the opportunity to correct its deficiencies and shortcomings before they become part of the permanent record. This notification leads to the need for an appeals process whereby the contractor can refute an unfavorable rating and that provides the contractor due process before it is penalized by the system. Of the survey respondents, 77% indicated that they had an appeals process for their contractor evaluations and all but one (Quebec) stated that a negative performance INTRODUCTION Any change to the status quo will encounter some resistance. Competitive bidding for highway projects has traditionally entailed an award that is made to the lowest responsive and responsible bidder (Minchin and Smith 2001). The defini- tions of “responsive and responsible” are well established in each highway agency’s area of operations. Integrating performance-based prequalification contingent on contrac- tor evaluation into those definitions has already been done in the federal sector (DOD 2007). To do so effectively at the state level may require overcoming a number of barriers to implementation. These barriers are both actual and per- ceptional. NCHRP Web Document 38 (Minchin and Smith 2001) lists the following implementation issues: Integration with existing construction administration • systems, such as site manager Qualifications of the evaluators• Evaluation process administrative rules • Frequency of evaluations – Appeals process development – Life span of evaluations and duration of – disqualification Impact on contractor bonding• Legal implications• ACTUAL BARRIERS TO IMPLEMENTATION The first implementation issue regarding integration of con- tractor evaluations with existing construction administration systems could require a significant commitment in person- nel to monitor it and be open to criticism. Scott et al. (2006) cites the fact that the federal CCASS system discussed in chapter three “has been accused of being resource intensive, overly subjective or biased, and subject to challenge.” An agency that is not using contractor evaluations will need to devote the appropriate level of personnel assets to implement a system that does use evaluations. Additionally, that staff will have to maintain the evaluation database and ensure that the most current information is used in each procure- ment action. The MTO (see chapter five for details) has a staff of three full-time employees to operate and administer their performance-based prequalification program (Tun-

38 evaluation does not automatically disqualify a contractor. Once again, the theme is to ensure that the process is both fair and justifiable. The next issue is the life span of a specific evaluation. The survey showed that the majority (73%) of the respon- dents had systems in which the evaluations were maintained in the active record for at least three years. The literature intersects with this finding from the survey and goes on to specify a “rolling 3-year average” (Hancher and Lambert 2002; MTO 2007). Thus, it would appear that this creates an added incentive for contractors to perform in a manner that earns them high ratings as it would take three years for a poor rating to be flushed out of the prequalification sys- tem. The issue of how long a contractor remains disqualified was not covered in the survey, but the literature shows that it tends to be a function of the reason for which they were sanctioned. Those that lost their qualification for criminal acts usually are debarred indefinitely. Those that are dis- qualified for marginal performance, usually for defaulting on a contract, can regain their qualification after proving to the agency that they have fixed the problems that led to their exclusion. The issue of multiyear contracts discussed in the previous chapter is also important when establishing admin- istrative rules for evaluation of life span. Evaluation Impact on Contractor Bonding This issue deals with contractors that normally have con- struction contracts with more than one owner or agency. The contractor’s bonding capacity must be spread across all its clients until it is exhausted. Therefore, an unfavor- able rating from a highway agency could be viewed by the surety in a manner that might cause it to reduce the avail- able bonding to work for other owners. NCHRP Synthesis 190 (Thomas and Smith 1994) concluded that, although state DOT capacity factors used in prequalification appear to parallel the bonding capacity computed by sureties, “it is highly improbable” that they will be the same. Neverthe- less, bonding companies do adjust final bonding capacities for factors that are outside the financial analysis (Minchin and Smith 2001) and, therefore, it is probable that an unfa- vorable rating by a DOT might reduce a contractor’s ability to bid work for other owners. Legal Implications The last issue deals with legal implications. None of the survey respondents indicated that they had laws that would prohibit performance-based contractor prequalification. Only one indicated that it had internal procurement policies that prohibit it. The contractor structured interviews showed a similar trend with only one local contractor citing DOT internal procedures as a possible barrier to implementation. Therefore, while the literature does contain a number of dis- cussions regarding possible legal issues, it can be concluded that for the most part these issues are not going to constitute a barrier to implementation. PERCEIVED BARRIERS TO IMPLEMENTATION Changing a system that has been in place for decades requires changing the organization’s procurement culture and that will inevitably encounter resistance from both internal and exter- nal stakeholders that are both comfortable with the status quo and believe that change is truly unnecessary (Ellicott 1994). A previous NCHRP Synthesis (Gransberg and Molenaar 2008) measured the magnitude of the impact of perceptions on the ability to change the procurement paradigm for trans- portation projects. The procurement change in that study was the shifting of some QA responsibilities to the contractor in DB projects. That study found that in spite of quantitative evi- dence the change would not be deleterious to ultimate proj- ect quality and that 50% of the public officials still felt that change was dangerous. It also found that 86% of the members of the construction industry who would be affected directly by the procurement system change believed that the change ultimately would be beneficial. The difference in the two groups’ perceptions was both interesting and important. The study showed that the greatest resistance to change came from within the highway agencies that are the ultimate decision makers on procurement system change. Thus, it is important to consider the issue of perceived barriers to the implementa- tion of performance-based contractor prequalification. Perceived Barriers from the Literature The literature shows that a major perceived issue is the ability for an agency to develop and implement a system that is both fair and equitable (Al-Gobali and Bubshait 1996; Minchin and Smith 2001; Hancher and Lambert 2002). NCHRP Web Document 38 took a close look at some of these perceived barriers to implementation of performance-based contractor prequalification. That study issued a survey that included the following questions and response rates: Is it possible to justly rate a contractor’s quality of work • and tie it to qualification? When all of the respondents are considered, 80% said yes, 12% said no, and 8% gave some other answer. Of the DOT respondents, 83% said yes, 9% said no, and 8% gave some other answer. Of the contractors, 66% said yes, 28% said no, and 6% gave some other answer. Is it possible to justly rate a contractor’s quality of • work and factor it into a bid to determine the awardee of a contract? When all respondents are considered, 47% said yes, 37% said no, and 16% gave some other answer. Of the DOT respondents, 50% said yes, 33% said no, and 17% gave some other answer. Of the con- tractors, 34% said yes, 57% said no, and 9% gave some other answer (Minchin and Smith 2001).

39 consistency issues in the development of contractor evalua- tion programs. The real measure from this research instrument of con- tractor perceptions was the output from the portion of the interviews for which the contractors were asked to rate their perception of how implementing performance-based prequalification would affect the same set of project aspects as those that were listed in the general agency survey. The contractor responses are shown in Table 11 and demonstrate solid support for the system. They agreed with owners that the numbers of bidders would be the only negative impact because performance-based prequalification would neces- sarily reduce the number of marginal contractors that were qualified to bid. After that, the contractor rated every other aspect as either better or no change, tracking closely to the agency responses. This leads to the conclusion that the per- ception that the construction community will be resistant to this change in the procurement process is incorrect. As previously stated, however, agencies that are contemplat- ing the change need to be sensitive to the possible negative perceptions and take steps to ensure that their internal and external stakeholders are assured that the change will result in a system that is both fair and equitable. Looking at the output from Table 11, it is apparent that agency personnel believe that the area that will show the greatest level of improvement is in workmanship quality. The contractors perceived that the prequalification process drove them to increase the experience and competence of their personnel. Both groups agreed that implementing performance-based prequalification would not appreciably change the number of contractor-initiated change orders. The perceptions were different regarding the impact on the number of claims, however, with the contractors feeling that such a program would create an environment that would discourage claims. The other project aspect in which agency and contractor perceptions diverged was in the area of level of agency inspection required. Most of the contractors believed that the agency could reduce its inspection levels, which may be related to their belief that they would need to increase the quality of the people they assigned to the job. Most of the owners felt that performance-based prequalifi- cation would have no appreciable impact on that aspect of their work. The final perceived issue addressed in the contractor struc- tured interviews was their opinion of how their competitors and subcontractors felt about performance-based prequalifi- cation. Nearly all (9 of 10) of the contractors indicated that their peers and competitors would support prequalification, and the remaining one, a national contractor, indicated that it would be neutral to the idea. Two-thirds of the contractors believed that their subcontractors would either support it or be neutral to the idea, with only one (a national contractor) indicating that its subcontractors would oppose it. Looking at the response rate to the first question, a poten- tial perceived issue could create a barrier to implementation. Nearly one-third of the contractors believed that it would be impossible for an agency to “justly rate a contractor’s quality of work.” Although a minority, a number of DOT personnel agreed with this notion. When the question was changed to cover the use of contractor past performance evaluations in the contract award decision, the percentages of negative perceptions increased for both groups. Thus, this leads to the inference that highway agencies that intend to implement performance-based contractor prequalification will need to address possible negative perceptions from both inside their organizations and within the construction contracting community. Perceived Barriers from the Contractor Interviews Public procurement policy research has found that perceptions are often of equal importance to facts. Legislative action is heavily influenced by perceptions. One report on construction procurement policy implementation classifies perceptions as “barriers to broad acceptance” (Byrd and Grant 1993). Thus, it is important to understand the breadth and range of potential negative perceptions when developing a plan to implement a significant change in construction procurement policy such as performance-based contractor prequalification. Another perception issue regarding performance-based contractor prequalification is the possibility that smaller general contractors may be at a disadvantage as a result of having less financial capacity to both obtain and maintain the state-of-the-art in highway construction equipment and to pay the higher salaries it takes to attract and retain highly qualified project management personnel (Minchin and Smith 2001). This issue is particularly important in states with relatively small construction budgets whose construc- tion industry is particularly dependent on local firms. The fear is not that these firms cannot furnish satisfactory quality and timeliness, but rather that their work will be compared with national contractors and evaluated lower for the reasons previously cited. As part of this synthesis, a series of con- tractor structured interviews were conducted, half of which were with smaller local and regional contractors. None of the contractors that were interviewed expressed this con- cern, regardless of size. However, most believed that there are contractors that feel this way. The contractor structured interview questionnaires asked three questions regarding barriers to implementation and in only one case (a national contractor) did a contractor express the idea that the industry would oppose such an ini- tiative. That was regarding post-project evaluations, and that contractor’s issue was the agencies’ ability to develop and implement a system that was fair. This intersects with the literature findings and, although the response rate was 1 out of 10, it still confirms the need to be sensitive to equity and

40 be sensitive to some of the perceived barriers discussed above. This sensitivity can take the form of ensuring that all aspects of performance-based contractor prequalification are trans- parent to the industry and that the agency implements the sys- tem exactly as it is published. Internal checks and balances need to be in place and operating to assuage industry concerns regarding fairness and consistency. Agency personnel need to be trained on how to conduct the evaluations. Agencies need to ensure that the algorithm with which the evaluations are fed into the prequalification formula does not contain any inher- ent bias regarding contractor size. In summation, the research shows that if an agency is sensitive to potential perceptions, there are no major barriers to implementation. TABLE 11 IMPACT OF IMPLEMENTING PERFORMANCE-BASED CONTRACTOR PREQUALIFICATION ON VARIOUS PROJECT ASPECTS: AGENCY AND CONTRACTOR PERCEPTIONS Project Aspect Agency Responses Contractor Responses B N W B N W Number of bidders 5% 37% 37% 0% 0% 100% Material quality 37% 41% 0% 78% 22% 0% Workmanship quality 59% 15% 0% 89% 11% 0% Safety 46% 22% 0% 89% 11% 0% Maintenance of traffic 34% 37% 0% 78% 22% 0% Level/amount of agency inspection required 20% 54% 2% 89% 11% 0% Timely project completion 44% 29% 0% 78% 22% 0% Timely construction submittal completion 39% 37% 0% 89% 11% 0% Timely punchlist completion 41% 34% 0% 89% 11% 0% Personnel experience 29% 41% 0% 100% 0% 0% Personnel competence 37% 37% 2% 100% 0% 0% Number of contractor-initiated change order requests 17% 54% 0% 33% 67% 0% Number of claims/disputes 24% 46% 0% 67% 33% 0% Responsiveness on warranty call-backs 24% 34% 0% 67% 33% 0% Achievement of DBE goals 32% 41% 2% 22% 78% 0% Environmental compliance 27% 44% 0% 56% 44% 0% Contractor cooperation with agency 44% 32% 0% 78% 22% 0% Contractor cooperation with property owners 24% 49% 0% 67% 33% 0% Contractor cooperation with third-party stakeholders 24% 49% 0% 56% 44% 0% Contractor cooperation with public concerns 29% 44% 0% 67% 33% 0% B = better; N = no change; W = worse. Note: Percentages by category will not all sum to 100% as the survey allowed respondents to also select “no opinion” as a response. SUMMARY The previous discussion and the comparison of both agency and contractor output from the survey and interviews support the conclusion that there are few actual barriers to implementing performance-based contractor prequalification. Many high- way agencies already have some form of contractor evaluation in their system. Many have some form of performance-based prequalification as well. The contractors interviewed for this report were receptive to implementing this system, because it reduces the number of marginally qualified contractors against which they must compete. Nevertheless, implementation must

41 CHAPTER FIVE PREQUALIFICATION CASE STUDIES METHODOLOGY Case study information was collected using a combination of Yin’s (1994) methodology for case study research and Oppenheim’s (1992) approach to conducting structured interviews. A case study outline was developed and used to collect the same information for each of the three prequali- fication case studies. Additionally, a structured interview questionnaire was prepared and used to guide the interview of each agency expert. This allowed the output to be directly compared and contrasted. Before getting into the details of each specific case, it is instructive to first compare the three case study agencies’ responses with the outline and survey. Prequalification Factors Table 12 is an inventory of each agency’s performance-based prequalification factors based on the factors given in the sur- vey. The important point is that none of the agencies used any of the last seven factors shown in the table. This is logical in that those factors tend to be project-specific and, as a result, neither the contractor nor the agency has the ability to evalu- ate them on a general basis. The previous discussion of proj- ect-specific performance-based contractor prequalification found these factors to be important components for identify- ing qualified contractors for a specific project. Thus, this leads to the conclusion that factors that tend to be project-specific could be eliminated from a general prequalification system. Next, Table 12 shows that all three case study agencies include the following factors in their systems: Financial capability• Calculated capacity factor from financials• Detailed financial analysis• Equipment and plant• Performance evaluations• Past project experience• These six factors can be logically divided into two cat- egories. The first four factors involve measuring the contrac- tor’s financial health, which is done in most administrative prequalification systems reviewed in this research. The second category is composed of the last two factors, which include rating the contractor’s past performance and appli- cable experience for the work for which it is applying to be INTRODUCTION The preceding chapters have discussed the trends in the United States and Canada regarding performance-based contractor prequalification. Many states have integrated performance-based features into their standard prequalifi- cation process. Interviews with both agency personnel and construction contractors lead one to the conclusion that the salient feature of performance-based contractor prequali- fication is a contractor evaluation program that feeds the prequalification process in a fair and equitable manner. One interviewee termed this “a contractor evaluation system with teeth.” Another comment captured on a survey indi- cated that “we need a system that rewards good contractors and limits poor contractors. In my mind, the only way to do that is to limit the amount of work a contractor can be qualified to do.” Thus, combining those two thoughts led the researcher to seek specific case studies from agencies that have the following: A seemingly objective contractor performance evalu-• ation system that feeds its prequalification process in a meaningful way A specific process that uses the performance evaluation • output to reward contractors with a good record and encourage contractors with a poor record to improve With those criteria in mind, the following three agencies were found to exemplify the objectives stated above: Michigan Department of Transportation (MDOT)• Florida Department of Transportation (DOT)• Ontario Ministry of Transportation (MTO)• Additionally, a seminal court case regarding the con- tent of the Minnesota DOT’s prequalification program for DB projects furnishes an insightful look at how an agency can successfully defend itself against a legal challenge to contractor prequalification by structuring its program in a thoughtful, objective manner. Therefore, this chapter pres- ents the details of four case studies, which exemplify the successful implementation of performance-based contrac- tor prequalification for agencies that may want to strengthen their programs.

42 a qualified bidder. Thus, the case study agencies all use a combination of financial qualification and performance or experience qualification to form the core of their perfor- mance-based prequalification programs. TABLE 12 COMPARISON OF CASE STUDY AGENCY PERFORMANCE- BASED PREQUALIFICATION FACTORS Performance-Based Prequalification Factor Florida DOT Michigan DOT Ontario DOT Financial capability x x x Calculated capacity fac- tor from financials x x x Bonding capacity x Surety statements x Detailed financial analysis x x x Bank statements x Insurances x Managerial ability x x Resumes for key personnel x Professional licensing for key personnel Key personnel past proj- ect experience x Equipment and plant x x x Technical ability x x Calculated ability factor from financials x Past illegal behavior x x Performance evaluations x x x Claims history x x Past project experience x x x Timely completion of past projects x Quality of material and workmanship x Workman’s compensa- tion modifier Quality assurance plans Safety plans Environmental plans Traffic control plans Level of subcontracting Use of DBEs The two U.S. DOTs also include factors related to mana- gerial and technical ability as well as past illegal behavior. This is in contrast to the Ontario system, which is much less complex, and even more different when one considers that Ontario does not require performance bonding from its prequalified contractors. Although no conclusions can be drawn from this comparison, it does make the argument for keeping these types of programs as simple as possible. This idea coincides with the contractor interview com- ments regarding the requirement to furnish large amounts of tedious company information that would not seem to affect the prequalification decision. TABLE 13 COMPARISONS OF CASE STUDY AGENCY PERFORMANCE EVALUATION FACTORS Performance Evaluation Factor Florida DOT Michigan DOT Ontario DOT Level of effort displayed on the job x Proper maintenance of traffic x x x Impacts to the traveling public x x Timely and complete submittals x x x Timely project completion x x x Environmental compliance x x x Coordination/coopera- tion with agency x x x Coordination/coopera- tion with property owners x x x Coordination/coopera- tion with third-party stakeholders x x x Conformance with con- tract documents x x x Quality assurance pro- gram effectiveness x x x Safety program effectiveness x x x DBE utilization x Mitigate cost overruns x Mitigate time overruns x Timely punchlist completion Responsiveness to war- ranty call-backs Other: Personnel and equipment; Management competence x x

43 largely beneficial, rating most categories as better with performance-based contractor prequalification. An excep- tion is the impact on the number of bidders: two agencies indicated no change and one believed that it reduced the number of bidders. Both Florida and Ontario have had sub- stantial experience with this process and, as a result of its longevity, may have given a “no change” answer because the system has been institutionalized. Intuitively, if an agency is able to eliminate poor-performing contractors through prequalification procedures, it would follow that the number of potential bidders would decrease. Another result is that two of the three agencies felt that the program would have no change on contractor-initiated change order requests. In the interviews, both indicated that those were a function of project aspects that had no relationship to the ability of the contractor. Therefore, contractors would have no choice but to request a change, if one was authorized, regardless of their qualifications or the mechanics of the prequalification system. After comparing the three case study agencies side by side, we can delve into the details of each agency’s contractor prequalification program. Because Michigan does not directly incorporate its evaluation output into a calculation of bidding capacity with the state, it will be reviewed first. MICHIGAN DEPARTMENT OF TRANSPORTATION PREQUALIFICATION PROGRAM CASE STUDY The MDOT’s performance-based contractor prequalification program revolves around a system of combining adminis- trative prequalification calculations and declarations with a program of contractor performance evaluation on a project- by-project basis. MDOT evaluates each contractor in the fol- lowing four areas: Organization and Management• Resources• Work Performance• Subcontractor Management• The first area includes the managerial effectiveness and technical competence of the contractor’s personnel. It also covers scheduling, timeliness, and completeness of sub- mittals, quality control plans, and prompt payments. The “resource” area judges the contractor’s ability to provide and maintain the requisite number of equipment and workers, as well as the efficiency of those resources. Work performance entails the effectiveness of safety programs, maintenance of traffic, the contractor QC program, its cooperation with third-party stakeholders (such as utilities), environmental compliance, and timely project closeout. The last area grades the contractor’s ability to manage the work performed by its subcontractors. Contractor Evaluation Factors Table 13 shows each agency’s performance evaluation fac- tors. The reader could note that contractor evaluation is project-specific and therefore one would expect it to be more detailed than prequalification. The upshot is that in all three cases the contractor evaluations are used to feed the performance-based prequalification systems and, thus, could be viewed as input. Both Florida and Ontario use the evaluation output to adjust the amount of work on which a given rated contractor can bid. The table clearly shows that the heart of a successful contractor evaluation system includes evaluation of various aspects of project timeliness, the contractor’s level of coordination and cooperation, and effectiveness of its quality and safety programs. Addition- ally, proper maintenance of traffic during construction was also important. Table 14 compares how each case study agency admin- isters its contractor performance evaluation system. All three agencies provide the contractor with a copy of a per- formance evaluation and have an appeals system to address negative evaluations. Perhaps most important, none of them uses a single negative performance evaluation to disqualify a contractor. TABLE 14 COMPARISON OF CASE STUDY AGENCY PERFORMANCE EVALUATION SYSTEM ADMINISTRATION Output Usage Florida DOT Michigan DOT Ontario DOT Does the contractor receive a copy of the completed performance evaluation? Yes Yes Yes Are you required to notify the contractor prior to submission if the performance evaluation is considered negative? Yes No Yes Is there an appeals process for a contractor that receives a negative performance evaluation? Yes Yes Yes How long do performance evaluations remain on the record? >3 years 3 years 3 years Does a negative performance evaluation automatically dis- qualify a contractor? No No No Impact of Implementing Performance-Based Prequalification Table 15 shows a side-by-side comparison of each agency’s assessment of the impact of its performance-based con- tractor prequalification program on various aspects of a generic project. It is apparent that the interviewed agency representatives felt that their respective programs were

44 is an entity called the “contractor performance evalua- tion review team,” which conducts the physical review of all contractor performance evaluations, looking for trends of unsatisfactory performance by each individual con- tractor. If a trend is identified, the team takes one of the following actions: Tier one: If the performance trend is considered insig-• nificant, no action is taken. Tier two: If the performance trend is considered impor-• tant but not critical, the review team will note this and continue to monitor that contractor for further evidence that the trend is getting worse. Tier three: If the performance trend is considered • significant and critical, the review team will issue a “tier-three letter” that notifies the contractor that it is not performing satisfactorily, the details of the trend, and directing the contractor to respond to the team detailing its plans to correct the unsatisfactory performance. If the contractor is either unresponsive or fails to make the promised corrections, the review team then issues a “tier-four” letter to the prequalification committee recom- mending action be taken to adjust the contractor’s ability to bid on MDOT projects. The prequalification committee has the authority to reduce a contractor’s bidding capacity with MDOT (and hence the amount of work it can bid on), sus- pend a contractor’s prequalification for a specific period, or revoke the contractor’s prequalification entirely. Adverse Action Process The connection between the contractor’s project perfor- mance evaluation record and its prequalification is made through the two administrative bodies. The scope of the impact of poor performance is determined on a judgmen- tal basis with the prequalification committee deciding an appropriate adverse action on a case-by-case basis. This sys- tem allows MDOT to have a great deal of flexibility in the mechanics of connecting the project performance evaluation with the ability of a contractor to bid its maximum capacity of work. Thus, some of the subjective issues that often arise in contractor evaluation programs, such as personal bias of a particular state employee or assigning fault for acts of force majeure, can be addressed qualitatively in the prequalifica- tion committee before a decision is made to affect a contrac- tor’s ability to bid. FLORIDA DEPARTMENT OF TRANSPORTATION PREQUALIFICATION PROGRAM CASE STUDY Unlike Michigan’s system, the FDOT uses a performance- based contractor prequalification program with a direct mathematical link between contractor evaluations and a TABLE 15 CASE STUDY AGENCY ASSESSMENT OF THE IMPACT OF PERFORMANCE-BASED CONTRACTOR PREQUALIFICATION ON VARIOUS PROJECT ASPECTS Project Aspect Florida DOT Michigan DOT Ontario DOT Number of bidders N W N Material quality N B B Workmanship quality B B B Safety B B B Maintenance of traffic B B B Level/amount of agency inspection required B W B Timely project completion B B B Timely construction submit- tal completion B B B Timely punchlist completion B B B Personnel experience B B B Personnel competence B B B Number of contractor-initi- ated change order requests B N N Number of claims/disputes B B N Responsiveness on warranty call-backs B B B Achievement of DBE goals B B O Environmental compliance B B B Contractor cooperation with agency B B B Contractor cooperation with property owners B B B Contractor cooperation with third-party stakeholders B B B Contractor cooperation with public concerns B B B B = better; N = no change; W = worse; O = no opinion. Michigan Performance Tiers MDOT’s performance-based contractor prequalification program is governed by the State Office of Administrative Hearings and Rule (SOAHR) 247, entitled “Classification and Rating of Bidders” (SOAHR 2002). That authority establishes a “prequalification committee” that will “deter- mine and award or renew numerical ratings, an overall financial rating, and work classifications for the bidder. The prequalification committee shall review the bidder’s con- tractor performance evaluations for at least the last 2 con- struction seasons, if available, before awarding or renewing prequalification” (SOAHR 2002). Supporting this effort

45 Performance Deficiency Letter FDOT uses an administrative technique called the “perfor- mance deficiency letter” to communicate its concerns to the contractor regarding issues that arise in the rated areas. It can be preceded by a “performance deficiency warning let- ter” that notifies the contractor of “shortcomings/noncom- pliances” and gives it a chance to correct the issue before a formal “performance deficiency letter” is issued. Once a formal performance deficiency letter has been issued, it is then recorded and the number of these received in a pro- rated “year” is used to calculate a “deficiency letter factor.” The deficiency letter factor is used to determine the grade in those rated areas for which deficiency letters are appli- cable. For instance, a contractor with no deficiency letters will receive the maximum score in “maintenance of traffic,” and one that has six or more letters regarding maintenance of traffic deficiencies receives a score of zero (0). FDOT has incorporated incentives for exceeding the standard in the areas of timely project completion, environ- mental compliance, and DBE utilization. A contractor that exceeds the minimum requirements and goals in these cat- egories can receive bonus points on its performance rating. The maximum number of total points without bonuses is 98 and with bonus points is 110. In 2007, FDOT had an average score of 94.6 with a range of 26 to 110 (Sadler 2007). Impor- tantly, a contractor does not have to finish early to receive a bonus score in the timely completion area. The instruction for that category indicates that a bonus point is in recognition “that a contractor may have to work through weather, utili- ties, added work, or other unforeseen conditions or delays” (FDOT 2005). Therefore, FDOT recognizes the contractor’s efforts to maintain the schedule despite difficulties encoun- tered on the job. Computing Ability Factor The accumulated record of a given contractor’s rated perfor- mance is termed the “contractor’s past performance record” and is used to determine an “ability factor” for each contrac- tor in the database (Sadler 2007). This ability factor is used to determine the “maximum capacity rating” of a contractor that seeks to be deemed qualified to bid on FDOT contracts. Figure 25 shows the conversion of the “contractor’s past per- formance record” to “ability factor.” The change in the slopes of the lines between the lower and upper portions of the acceptable range illustrates the incentive to stay on the high end of the rating spectrum—for example, a three-point rating difference from 93 to 96 yields an ability factor increase of 2 points, whereas the same three-point difference from 76 to 79 yields an ability factor increase of only 1 point. Once the ability factor based on past performance ratings is determined, it is used in the following formula (Eq. 1) to determine a contractor’s “maximum capacity rating,” which contractor’s ability to bid. As with MDOT, however, FDOT combines administrative prequalification calculations and declarations with a program of contractor performance eval- uation on a project-by-project basis. FDOT evaluates each contractor’s projects in the following nine areas: Work pursuit• Maintenance of traffic• Timeliness of submittals• Timely project completion• Coordination and cooperation• Mitigation of cost and time overruns• Environmental compliance• Conformance with contract• DBE utilization (if applicable to the given project)• Evaluation and Rated Performance Areas Most of these rated areas are self-explanatory. FDOT’s emphasis is on rating how well the contractor “partners” with the agency in its coordination, cooperation, and miti- gation areas. According to the Florida contractor interview, these are the most effective portions of the FDOT evalua- tion system. FDOT has an explicit definition for each area and prescribes the number of points to be awarded based on achievement of the definition. For example, the “pursuit of work” area is defined as follows: Contractor diligently and systematically pursues the work with sufficient labor, materials, and equipment at all times. Active progress is made on critical path items each day in accordance with the approved schedule. The contractor schedules the subcontractors so that they are pursuing their work as well. Contractor worked five (5) days a week unless the contract states otherwise, excluding weather days. Percent is based on allowable contract time (minus weather days) and on a five (5) workday week unless otherwise stated in the contract (FDOT 2005). And in the first area, a contractor can earn 12 points if it meets the following definition: The contractor aggressively pursued the work 90% of the days. Documentation in the project files by the CEI [construction engineering and inspection] reveals that the progress of the work was unsatisfactory no more than 10% (FDOT 2005). The definition furnishes an element of objectivity by pre- scribing that the contractor have no more than 1 day in 10 of unsatisfactory work progress, and this standard is some- thing that can be measured. Although it contains subjective wording such as “diligently and systematically pursues the work,” the project-specific understanding of these can eas- ily be discussed and agreed upon between FDOT and the contractor at the project’s preconstruction meeting, which would enhance understanding of the system’s application on the given project.

46 Adverse Action Process Presumably to keep their qualified bidders list as large as possible, while ensuring that only qualified contrac- tors are allowed to bid, FDOT also has a process to deal with contractors whose performance records are marginal or unsatisfactory. First, if a given contractor receives two or more evaluations less than 76 related to workmanship, cooperation, or environmental issues, it is issued a “letter of concern” that discusses the specific issues FDOT has found to be lacking. This generally leads to a meeting in which FDOT personnel explain the issues and corrective actions necessary. If a contractor has the same problems that led to a letter of concern as well as projects that are in liquidated damages, has continuing quality concerns, or has problems with specific types of work, FDOT may choose to restrict that company’s bid capacity until the situ- ation is remedied. If the problems continue, the contractor can be removed from the bidders list for a specific period of time. Finally, if major performance concerns continue or the contractor is found to be in default, FDOT can revoke the company’s qualification for periods allowed by the Stat- ute and Administrative Rule. The typical revocation period has been three years. is the “aggregate dollar volume of uncompleted work a firm is allowed by the Department to have under contract at one time” (FDOT 2004). This, in turn, determines the amount of work that it can be awarded: FIGURE 25 FDOT ability factor conversion from contractor’s past performance record (Sadler 2007). MCR = AF * CRF * ANW (1) Where: MCR = maximum capacity rating, AF = ability factor, CRF = current ratio factor (current assets/ current liabilities), and ANW = adjusted net worth. Table 16 shows how this formula is used to determine the MCR for differently sized companies with different past performance rating histories. The “contractor’s past per- formance record” has a significant impact within the same company on its ability to pursue FDOT projects. The posi- tive impact of staying in the upper range of the ability factor calculation is also evident.

47 the norm for the state. If these evaluations in question were detrimental to the contractor, they are changed. The bottom line for this issue is that there will always be an element of subjectivity in the most carefully devised contractor perfor- mance evaluation system. Therefore, the agency must have a review process like the one FDOT uses that demonstrates a tangible commitment to fairness and includes a remedy for undeserved contractor evaluations. Connection with Contractor Quality Control Program Additionally, the contractor expressed frustration with an apparent conflict between the performance evaluation sys- tem and the contractor QC program. The example cited was one in which the contractor’s QC program identified and reported a quality issue that, because of the timing, was cor- rected by the contractor after the monthly evaluation report was issued but within a reasonable period of finding the defi- ciency. In this case, the evaluation was scored down because of the quality issue and no mechanism was available to raise the contractor’s score after the quality issue was satisfacto- rily resolved. Thus, this conflict could create an unintended bias for the contractor to “manage” or, worse, under-report the flow of adverse contractor QC information to retain as high a performance score as possible. Again, FDOT is aware of the issue and is working to address it through its contractor QC specifications. This perceived conflict may only occur on that particular contractor’s project as a result of an indi- vidual interpretation of the FDOT program. Nevertheless, TABLE 16 IMPACT OF FDOT ABILITY FACTOR ON MAxIMUM CAPACITY RATING Contractor’s Past Performance Record Ability Factor Current Ratio Factor Adjusted Net Worth (in $Millions) Maximum Capacity Rating (in $Millions) Large Company—ANW > $100 million 98–100 15 1.3 $334.1 $6,515 74–76 4 1.3 $334.1 $1,737 64 or less 1 1.3 $334.1 $434 Medium Company—ANW $20 Million to $100 million 98–100 15 1.3 $52.7 $1,028 74–76 4 1.3 $52.7 $274 64 or less 1 1.3 $52.7 $69 Small Company—ANW < $20 million 98–100 15 1.3 $1.5 $29 74–76 4 1.3 $1.5 $8 64 or less 1 1.3 $1.5 $2 Source: Sadler (2007). Two-Way Communications All in all, one can see that the FDOT performance-based contractor prequalification program is quite robust. It is made further so by the way it is implemented. Contractors are given interim evaluations on a monthly basis through- out the life of the project. This enhances communications regarding contractor performance and effectively eliminates an argument regarding performance evaluations at the end of a project by forcing the issue at a time when corrections can still be made, if the contractor is indeed willing. The inter- view with the Florida contractor showed that these monthly evaluations were valuable and provided a means of focusing attention on overall performance as well as production. That interviewee indicated that the evaluated areas that were focused on partnering between the two organizations were effective. However, two major concerns with the system were cited. First, in spite of the efforts to make this evaluation as objective as possible, there is still an element of subjectiv- ity, and the contractor felt that FDOT needed to better rein- force consistency between contracts and different evaluators. In an effort to address this issue, the FDOT state construc- tion engineer’s office reviews every Contractor’s Past Perfor- mance Record (CPPR) score for conformance with procedure as well as posted guidance regarding an interpretation of the CPPR rating process. If a particular evaluator is found to be too aggressive or too lax, that evaluator is directly counseled in an effort to bring the individual’s evaluations back into

48 being allowed to bid on any work in which that contractor is engaged in specifically defined legal proceedings against the owner. Essentially, any firm that is actively engaged in legal proceedings against the MTO may not be allowed to tender a bid for its projects for an “exclusion period” of up to three years. The provision is decided by the qualification commit- tee on a case-by-case basis. Quoting from the provisions: It is prudent for the Ministry to avoid the additional costs associated with extraordinary time required of the contract administrator and Ministry staff in the management of projects with contractors, who have been or are engaged in a legal proceeding. It is therefore prudent for the Ministry to avoid awarding contracts to those contractors and related persons whose past performance demonstrates a significant increase in the level of management effort demanded by the Ministry and its representatives (MTO 2007). The entire qualification system is overseen by the qualifi- cation committee made up of ministry executives who estab- lish the policies and procedures and decide on sanctions. Components of the Integrated Prequalification System The interview with the agency stressed that the success of the system lies in the interrelationships of all the compo- nents with one another. Although certain components of the system can be used independently (the Contractor Perfor- mance Index was implemented about eight years ago), the program works best when all the components are integrated. An agency should be cautious if an attempt to implement any single component without the support of the others is considered. The Integrated Infraction Report System is related to the issuance of an infraction, which is a serious breach of contract and includes but is not limited to the following specific reasons: Failing to abide by tendering requirements• Tender declarations that are incomplete or inaccurate• Failing to abide by General Conditions of Contract• Serious issues affecting safety or the environment• The [unsatisfactory] timeliness of the completion of • the work and services The issuance of any Notice of Default• The manner of the [unsatisfactory] resolution of any • disputes and whether such disputes were resolved in accordance with the prescribed provisions of the Contract When an Infraction Report is issued, the Qualification • Committee may take no action, issue a warning letter, or reduce the Contractor’s available financial rating for a specified period (MTO 2007) Obviously, these are critical issues for contractors that wish to compete for work in Ontario, and the imposition of it is an excellent cautionary notice to review performance- based contractor prequalification programs in the context of the entire environment in which they must be implemented to ensure that they do not create an unintended impact on another vital program, such as quality management. Thus, it can be concluded that the FDOT performance- based contractor prequalification program does furnish the two aspects mentioned in the introduction to this chapter: an evaluation system with teeth and a means to restrict poor- performing contractors’ ability to win work. Considering the fact that it has been in constant use for a significant period of time, it is an excellent case study example for agencies that might wish to implement a full-scale performance-based contractor prequalification program. ONTARIO MINISTRY OF TRANSPORTATION PREQUALIFICATION PROGRAM CASE STUDY The MTO’s performance-based contractor prequalification program is perhaps the most interesting example because this agency does not require performance/payment bonds or bid deposits from prequalified contractors. Thus, the pro- gram that was started in the late 1950s not only provides an incentive for contractors to perform well, but also allows the agency to accrue a tangible monetary benefit. With an annual construction program of roughly CDN$1.4 billion and average Ontario performance/payment bond costs of 5% of contract cost (Tunistra 2008), the estimated savings to the province is roughly $70 million per year in bond costs. Thus, this system certainly bears analysis. The Ontario program essentially integrates the following three major components: An administrative qualification system that determines • financial and experiential capabilities used to compute the “contractor’s basic financial rating.” An “integrated construction performance rating sys-• tem” system that rates a contractor’s performance on a given project and is accumulated with all other projects on an annual basis to calculate an “approved contractor performance index” each year for the most recent three consecutive years. It is used to calculate a “contractor’s maximum workload rating,” which effectively reduces the amount of work that a marginally performing, but prequalified, contractor is allowed to bid. An “integrated infraction system” that allows the • owner to reduce a “contractor’s basic financial rating” by up to 100% for serious breaches of the contract. Contractor Exclusion Policy The ministry has a “contractor prequalification exclusion” provision that provides for excluding a contractor from

49 CY = current year, CY − 1 = last year, CY − 2 = year before CY − 1, and CY − 3 = year before CY − 2. Based on the CPI, a contractor is found to be in one of the three following zones: Green Zone: CPI = 70 to 100• Yellow Zone: CPI = >55 to <70• Red Zone: CPI = 35 to 55• Not Qualified to Bid: CPI = <35 • Green Zone contractors are allowed to bid on work up to their “financial basic rating” without adjustment for per- formance. Yellow Zone contractors may have to meet the contract MWL and also may have their MWL reduced by a factor of up to 20%. Finally, Red Zone contractors will have their MWL reduced by a factor “calculated linearly 20% and 100% depending on their position in the zone (20% at 55 and 100% at 35).” Table 17 shows how five contractors that fell in each of the three zones with the same “basic financial rating” and MWL would be affected by this system. Computing Maximum Workload Rating A contractor’s MWL is defined as the highest annual total dollar value of work awarded to a contractor in one of the five fiscal years (April to March) preceding the current fiscal year as shown in the following example (MTO 2007): Year 1 = $2,000,000 Year 2 = $5,000,000 Year 3 = $10,000,000 (Contractor’s MWL) Year 4 = $3,000,000 Year 5 = $6,500,000 Thus, for this specific project, each of the contractors in Table 17 must meet or exceed both the contract “financial and maximum workload” ratings to prequalify. The finan- cial basic rating is reduced by the amount of any and all ongoing work the contractor has and the MWL is reduced, if applicable, by the total value of all awarded MTO work. Contractors not meeting these requirements will not be allowed to submit a bid. Contractor B was penalized for its environmental infraction, but the penalty was not enough to keep it from prequalifying for this particular project. Addi- tionally, Contractor C, which was in the Yellow Zone, did not have its MWL reduced even though MTO could have sanctions may create a severe hardship for an Ontario-based company that must leave the province to seek work. In the case of the contractor exclusion provisions, the policy cre- ates a distinct disincentive to immediately seek redress in the court system because the MTO may not do business with a contractor that is suing it. It creates an incentive to com- plete MTO projects in a timely manner and in a fashion that is satisfactory to the MTO. Mechanics of the Prequalification System Contractors are rated in five work classifications: general road, structures, electrical, structural coating, and general maintenance. Both rated and new contractors must dem- onstrate satisfactory experience in a given classification to be awarded their full “basic financial rating.” If they do not have MTO experience, their experience with other public road agencies is considered and their “basic financial rat- ing” may be reduced in accordance with a published formula (MTO 2007). Additionally, a fully qualified contractor’s “basic financial rating” is reduced by the amount of its ongo- ing work to determine an “available financial rating.” This parallels the same theory as that used in determining bond- ing capacity in the United States. The contractor’s “available financial rating” is deter- mined when a bid is tendered and it must equal or exceed the contract rating. Contractors that have a low performance index may be required to equal or exceed the “maximum workload (MWL) rating” of the project. The “contractor performance index” (CPI) is calculated from the contractor’s previously approved “contract per- formance ratings” (CPR) for each project and is used to determine whether a contractor needs to satisfy the contract rating and the contract MWL. The CPI is calculated as fol- lows (see Eq. 2): Annual average CPR in current year minus 1, times a • weight of 3 Annual average CPR in current year minus 2, times a • weight of 2 Annual average CPR in current year minus 3, times a • weight of 1 All divided by 6• If no contracts are rated in one of the three years, then the denominator is adjusted accordingly. CPI = ((CY − 1) * 3) + ((CY − 2) * 2) + ((CY − 3) * 1) (2) (3 + 2 + 1) Where: CPI = contractor performance index for a given contractor,

50 Table 17 shows that a contractor in this system that is financially sound but does poor quality work has its ability to bid on MTO projects severely restricted by the effect of the CPI. Thus, both aspects of contractor performance-based prequalification discussed in the opening section of this chap- ter (i.e., a system with “teeth” and the ability to reward good performance and restrict poor performance) are satisfied in the MTO system. This conclusion becomes even more signifi- chosen to reduce it by up to 20%. However, the imposition of the MWL of $6 million and that it already had $1.5 million in MTO work disqualified it from further work, because this contract had a minimum MWL of $5 million. The impact on Contractor D, the Red Zone contractor, was even more severe. Finally, Contractor E, which was in the Green Zone, was excluded from bidding because of its ongoing lawsuit against MTO. TABLE 17 ExAMPLE OF SEVERAL MTO QUALIFIED CONTRACTORS UNDER A MAxIMUM WORKLOAD (MWL) RATING ADJUSTMENT Table 5.6 Part 1 Contract financial rating = $8,000,000; Contract maximum workload rating = $5,000,000 Contractor Basic Financial Rating (MWL Rating) (1) Contractor Ongoing Work (MTO work) (2) Available Rating (3) Infraction Sanctions (4) Contractor Performance Index (5) Impose Maximum Workload Rating? (6) A $12,000,000 ($6,000,000) $2,000,000 ($1,500,000) $10,000,000 0 85 = Green Zone No B $12,000,000 ($6,000,000) $2,000,000 ($1,500,000) $10,000,000 5% (Environ- mental infraction) 74 = Green Zone No C $12,000,000 ($6,000,000) $2,000,000 ($1,500,000) $10,000,000 0 *65 = Yellow Zone Yes: MWL = $6,000,000 D $12,000,000 ($6,000,000) $2,000,000 ($1,500,000) $10,000,000 0 **50 = Red Zone Yes: MWL = $6,000,000 E $12,000,000 ($6,000,000) $2,000,000 ($1,500,000) $10,000,000 15% (Safety infractions) 72 = Green Zone No Table 5.6 Part 2 Contractor MWL Reduction Adjustment (7) Exclusion? (8) Adjusted Finan- cial Rating (9) Adjusted Max- imum Work- load Rating (10) Prequalified? (11) A N/A No $10,000,000 N/A Yes $10 million > $ 8 million B N/A No $9,500,000 (5% infraction = $500K reduction) N/A Yes $9.5 million > $ 8 million C *0 No $10,000,000 $4,500,000 (MWL $6.0 million less MTO work $1.5 million) No $4.5 million < $ 5 million D **40% No $10,000,000 $2,700,000 (60% of MWL $6.0 million less MTO work $1.5 million) No: $2.7 million < $ 5 million E N/A Yes (ongoing lawsuit) Excluded N/A No: Excluded due to ongoing lawsuit * Yellow zone penalty is discretionary. MOT can impose a reduction of the MWL of up to 20%. For this example no penalty is assessed, but MWL is imposed. **Red zone = automatic 40% penalty.

51 this method is to evaluate the qualifications of interested DB contractors and develop a short-list composed of the best qualified group—that is, to prequalify the field. The RFQ stated, “The selection team shall evaluate the design-build qualifications of responding firms and shall compile a short- list of no more than five most highly qualified firms in accor- dance with qualifications criteria described in the request for qualifications” (Shane et al. 2006). To accomplish this task, it established a set of evaluation criteria and a method for scoring each potential competitor (shown in Table 18). These factors are almost identical to the performance-based contractor prequalification factors reported in the previous chapters of this report. TABLE 18 MnDOT PREQUALIFICATION EVALUATION CRITERIA Prequalification Evaluation Criteria Possible Points Legal and Financial Qualifications Pass/Fail Organization and Experience 15 Key Personnel 30 Project Understanding 10 Project Approach 25 Project Management 20 Total 100 Within the “Organization and Experience” category, the prequalification criteria were further defined to include the following: Effective project management authority and structure• Design and construction management structure• Effective utilization of personnel• Owner/client references• Experience on projects of similar scope and complexity• Experience with timely completion of comparable • projects Experience with on-budget completion of comparable • projects Experience with integrating design and construction • activities Experience of DB team members working together• These subcategories align nicely with the specific infor- mation contained on most agencies’ administrative and per- formance-based prequalification forms. The RFQ asked the contractors to both describe their specific DB experience by listing at least one completed DB project and to list other projects “with scope comparable” to the project upon which they were competing. It went on to state that DB experience would be considered but was not required. cant when one considers that fact that Ontario does not require performance or payment bonds or bid deposits from prequali- fied contractors and thus is totally dependent on this system to protect itself from contractor default. The fact that this system has been in use for some years testifies to its efficacy. Barriers to Use in the United States The major barrier to implementing this type of system in the United States will be the issue of disqualifying contractors for bringing legal action against the agency. U.S. contrac- tors, as verified by the contractor interviews, would argue that not all the lawsuits are the fault of the contractor and that standard construction contracts are crafted with the idea that the courts are the final recourse when the means provided by dispute resolution clauses fail to bring a satis- factory result. The MTO also considered these same argu- ments when implementing the exclusion policy, and for that reason, contractor exclusions are decided by the qualifica- tion committee on a case-by-case basis. Tweaking that one aspect is both doable and justifiable, and a system similar to the Ontario one could be implemented by a U.S. DOT whose enabling policies allowed it to rate contractor performance and utilize that output to adjust a given contractor’s ability to bid on its work. MTO’s system spans both general prequalification and project-specific prequalification. This is necessary because it substitutes for the bonding component of most agencies’ procurement systems. In an agency that requires bonds, the prequalification can be general, because the surety industry will regulate the amount of work a given contractor can bid on by withholding bonding when the contractor reaches its bonding capacity. In Ontario, the agency must perform the calculations shown in Table 17 to determine a given contrac- tor’s financial capacity to be awarded a given project without undue risk of financial collapse. Thus, the computation of the basic financial rating, the MWL rating function as the gen- eral prequalification factor, and the remaining calculations that determine a contractor’s eligibility for a given project based on past performance act as a de facto bonding capacity for the project. MINNESOTA DEPARTMENT OF TRANSPORTATION PREQUALIFICATION PROTEST CASE STUDY The Minnesota DOT (MnDOT) case lays down a number of rules for an agency that wants to implement performance- based contractor prequalification to abide by while devel- oping the details of its proposed system. Although this case revolved around project-specific prequalification, its results are generally applicable to any type of contractor prequalification. In this case, MnDOT chose to use a two- step best-value procurement to select a DB contractor for a $110 million project (Shane et al. 2006). The first step of

52 tion industry. In it, he highlighted the following two points that are of interest in this case: “Clearly state the evalua- tion criteria and weight given for each item and ensure that the evaluation team uses them,” and “[l]eave no doubt about the honesty and integrity of the public agency’s evaluation team” (Parvin 2000). He argued that, without a transparent evaluation plan, the owner would find itself constantly defending award protests. In this case, MnDOT clearly articulated the definition of qualification. It helped its case by stating that it would nar- row down the field to a short-list of the “most highly qualified firms in accordance with qualifications criteria described in the request for qualifications.” The requirements for prequal- ification were clear, and each potential competitor could compare itself to the competition and make an informed decision as to its ultimate competitiveness in the known field of players. This transparency serves to reduce the element of subjectivity that is inherent to best-value award evaluation systems by spelling it out rather than hiding it. Second, as the requirements for prequalification were clear, it was easy for the courts to find that MnDOT had fol- lowed its own evaluation plan. This speaks to the second part of the Parvin quote. Once an owner publishes its prequali- fication program, it loses all flexibility in applying it to the competitors that respond. If it wants to defend against a pro- test, it must follow its own rules. If it does, the second part of the Parvin quote about leaving “no doubt about the honesty and integrity … of the evaluation team” is satisfied. Finally, the use of the terms “most highly qualified” gave MnDOT great latitude about determining the final size of the short-list. The resultant scores showed that three competitors fell within a range of three points of each other, whereas the plaintiff was 10 points below the third highest score. This created a solid argument that the three competitors on the short-list were indeed the most highly qualified. MnDOT was unintentionally doing the plaintiff a favor by not being conservative in forming the short-list with four firms. The cost of preparing a technical and price proposal for a $110 million project would probably be in the range of $300,000 to $500,000 (Shane et al. 2006). If MnDOT had arbitrarily set a minimum point score of 70 to be considered “qualified” (i.e., minimally qualified), then three instead of two firms would have had to invest a significant amount of money in a losing effort. Thus, the logic of short-listing only the “most highly qualified” instead of all the minimally qualified firms is compelling in an economic sense and in the long run is fair to industry. Thus, several lessons can be learned from this case and applied to a broad-based prequalification program. First, the owner must publish transparent prequalification criteria along with its procedures for using the output of contractor performance evaluations in determining prequalification. Issues at Stake in the Protest Five contractors responded to the RFQ, submitting their qualifications in accordance with the RFQ and were eval- uated as described in that same document. Scores ranged from 69.4 to 85.7, with three contractors being rated above 80. Those three were then announced as the “most highly qualified firms” and deemed to be qualified to continue in the competition. The fourth-ranked firm with a score of 71.9 filed a protest citing the following reasons: MnDOT violated the state DB statute by requiring that • the evaluation would rely on and emphasize previous DB experience, which would restrict competition [as this was MnDOT’s first DB project and therefore, no local firms had DB experience], MnDOT engaged in unpublicized rule making, and• The judgment regarding which agencies were short-• listed was arbitrary and capricious fact-finding and the conclusions were not substantiated by the evidence (Shane et al. 2006). The court overturned the protest and upheld the valid- ity of MnDOT’s prequalification system for this specific project. First, it found that considering DB experience for MnDOT’s first DB project was entirely reasonable and, because it was considered but not required, it did not restrict competition. Second, it found that as MnDOT published the details of its qualification evaluation plan and as those rules applied only to a single project, the process did not constitute “unpublicized rule making” as alleged. Finally, it found that MnDOT had followed its procedures exactly as they were published and had a rational basis for justifying its prequali- fication decision. Therefore, the process was not “arbitrary and capricious.” Lessons Learned for Implementing Prequalification Programs This court test yielded some excellent information for agen- cies that plan to use performance-based contractor prequali- fication on a general scale. MnDOT won this case for three major reasons: The prequalification evaluation criteria were transpar-• ent to all offerors. The owner followed its prequalification plan as • published. The owner could defend its decision logically.• Transparency in prequalification programs is a key ele- ment of success. Parvin, a lawyer who defends construction contractors, wrote an insightful article regarding DB evalu- ation planning for highway construction projects (Parvin 2000). The article discussed the legal view of the need for fair and open evaluation processes in the transporta-

53 This puts all the contractors on an even footing and makes the defense against a possible protest stronger. Second, once published, the owner must follow its prequalification proce- dures to the letter, collecting documentation along the way to prove that the decisions made for the project flow directly from the published evaluation plan and its attendant criteria. Finally, the performance-based contractor prequalification program must be logical and the decisions that flow out of it must also be based on defensible logic. SUMMARY The study of the three agency performance-based contractor prequalification programs and the MnDOT protest case lead to a number of conclusions about implementing these types of processes. First, a direct link of some sort between the contractor evaluation system and prequalification is essen- tial. Two of the three case study agencies (FDOT and MTO) made that link automatic via a mathematical adjustment based on the contractor’s record of rated performance. The third one (MDOT) used the evaluations to trigger an investi- gation and potential adjustment to the contractor’s prequali- fication status. In chapter three, the interviewed contractor indicated that the state in which it did business assembled evaluations but that those ratings did not affect its ability to pursue new work with that DOT. Thus, the interviewee indicated that the process was “tedious” and provided little incentive to perform above minimum requirements. There- fore, a successful prequalification program should include an evaluation system that results in a positive or negative impact that is commensurate with contractor project performance. Next, that all three cases used their evaluation program to restrict the amount of work a contractor could effectively pursue by adjusting their financial capacity, demonstrates that this approach furnishes a good means to limit poor con- tractors and reward good contractors. Thus, manipulating the amount of work a given contractor can be awarded in proportion to its performance record seems to be an effec- tive means to influence contractor behavior. Additionally, FDOT showed it was also possible to create incentives to exceed the minimum requirement by awarding bonus points in its evaluation system and by varying the calcu- lated “ability factor” between a lower and upper level of acceptable performance. Finally, the MnDOT protest case showed that no matter how an agency decides to implement performance-based contractor prequalification, it must do so with an eye to being able to defend its process in court. MnDOT won its case because of the following three reasons: They published their entire system, thus making it • transparent to all competitors, They followed their system exactly as it was published, • and They could justify and defend the logic of their deci-• sion making process. Therefore, no matter what the mechanics of a given performance-based contractor prequalification program, it should be transparent, applied consistently as published, and follow a defensible train of logic in the decisions it makes.

54 CHAPTER SIx CONCLUSIONS AND SUGGESTED RESEARCH INTRODUCTION The objective of this synthesis was to answer the following question: Can performance-based construction contractor prequalification be implemented in a manner that rewards good contractors, encourages poor contractors to improve their performance to remain competitive in the industry, and adds value to the completed construction project? This question can be subdivided into three questions and answered individually: Can performance-based construction contractor • prequalification be implemented? This report found that prequalificiation has been imple- mented successfully by both the MTO and FDOT, as well as by another 12 U.S. DOTs and two Canadian MOTs that responded to the survey. Although chapter four discusses potential barriers to those that have not yet implemented performance-based construction con- tractor prequalification, the contractor interviews and the survey responses indicated little perceived opposi- tion to the idea. No U.S. DOTs indicated that they had laws or regulations that would preclude implementation and only three believed that the construction industry would oppose it, which was validated by the contrac- tor interviews in which 9 of 10 interviewees felt their industry supported the idea. Can performance-based construction contractor • prequalification be implemented in a way to reward good contractors and encourage poor contractors to improve performance? The FDOT ability factor shown in Figure 25 and Table 16 is designed to reward a contractor’s past perfor- mance by giving a higher ability factor for those that have exceeded the minimums than those that have met the minimums. The MTO has successfully imple- mented and sustained a system that encourages good performance by granting a workload rating based on its contractor performance index shown in Table 17. Can performance-based construction contractor • prequalification add value to the completed construc- tion project? Both the survey and the contractor interviews found that implementing performance-based contractor prequalifica- tion was perceived as having a positive impact on the qual- ity of various project aspects, thereby enhancing the value of that project. Some of the potential perceived benefits that are indicated by both the agency and contractor responses include enhanced quality, safety, timely completion, and cooperation with the agency (see Table 11). MTO is so rigor- ous with its prequalification process that it is able to do away with the performance bond requirement for the contractor. CONCLUSIONS Given the answers to the above questions, the following con- clusions were reached in the synthesis based on the proce- dure for establishing conclusions cited in chapter one. Conclusion 1—Transition to performance-based • prequalification When comparing the results of the survey and the prequalification form content analysis, many factors used in performance-based prequalification were identified in the administrative prequalification process. Figure 26 maps the results of the two systems and graphically illustrates this conclusion. Therefore, the transition from a system of admin- istrative prequalification to performance-based prequalifi- cation will not be difficult. This is due to the fact that the agency is already adept at evaluating many of the factors and the contracting industry is familiar with furnishing the necessary information. Additionally, many contractors work in more than one state and, thus, many already have experi- ence with performance-based qualification. By comparing the two systems in Figure 26, it is clear that there are slight differences to what is asked in admin- istrative and performance-based prequalification systems, but the main conceptual relationship is clearly evident. The difference between the two prequalification systems lies with how deeply the data for each factor are processed. In administrative prequalification, the data are the governing issue and they are collected to determine responsiveness. For instance, a contractor is deemed qualified to bid by merely submitting the required past project experience information. Conversely, performance-based prequalification looks to see

55 how well the contractor performed on their past projects to determine whether a contractor can be allowed to bid. This is the fundamental difference between the two types of fac- tors, but the factors are very much alike, both requesting roughly the same information but looking at it in different lights. That being said, in those cases in which administra- tive prequalification asks for the same information as perfor- mance-based prequalification it is safe to conclude that the transition between the two would be relatively smooth. Conclusion 2—Role of bonding• Bonding did not carry the weight in both prequalifica- tion processes as the other factors listed in this research. Chapter two presented the criteria breakdown chart for the administrative prequalification form, the administrative prequalification survey results, and the performance-based prequalification survey results. Respectively, the percent- ages for bonding, sureties, and insurance were 6%, 9%, and 12%, which is much less than the sum of managerial criteria and performance criteria, at 76%, 60%, and 75%, respectively. Given the plausible explanation that agencies are expecting the bonds, sureties, and insurance to be fur- nished on a project-specific basis lead to the conclusion that bonding, sureties, and insurance criteria should be used in administrative rather than performance-based prequalifica- tion. This is validated because four U.S. and three Canadian respondents indicated that they do not currently require performance bonds on their contracts. Additionally, 23 U.S. and five Canadian survey respondents indicated that a per- formance bond was not sufficient evidence of contractor prequalification. Additionally, as most contractors’ bond- ing capacity is finite, it would also serve as a final project- specific prequalification standard when the low bidder must furnish the necessary bonds, sureties, and insurance to be awarded the final construction contract. These results also lead one to question what value performance bonds add to construction projects. Conclusion 3—Simplify performance-based • prequalification A rigorous post-project performance evaluation system can be used in place of most, if not all, the minor factors of performance-based prequalification. Chapter two presented the major factors of performance-based contractor prequali- fication as identified by the survey respondents. However, a number of commonly used minor factors received survey responses (See Appendix B for details). When the minor factors are compared with the post-project performance evaluation factors, a correlation is evident between the two sets. This leads to the conclusion that the post-project perfor- mance evaluation system can be used to cover those minor factors and simplify the performance-based prequalification system to only the major factors found in this study. In Fig- ure 27, the correlation between the two factors is mapped in the same manner as the previous figure. The first eight factors of performance-based prequali- fication directly correlate with the first nine factors of post-project performance evaluation. After careful analy- sis of the remaining performance-based prequalification factors, one can move “Performance Evaluations,” the crux of this study, to become classified as a major factor. “Capacity Factor” and “Ability Factor” can remain in the performance-based prequalification process because they are mathematical determinates calculated to quantify the contractor’s attributes and are key targets for adjustment based on the output from the contractor post-project perfor- mance evaluation system. Another aspect of simplification is standardization of the information required for prequalification across the nation. This would make it easier and less costly for contractors that work in more than one state to develop and furnish the require information. This idea was expressed by sev- eral national contractors in the structured interviews. One regional contractor mentioned the effort required to keep up with various states’ administrative prequalification, terming it “tedious.” Chapter five reports the concern of a national contractor that objected to being required to furnish infor- mation that seemingly had no impact on the prequalifica- tion decision but could disqualify a company if it failed to furnish all the required data. Therefore, standardizing the way highway agencies approach performance-based con- tractor prequalification, perhaps as detailed in conclusion 5, would accrue benefits not only for contractors by reducing the administrative burden but also for agencies by making comparisons across states possible and meaningful for con- tractors that are new to a given state.

56 as reported in the literature and the contractor interviews. MTO’s program is rigorous enough that it has done away with performance bonding altogether. The survey found that four states (Illinois, Iowa, Maryland, and New Mexico) do not require performance bonding as well. Because the cost of bonding is included in the contractor’s bid, it seems that a performance-based contractor prequalification system also FIGURE 26 Comparison between the eight major factors of administrative and performance-based prequalification. Administrative Prequalification Performance-Based Prequalification Performance Criteria• – Major project experience – Available equipment – Performance evaluations Financial Criteria• – Financial capability – Financial analysis – Capacity factor Managerial Criteria• – Past illegal behavior – Technical ability Bonding, sureties, and insurance• Performance Criteria• – Major project experience – Available equipment – Quality and Workmanship Financial Criteria• – Financial capability Managerial Criteria• – Technical ability – Past illegal behavior – Key personnel experience – Managerial Ability Bonding, sureties, and insurance• FIGURE 27 Comparison between minor factors of performance-based prequalification and the factors of post-project performance evaluation. Performance-Based Factors: Post-Project Evaluation Factors: Performance Evaluations• Timely Completion• Safety Plans• Workman’s Comp. Modifier• Claims History• Quality Assurance Plans• Environmental Plans• Traffic Control Plans• Use of DBE’s• Capacity Factor• Ability Factor• Level of Subcontracting• Bonding Capacity• Insurances• Financial Analysis• Surety Statements• Bank Statements• Timely Project Completion• Timely And Complete Submittals• Safety Program Effectiveness • Mitigate Time Overruns• Mitigate Cost Overruns• Quality Assurance Program Effectiveness• Responsiveness to Warranty Call Backs• Environmental Compliance• Proper Maintenance of Traffic• Impacts to the Traveling Public• DBE Utilization• Coord./Coop. w/Property Owners• Coord./Coop. w/Agency• Coord./Coop. w/3rd Party Stakeholders• Conformance with Contract Documents• Level of Effort Displayed on Project• Timely Punchlist Completion• Conclusion 4—Both bidding and bonding capacity • could be adjusted The performance-based contractor prequalification pro- grams detailed in the case studies all used a method that reduced a marginal contractor’s bidding capacity in various ways. The idea is to avoid “subsidizing” poor performance

57 performance-based prequalification system and placed in an administrative prequalification system that is focused strictly on the contractor’s financial condition. This would be the first tier of a three-tiered system. By eliminating any duplication of required information, it strengthens the post- project performance evaluation process and streamlines the performance-based prequalification system. The first tier includes the general evaluation of the given contractor’s bonding capacity. The factors come from the grouping used in Figure 7. Including bonding capacity in this manner furnishes an external check of contractor financial capability without solely relying on it for prequalification. As concluded in chapter two, there is value to the agency to know a given contractor’s overall ability to furnish the requisite bonds and other legal instruments. Including it in the first tier constitutes the first qualification filter against past incompe- tence. A contractor with a record of default or that is under- capitalized will have that record reflected in the amount of bonding it can bring to a given project. Finally, this process is different than the project-specific bonds, sureties, and insur- ances that will be evaluated in the third tier. This difference is necessary because a contractor whose bonding capacity is committed to ongoing work at the time of award will not be able to furnish the necessary bonds, sureties, and insurance regardless of how excellent its record of past performance. The second tier, shown in Figure 28, would consist of those performance-based prequalification factors shown in Figure 26 and would include the contractor performance evaluation system described in chapter three. The factors shown below “Managerial Ability” come from Figures 8 and 10 and represent the intersection of the major managerial factors found in the two analyses. Integrating the evaluation of the managerial factors with the past project performance evaluation data creates the performance-based prequalifi- cation component. It is consistent with the conclusion that the “soft” factors used to rate a contractor’s technical and managerial ability are more important in evaluating con- tractor performance than the “hard” factors such as bonding capacity and financial condition. Finally, if an agency needs specific qualifications for a given project, it can then move to tier three (shown in Figure 29). The third and final tier is project-specific prequalification, which would be used if the agency deemed it would add value to the selection process for a given project. In most cases, this would occur only on those projects being delivered by alter- native project delivery methods. The factors shown in this tier spring from managerial and performance factors found in the RFQ prequalification content analysis discussed in chapter two and shown in Figure 14. The third tier furnishes the ability to apply performance-based prequalification to all projects regardless of delivery method. It also shows the flex- ibility inherent in the various performance-based prequalifi- cation systems found in the study. could adjust the amount of the project for which a perfor- mance bond is required. This would create an incentive for top performers by literally reducing their bid price. In this system, the marginal contractor’s “quality subsidy” would be offset by the requirement to fully bond the project. Using the same logic, the good contractor’s bonding requirement could be adjusted to something less than the full amount. With U.S. performance bond rates running from 1% to 3% on the total (Keith 2008), it could change the low bidder. Best of all, the savings would accrue directly to the owner in lower construction costs. The states of Florida, Maine, Virginia, and Washington have all experimented with bond- ing less than the entire contract amount and this fact shows that bonding less than 100% of the contract may add value to the project. Conclusion 2 found that “Bonding” was the least impor- tant prequalification criterion. Many agencies perceive bonding as protecting the state against contractor default. However, 23 U.S. and five Canadian survey respondents indicated that a performance bond was not sufficient evi- dence of contractor prequalification. With a rigorous perfor- mance-based prequalification system in place, contractors failing to complete a project can be penalized with a reduc- tion of their performance rating or, ultimately, with removal from the bidders list. A rigorous performance-based prequalification system has more bite than a soft bonding system. As further proof of this conclusion, the MTO case study is significant because MTO has had a history of not requiring bonds from its contrac- tors. MTO’s annual construction program is about CDN$1.4 billion and the average MTO performance or payment bond covers 5% of contract cost (Tunistra 2008). Therefore, the estimated savings to the province is roughly $70 million per year in bond costs. MTO uses a three-component system: “Administrative Prequalification,” “Performance Prequali- fication,” and “Infraction System.” The agency stresses that the success of the system lies in the interrelationships of all the components with one another. Although certain components of the system can be used, the program works best when all the components are integrated. This system has saved the province construction costs, which can then be diverted to further enhance its infrastructure. Conclusion 5—Three-tiered prequalification • framework Figure 28 synthesizes the findings of the entire study. It graphically portrays a framework for performance-based contractor prequalification and shows how Conclusions 2, 3, and 4 might be implemented in a generalized fashion. Based on the results reported in chapter two, the last five prequalification factors listed in Figure 27, “Bonding Capac- ity,” “Insurances,” “Financial Analysis,” “Surety State- ments,” and “Bank Statements,” can be removed from the

58 FIGURE 28 Three-tiered performance-based prequalification process.

59 have some form of contractor evaluation in their system. Many have some form of performance-based prequalifica- tion as well. The contractor’s perspective seems to welcome implementation of this type of system because it reduces or removes the number of marginally qualified contractors ! FIGURE 29 Three-tiered framework with Tier 3 details. Conclusion 6—Low barriers to implementation• The analyses conducted in this study found that there are very few actual barriers to implementing performance-based contractor prequalification. Many highway agencies already

60 against which it has to compete. Thus, it appears that both the highway agencies and their industry partners might be receptive to this change if some of the administrative and perceived hurdles found in chapter four can be eliminated or surmounted. SUGGESTIONS FOR FUTURE RESEARCH One purpose of a synthesis is to identify the gaps in the body of knowledge so that they can be filled by future research. The study uncovered four areas for which more work is needed to furnish the necessary information for highway agencies to thoughtfully implement performance-based con- tractor prequalification. These are as follows: Determine the cost-effectiveness of a performance-• based contractor prequalification system Determine the cost-effectiveness of performance • bonds Determine the components of a performance-based • contractor prequalification system that could be stan- dardized across the nation Relate performance-based contractor prequalification • to the impact on quality assurance programs Cost-Effectiveness of a Performance-Based Contractor Prequalification System The MTO case study demonstrated a significant tangible benefit of using performance-based contractor prequalifica- tion in lieu of performance bonding. Dropping performance bonds in favor of a rigorous prequalification process, how- ever, is a drastic paradigm shift for most highway agencies. The 29 U.S. DOT respondents that currently do not use performance-based contractor prequalification will need to present hard facts to their executive managers before they can implement such a system, if they so desired. The best information to induce change is financial information that demonstrates benefits accrued by other agencies and the costs associated with achieving those benefits. The survey found that 12 U.S. DOTs are using performance-based con- tractor prequalification. The research would focus on quantifying both the costs and the potential benefits discussed above. Particular atten- tion would need to be given to personnel requirements to administer the program. For example, MTO has a dedicated group of three public employees who administer its prequal- ification program. Additionally, the impact on the number of bidders could be quantified based on the survey respondents’ and contractor interviewees’ perception that implementing this system probably would reduce the size of the quali- fied bidders list. The research would differentiate between large and small state agencies regarding costs, administra- tive personnel requirements, and the effort needed to con- duct and maintain the contractor evaluations. It could do this by using targeted case studies of both large and small states that have implemented some form of performance- based contractor prequalification to identify trends in each population. Finally, the research could seek to quantify the impact on claims and contractor-initiated change orders to determine whether those risks to the owner on every project are reduced by implementing performance-based contractor prequalification. Performance bond cost-effectiveness analysis• The Miller Act requires performance bonding on all fed- erally funded projects, which the secretary of transporta- tion can waive in certain instances (The Miller Act 2008). Therefore, solid evidence will need to demonstrate that this requirement is not adding value to public construction proj- ects to alter the requirement. The proposed research would quantify the cost of performance bonding on a national basis and compare it with the potential benefit of replacing it with a rigorous performance-based contractor prequali- fication process based on the Ontario model. The research would specifically look at contractor default rates on a state- by-state basis and seek to identify those types of public transportation projects that were at the most and least risk of default. This would allow highway agency officials a base of knowledge to selectively apply performance bonding as a risk management tool on those types of projects with the greatest risk of default while not adding unnecessary costs to those types where defaults are rare. This research could develop an algorithm to adjust a good contractor’s performance bonding requirements in the event the agency chooses to implement the system discussed in con- clusion 4. The study would start with the current systems that adjust a contractor’s bidding capacity based on performance and warp that over to one in which its bonding requirements are reduced to reward those contractors that consistently furnish projects that exceed the minimum requirements for cost, schedule, and quality. Thus, the two-pronged adjust- ment shown in Figure 28 could be implemented in cases in which a poor-performing contractor’s bidding capacity is reduced for failing to achieve the minimum requirements and a well-performing contractor is rewarded for exceeding the minimums by a reduction in the amount of bond it must furnish in its bids. Standardize administrative prequalification• Conclusion 3 found that the prequalification process could be simplified by eliminating redundancies among the admin- istrative, performance-based, and post-project performance evaluation components of the three-tiered performance-based prequalification framework shown in Figure 28. Conclu- sion 5 indicated the potential benefits of creating a standard national administrative prequalification system that could be

61 used by all U.S. highway agencies in tier one. The proposed research would focus specifically on determining the appro- priate components of the administrative prequalification in the first tier. It would seek to quantify the benefits that would be associated with fielding a standard one-tier system. These might be achieved in terms of reduced administrative burden to both contractors and agencies. The research would also determine the overhead costs that regional and national con- tractors realize to ensure that they are able to bid in their chosen markets. As these costs are passed on to the highway agencies, they would furnish the motivation for developing and implementing such a system. The research could use the input aspects of the federal CCASS as a parallel system that has been fielded and used on a national basis for several decades. Additionally, as a large proportion of highway con- struction projects are awarded to be constructed in a single season, it would look specifically at the length of the project schedule and determine whether different levels of prequali- fication need to be done for multiyear projects. Performance-based prequalification impact on • quality assurance program The surveys of both the agencies and contractors indi- cated that implementing performance-based contractor prequalification would have a positive impact on project quality. However, one of the contractor interviews reviewed in the FDOT case study indicated that a potential conflict exists between contractor evaluations and proactive con- tractor QC programs. Therefore, more research is needed to determine how performance-based contractor prequali- fication can be implemented in a fashion that realizes the perceived benefits reported in this synthesis. Additionally, because the level of subcontracting in current highway projects is high, this research needs to drop down to the subcontractor level to ensure that the potential benefits per- ceived at the general contractor level are not lost because a large amount of the actual construction is being completed by subcontractors.

62 REFERENCES Al-Gobali, K. H., and A. A. Bubshait, “Contractor Prequali- fication in Saudi Arabia,” Journal of Management in Engineering, Vol. 12, No. 2, Mar./Apr. 1996, pp. 50–54. Arizona DOT, Design-Build Procurement & Administration Policy, Arizona Department of Transportation, Phoenix, 1997. Byrd, L. G., and A. A. Grant, Prerequisites for a Successful Design/Build/Warranty Highway Construction Contract, A Report to the U.S. Department of Transportation, Fed- eral Highway Administration, Washington, D.C., 1993. www.fhwa.dot.gov/programadmin/contracts/byrd.cfm. Accessed Feb. 4, 2007. Cabana, G., G. Liautaud, and A. Faiz, “Areawide Perfor- mance-Based Rehabilitation and Maintenance Contracts for Low-Volume Roads,” Transportation Research Record 1652, Transportation Research Board, National Research Council, Washington, D.C., 1999, pp. 128–137. Contractor Performance Assessment Reporting System, PowerPoint Presentation for CPARS Seminar, Naval Sea Logistics Command, Dec. 2007. www.cpars.csd.disa. mil. Accessed Apr. 14, 2008. Delaware Code, Public Works Contracting, Title 29, Chapter 69 (29 Del. C. § 6962), 2001. DOD (U.S. Department of Defense), Construction Contrac- tor Appraisal Support System (CCASS) Policy, U.S. Department of Defense, Washington, D.C., 2007. Ellicott, M. A., Best-Value Contracting, In Proc. Area Engi- neer’s Conference, TransAtlantic Division, U.S. Army Corps of Engineers, Winchester, Va., 1994. Ernzen, J., and T. Feeney, “Contractor-Led Quality Control and Quality Assurance Plus Design-Build: Who Is Watching the Quality?,” Transportation Research Record 1813, Transportation Research Board, National Research Council, Washington, D.C., 2002, pp. 253–259. Ernzen, J., and K. “Vogelsang, Evaluating Design-Build Procurement Documents for Highway Projects How Good Are They?,” Transportation Research Record: Journal of the Transportation Research Board, No. 1761, Transportation Research Board, National Research Council, Washington, D.C., 2002, pp. 148–158. FDOT (Florida Department of Transportation), “Contrac- tor’s Past Performance Rating,” Section 13.1 in FDOT Construction Project Administration Manual, Florida Department of Transportation, Tallahassee, 2004. FDOT. Contractor’s Past Performance Rating, FDOT Form 700-010-25, Florida Department of Transportation, Tal- lahassee, 2005. FHWA (Federal Highway Administration), Contractor Quality Control Guidelines and Example Quality Con- trol Plan, Federal Lands Highway Office, Federal High- way Administration, Washington, D.C., 1998a. FHWA, FHWA Initiatives to Encourage Quality Through Innovative Contracting Practices, Special Experimental Project No. 14, Federal Highway Administration, Wash- ington, D.C., 1998b. FHWA, Use of Contractor Test Results in the Acceptance Decision, Recommended Quality Measures, and the Identification of Contractor/Department Risks, Techni- cal Advisory 6120.3, Federal Highway Administration, Washington, D.C., 2004. FHWA, Construction Management Manual, Western Fed- eral Lands Highway Division, Federal Highway Admin- istration, Vancouver, Wash., 2005. FHWA, Design-Build Effectiveness Study—As Required by TEA-21 Section 1307(f): Final Report, Federal Highway Administration, Washington, D.C., 2006. www.fhwa.dot. gov/reports/designbuild/designbuild0.htm. Accessed Aug. 30, 2006. Gransberg, D. D., and M. A. Ellicott, “Best Value Contract- ing Criteria,” Cost Engineering, Journal of AACE, Inter- national, Vol. 39, No. 6, June 1997, pp. 31–34. Gransberg, D. D., and K. R. Molenaar, “A Synthesis of Pub- lic Design-Build Source Selection Methods,” Journal of Construction Procurement, Vol. 9, No. 2, Nov. 2003, pp. 40–51. Gransberg, D. D., and K. R. Molenaar, NCHRP Synthesis of Highway Practice 376: Quality Assurance in Design- Build Projects, Transportation Research Board of the National Academies, Washington, D.C., 2008. Hancher, D. E., and S. E. Lambert, Quality-Based Prequali- fication of Contractors, Transportation Research Record: Journal of the Transportation Research Board, No. 1813, Transportation Research Board of the National Acade- mies, Washington, D.C., 2002, pp. 260–275. Hancher, D. E., S. E. Lambert, and W. Maloney, Quality Based Prequalification of Contractors, Research Report KTC-01-24/SPR-212-00-1F, Kentucky Transportation Center, Lexington, 2001. IDOT (Indiana Department of Transportation), Report on Innovative Contracting Practices, Indiana Department of Transportation, Indianapolis, 1998. Keith, R., Bond FAQ’s for Contractors, Robert Keith & Associates, 2008. www.rkabonds.com/faq.asp. Accessed Apr. 1, 2008.

63 Konchar, M., and V. Sanvido, “Comparison of U.S. Project Delivery Systems,” Journal of Construction Engineering and Management, Vol. 124, No. 6, 1998, pp. 435–444. Litkowski, K., Using Structured Interviewing Techniques, GAO/PEMD-10.1.5, GAO Program Evaluation and Meth- odology Division, Washington, D.C., 1991. McLawhorn, N., “Contractor Prequalification Quality-Based Rating,” Transportation Synthesis Report, Wisconsin Department of Transportation, Madison, 2002. Minchin, R. E., Jr., and G. R. Smith, NCHRP Web Document 38: Quality-Based Performance Rating of Contractors for Prequalification and Bidding Purposes, Transporta- tion Research Board, National Research Council, Wash- ington, D.C., 2001. Molenaar, K. R., and A. D. Songer, “Model for Public Sector Design-Build Project Selection,” Journal of Construction Engineering and Management, Vol. 124, No. 6, 1998, pp. 467–479. Molenaar, K. R., and D. E. Johnson, “Engineering the Pro- curement Phase to Achieve Best Value,” Leadership in Management Engineering, Vol. 3, No. 3, July 2003, pp. 137–141. Neuendorf, K. A., The Content Analysis Guidebook, Sage Publications, Thousand Oaks, Calif. New South Wales Department of Commerce, Contractor Prequalification and Best Practice Accreditation Scheme for Construction and Related Works, Sydney, New South Wales, Australia, 2002. Norman-Eady, S., Contractor Prequalification Program, OLR Research Report 2007-R-0596, Conn., 2007. Oppenheim, A. N., Questionnaire Design, Interviewing and Attitude Measurement, Continuum, London, 1992. Palaneeswaran, E., and M. M. Kumaraswamy, “Benchmark- ing Contractor Selection Practices in Public-Sector Con- struction,” Engineering, Construction and Architectural Management, Vol. 7, No. 3, 2000, pp. 285–299. Parvin, C., Design-Build: Evaluation and Award, Roads and Bridges, Vol. 38, No. 12, 2000, p. 12. Pathomvanich, S., F. T. Najafi, and P. A. Kopac, “Procedure for Monitoring and Improving Effectiveness of Quality Assurance Specifications,” Transportation Research Record: Journal of the Transportation Research Board, No. 1813, Transportation Research Board of the National Academies, Washington, D.C., 2002, pp. 164–172. MTO (Ontario Ministry of Transportation), Registry, Appraisal and Qualification System (RAQS), Report of the Ontario Ministry of Transportation, Toronto, Canada, 2004. www.raqsa.mto.gov.on.ca/. Accessed Mar. 23, 2008. MTO, Qualification Procedures for Contractors, CRF 07-01, Ontario Ministry of Transportation, Toronto, Canada, 2007. Russell, J. S., D. E. Hancher, and M. J. Skibniewski, “Con- tractor Prequalification Data for Construction Owners,” Construction Management and Economics, Vol. 10, 1992, pp. 117–135. Sadler, D. A., Contractor Evaluation System, PowerPoint Presentation to Southern Association of Highway and Transportation Officials, Florida Department of Trans- portation, Tallahassee, 2007. Scott, S., K. R. Molenaar, D. D. Gransberg, and N. Smith, NCHRP Report 561: Best Value Procurement for High- way Construction Projects, Transportation Research Board of the National Academies, Washington, D.C., 2006. Shane, J. S., D. D. Gransberg, K. R. Molenaar, and J. R. Gladke, “Legal Challenge to a Best-Value Procurement System,” Journal of Leadership and Management in Engineering, ASCE, Vol. 5, No. 1, Jan. 2006, pp. 1–6. Singapore Building and Construction Authority, CONQUAS 21, The BCA Construction Quality Assessment System, Singapore Building and Construction Authority, Singa- pore, 2005. SOAHR (Michigan State Office of Administrative Hearings and Rules), Classification and Rating of Bidders, Michi- gan Department of Transportation, Bureau of Finance and Administration, Lansing, 2002. www.state.mi.us/ orr/emi/arcrules.asp?type=Numeric&id=2002&subId= 2002%2D007+TP&subCat=Admincode. Accessed Apr. 3, 2008. The Miller Act, Construction Specialty Group White Paper, True North, Cedar Rapids, Iowa, 2008. www.construc- tionspecialtygroup.com/documents/the_miller_act.pdf. Accessed May 11, 2008. Thomas, H. R., and G. R. Smith, NCHRP Synthesis of High- way Practice 190: Criteria for Qualifying Contractors for Bidding Purposes, Transportation Research Board, National Research Council, Washington, D.C., 1994. TRB (Transportation Research Board), NCHRP Synthesis of Highway Practice 65: Quality Assurance, Transportation Research Board, National Research Council, Washing- ton, D.C., 1979. Tunistra, T. “Ontario Ministry of Transportation Prequalifi- cation Program,” Unpublished working paper, St. Cathe- rines, ON, Canada, Apr. 10, 2008. Weber, R. P., Basic Content Analysis, Sage Publications, Beverly Hills, Calif., 1985. Yin, R. K., Case Study Research: Design and Methods, Sage Publications, Beverly Hills, Calif., 1994.

64 Yin, R. K., Conducting Case Studies: Collecting the Evi- dence, 2004. http://72.14.203.104/search?q=cache:jHb 6y55UFJ8J:www.idt.mdh.se/phd/courses/fallstudie/ slides%2520-%2520seminarie%25202/Yin%2520-% 2520kapitel%25204%2520Rev%25203.ppt+conducting + case + studies &hl = en&gl = us & ct = clnk& cd =1. Accessed March 13, 2006.

65 GLOSSARY OF TERMS Ability factor: A qualification component based on a con- tractor’s past performance, as well as technical and man- agerial experience. Administrative prequalification: A set of procedures and accompanying forms and documentation that must be submitted to gain entry to an agency-approved bidders list. These may include evaluation of financial statements, dol- lar amount of work remaining under contract, available equipment and personnel, and previous work experience. This information may be requested on a project-by-project basis or on a specified periodic basis, such as annually. Bonding capacity: A qualification component, determined by a surety, representing the financial capabilities of a contractor to perform a project. A firm has a maximum capacity that is reduced by each project’s award value to a point at which a firm must stop bidding when this capac- ity is reached. Capacity factor: A qualification component, determined by a highway agency, representing the financial capabilities of the contractor to perform a project. It is usually an aggregate factor for the firm rather than a single project. Construction manager/general contractor (CM/GC): A project delivery method in which the owner selects the GC to act as the CM on a basis of qualifications and awards a preconstruction services contract to assist the engineer-of-record during the design phase. Once the design is complete, the subcontractor work packages are bid out and the CM becomes the GC to complete the con- struction on a guaranteed maximum price basis. This delivery method is also commonly called CM-at-Risk. Design-bid-build (DBB): A project delivery method in which the design is completed either by in-house profes- sional engineering staff or a design consultant before the construction contract is advertised. Also called the “tra- ditional method.” Design-build (DB): A project delivery method in which both the design and the construction of the project are simultaneously awarded to a single entity. Performance-based prequalification: A set of procedures that must be followed by a construction contractor to qualify to submit a bid on a construction project based on quality, past performance, safety, specialized technical capability, project-specific work experience, key person- nel, and other factors. This may be on a project-specific basis or on a specified periodic basis such as annually. The project could be delivered using traditional DBB or alternative project delivery methods. Performance bond: A financial instrument furnished by a surety that guarantees the contract will be performed and the owner will receive the facility built in substantial accordance with the contract documents. Project-specific prequalification: Contractor prequalifica- tion requirements that exist only for a single project. These normally address project technical and procure- ment factors that are considered essential for the success of a given project. They may include criteria that require the contractor to have had past experience building a cer- tain technology (i.e., seismic retrofit, intelligent transpor- tations systems, etc.) or a given project delivery method such as DB. They may extend to cover specific experi- ence for key project personnel and specific types of plant and equipment. Public-private partnership (PPP): A project delivery method based on DB in which the contractor furnishes financial equity and may also operate and maintain the project after construction. Surety: A party that assumes liability for the debt, default, or failure of duty of another.

66 ABBREVIATIONS AND ACRONYMS CCASS Construction Contractor Appraisal Support System CM/GC construction manager/general contractor CPI contractor performance index CPR contract performance rating CPPR Contractor’s Past Performance Record DB design-build DBB design-bid-build DBE disadvantaged business enterprise DOH Kentucky Department of Highways DOTs departments of transportation (U.S. state) GAO U.S. Government Accounting Office MOTs ministries of transportation (Canadian) MTO Ontario Ministry of Transportation MWL maximum workload [rating] NCHRP National Cooperative Highway Research Program NSW New South Wales (Australia) PPP public-private partnership QA quality assurance QC quality control RFQ request for qualifications SOAHR State Office of Administrative Hearings and Rule SOQ statement of qualifications

67 APPENDIx A SURVEY AND SURVEY RESULTS SURVEY NCHRP Synthesis 39-04 Performance-Based Construction Contractor Prequalification INTRODUCTION: The purpose of this questionnaire is to identify how state highway agencies use performance- based contractor prequalification and, from that baseline, identify effective practices for dissemination and use by state highway agencies to implement as part of their procurement procedures for future projects. DEFINITIONS: Administrative prequalification: A set of procedures and accompanying forms and documentation that must be followed by a construction contractor to qualify to submit bids construction projects using traditional project delivery. These include evaluation of financial statements, dollar amount of work remaining under contract, available equipment and personnel, and previous work experience. This information may be requested on a project-by-project basis or on a specified periodic basis. Performance-based prequalification: A set of procedures and backup documents that must be followed by a construction contractor to qualify to submit a bid on a construction project based on quality, past performance, safety, specialized technical capability, project-specific work experience, key personnel, and other factors. This information may be requested on a project-by- project basis or on a specified periodic basis and the project could be delivered using traditional design-bid-build or alternative project delivery methods such as design-build, construction manager/general contractor, or any other methods. Design-bid-build (DBB): A project delivery method in which the design is completed either by in-house professional engineering staff or a design consultant before the construction contract is advertised. Also called the “traditional method.” Design-build (DB): A project delivery method in which both the design and the construction of the project are simultaneously awarded to a single entity. Construction manager/general contractor (CM/GC): A project delivery method in which the owner selects the GC to act as the CM on a basis of qualifications and awards a preconstruction services contract to assist the engineer-of-record during the design phase. Once the design is complete, the subcontractor work packages are bid out and the CM becomes the GC to complete the construction on a guaranteed maximum price basis. Public-private partnership (PPP): A project delivery method based on DB in which the contractor also furnishes financial equity and may also operate and maintain the project after construction.

68 General Information: 1. U.S. state or Canadian province or country (if not from North America) in which the respondent is employed: 2. You are employed by what type of organization? State/Province Department of Transportation Other public transportation agency; name of agency:      Federal agency; name of agency:      Other; please describe: 3. What group/section do you work in? Design group/section Construction group/section Operations group/section Maintenance group/section Design-build group/section Materials group/section Contracts/procurement group/section Other, please specify:      4. What project delivery methods is your organization allowed to use? DBB CM/GC DB PPP Other; please specify:

69      5. Please check those factors that are used in each type of prequalification program that is used by your organization in the matrix below. Prequalification Type Program Factors Administrative Performance Based Prequalification required for all projects Prequalification required for selected projects only Prequalification standards are the same for all projects Prequalification standards are different by project class If the standards are different, check all the below classes that apply to your program Project Classes Project monetary size Project technical complexity Project technical content (i.e., ITS, seismic features, etc.) Project delivery method (DBB, DB, CM/GC, etc) Project location (urban vs. rural) Project environmental issues Project third-party interface issues Project traffic control issues Project quality assurance requirements Other If “Other,” please specify:           Please use space below to elaborate on any of the above responses:      Administrative Contractor Prequalification Policy Information: 6. Does your organization require some form of administrative contractor prequalification? Yes No If you answered “No,” skip to question 10. 7. Please check those prequalification factors that are used in your organization’s admin- istrative contractor prequalification program in the matrix below. Check all that apply. Administrative Prequalification Factor Administrative Prequalification Factor Financial capability Equipment and plant Calculated capacity factor from financials Technical ability Bonding capacity Calculated ability factor from financials Surety statements Past project experience Detailed financial analysis Performance evaluations Bank statements Claims history Managerial ability Past illegal behavior Insurances Other; please specify below

70 Please use space below to elaborate on any of the above responses:      8. Does your organization require administrative prequalification of subcontractors? Yes, Always Project by Project No If you answered “No,” skip to question 11. 9. Please check those prequalification factors that are used in your organization’s admin- istrative subcontractor prequalification program in the matrix below. Check all that apply. Administrative Prequalification Factor Administrative Prequalification Factor Financial capability Equipment and plant Calculated capacity factor from financials Technical ability Bonding capacity Calculated ability factor from financials Surety statements Past project experience Detailed financial analysis Performance evaluations Bank statements Claims history Managerial ability Past illegal behavior Insurances Other; please specify below Please use space below to elaborate on any of the above responses:      10. Please list those aspects of your administrative contractor prequalification program that you think are particularly effective.      Point of contact for clarification on this question, if required (name/email/phone):     Performance-Based Contractor Prequalification Policy Information: 11. Does your organization require performance bonds from construction contractors? Yes No 12. If the answer to question 9 is yes, do you view the contractor’s ability to furnish the requisite performance bond as adequate to demonstrate its qualifications to successfully complete the project? Yes No Please use space below to elaborate if you wish:     

71 13. Does your organization require some form of performance-based contractor prequalification? Yes No If you answered “No,” skip to question 16. 14. Please check those prequalification factors that are used in your organization’s performance-based contractor prequalification program in the matrix below. Check all that apply. Performance-Based Prequalification Factor Performance-Based Prequalification Factor Financial capability Past illegal behavior Calculated capacity factor from financials Performance evaluations Bonding capacity Claims history Surety statements Past project experience Detailed financial analysis Timely completion of past projects Bank statements Quality of material and workmanship Insurances Workman’s compensation modifier Managerial ability Quality assurance plans Resumes for key personnel Safety plans Professional licensing for key personnel Environmental plans Key personnel past project experience Traffic control plans Equipment and plant Level of subcontracting Technical ability Use of disadvantaged business enterprises (DBEs) Calculated ability factor from financials Other 15. Please list those aspects of your performance-based contractor prequalification program that you think are particularly effective. Point of contact for clarification on this question, if required (name/email/phone):     Contractor Performance Evaluation Policy Information: 16. Does your organization develop post-project performance evaluations for construction contractors? Yes No If you answered “No,” skip to question 24.

72 17. Please check those factors that are used in your organization’s post-project contractor performance evaluation program in the matrix below. Check all that apply. Performance Evaluation Factor Performance Evaluation Factor Level of effort displayed on the job Conformance with contract documents Proper maintenance of traffic Quality assurance program effectiveness Impacts to the traveling public Safety program effectiveness Timely and complete submittals DBE utilization Timely project completion Mitigate cost overruns Environmental compliance Mitigate time overruns Coordination/cooperation with agency Timely punchlist completion Coordination/cooperation with property owners Responsiveness to warranty call-backs Coordination/cooperation with third-party stakeholders Other Please use space below to elaborate on any of the above responses:      18. When the post-project contractor performance evaluation is completed, what is it used for? Check all that apply. Administrative prequalification Performance-based prequalification Release of retainage/final payment It is reviewed and filed, but only used informally thereafter I don’t know Other; please explain:      19. Does the contractor receive a copy of the competed performance evaluation? Yes No 20. Are you required to notify the contractor before submission if the performance evaluation is considered negative? Yes No 21. Is there an appeals process for a contractor that receives a negative performance evaluation? Yes No 22. How long do performance evaluations remain on the record? 1 year 2 years 3 years more than 3 years 23. Does a negative performance evaluation automatically disqualify a contractor? Yes No

73 Legal and Political Barriers to Implementing Contractor Prequalification: 24. Does your organization operate under laws that prohibit construction contractor prequalification? Yes No Don’t know If yes, please explain:     25. Does your organization operate under procurement regulations that prohibit construction contractor prequalification? Yes No Don’t know If yes, please explain:     26. Does your organization have internal policies that prohibit construction contractor prequalification? Yes No Don’t know If yes, please explain:     27. Is the construction industry in the area in which you operate opposed to: Administrative contractor prequalification? Yes No Don’t know If yes, please explain:     Performance-based contractor prequalification? Yes No Don’t know If yes, please explain:     Post-project performance evaluation? Yes No Don’t know If yes, please explain:      28. What is the single largest barrier to implementing performance-based contractor prequalification by your organization? There isn’t one; we are already doing it. State/local laws Organizational procurement regulations Internal organizational policies Industry opposition Public opposition Political opposition Don’t know Other; please specify:      29. In your opinion, how does the construction industry in your area of operations view performance-based contractor prequalification? They support it They are neutral to it They oppose it No opinion 30. In your opinion, how do the subcontractors in the construction industry in your area of operations view performance-based subcontractor prequalification? They support it They are neutral to it They oppose it No opinion General Questions: 31. Does your organization utilize contractor quality assurance acceptance testing on any of its projects? Yes No

74 32. If the answer to question 31 is “Yes,” do you use a performance-based prequalification process in conjunction with the contractor acceptance testing program? Yes No 33. If the answer to question 31 is “No,” would you use it if you could prequalify contractors and/or their quality assurance personnel on a performance basis? Yes No 34. Regardless of your experience with contractor prequalification, in your opinion, what impact would performance-based contractor prequalification have on the following project aspects? Project Aspect Better No Change Worse No Opinion Number of bidders Material quality Workmanship quality Safety Maintenance of traffic Level/amount of agency inspection required Timely project completion Timely construction submittal completion Timely punchlist completion Personnel experience Personnel competence Number of contractor initiated change order requests Number of claims/disputes Responsiveness on warranty call-backs Achievement of DBE goals Environmental compliance Contractor cooperation with agency Contractor cooperation with property owners Contractor cooperation with third party stakeholders Contractor cooperation with public concerns 35. Do you have an experience with either the success or failure of performance-based contractor prequalification that the researcher could use to develop a case study? Yes No 36. If the answer to question 35 is “Yes,” please provide the point of contact name, phone number, and email:      37. Please furnish us with contact information for a point of contact to whom any follow-up questions can be addressed, if necessary. We will also send an electronic copy of the final research report to this individual: Name:      Telephone:     Email:     

75 SURVEY RESULTS The charts below are consolidated responses received to the online survey. ONLINE SURVEY RESULTS NCHRP Synthesis 39-04: Performance-Based Construction Contractor Prequalification General Information: 1. U.S. state or Canadian province or country (if not from North America) in which the respondent is employed: 50 Responses 2. You are employed by what type of organization? State/Province Department of Transportation 50 100% Other public transportation agency 0 0% Federal agency 0 0% Other, please specify 0 0% Total 50 100% 3. If you answered “Other public transportation agency” or “Federal agency” to the above question, please indicate the name of your agency. 4. What group/section do you work in? Design group/section 2 4% Construction group/section 28 56% Operations group/section 0 0% Maintenance group/section 0 0% Design-build group/section 0 0% Materials group/section 0 0% Contracts/procurement group/section 15 30% Other, please specify 5 10% Total 50 100% 5. What project delivery methods is your organization allowed to use? Check ALL that apply. DBB 48 96% CM/GC 6 12% DB 27 54% PPP 14 28% Other, please specify 8 16% 6. Please check those factors that are used in your ADMINISTRATIVE prequalification program that is used by your organiza- tion in the matrix below. Prequalification required for all projects 30 71% Prequalification required for selected projects only 9 21% Prequalification standards are the same for all projects 14 33% Prequalification standards are different by project class 17 40%

76 7. If you checked “the standards are different by project class” above, check ALL the below project classes that apply to your administrative prequalification program. Project monetary size 15 65% Project technical complexity 10 43% Project technical content (i.e., ITS, seismic ) 6 26% Project delivery method (DBB, DB, CM/GC, etc) 7 30% Project location (urban vs. rural) 2 9% Project environmental issues 1 4% Project third-party interface issues 2 9% Project traffic control issues 2 9% Project quality assurance requirements 2 9% Other, please specify 7 30% 8. Please check those factors that are used in your PERFORMANCE-BASED prequalification program that is used by your organization in the matrix below. Prequalification required for all projects 8 33% Prequalification required for selected projects only 14 58% Prequalification standards are the same for all projects 4 17% Prequalification standards are different by project class 9 38% 9. If you checked “the standards are different by project class” above, check ALL the below project classes that apply to your performance-based prequalification program. Project monetary size 3 25% Project technical complexity 6 50% Project technical content (i.e., ITS, seismic features, etc.) 6 50% Project delivery method (DBB, DB, CM/GC, etc.) 7 58% Project location (urban vs. rural) 1 8% Project environmental issues 1 8% Project third-party interface issues 0 0% Project traffic control issues 2 17% Project quality assurance requirements 1 8% Other, please specify 1 8% 10. Please use this text box to elaborate on your answers to questions 6 through 9 if you need to. 16 Responses Administrative Contractor Prequalification Policy Information: 11. Does your organization require some form of administrative contractor prequalification? If you answered “No,” scroll to the bottom of this page and click submit to get to question 19. Yes 36 72% No 14 28% Total 50 100%

77 12. Please check those prequalification factors that are used in your organization’s administrative contractor prequalification program below. Check all that apply. Financial capability 28 74% Calculated capacity factor from financials 20 53% Bonding capacity 14 37% Surety statements 4 11% Detailed financial analysis 19 50% Bank statements 3 8% Managerial ability 15 39% Insurances 9 24% Equipment and plant 29 76% Technical ability 21 55% Calculated ability factor from financials 8 21% Past project experience 28 74% Performance evaluations 18 47% Claims history 5 13% Past illegal behavior 23 61% Other, please specify 6 16% 13. Please use this text box to elaborate on any of the above responses: 8 Responses 14. Does your organization require administrative prequalification of subcontractors? If you answered “No,” skip to question 17. Yes, always 6 15% Sometimes on a project-by-project basis 6 15% No 29 71% Total 41 100% 15. Please check those prequalification factors that are used in your organization’s administrative subcontractor prequalifica- tion program below. Check all that apply. Financial capability 5 42% Calculated capacity factor from financials 3 25% Bonding capacity 1 8% Surety statements 1 8% Detailed financial analysis 5 42% Bank statements 1 8% Managerial ability 4 33% Insurances 0 0% Equipment and plant 11 92% Technical ability 10 83% Calculated ability factor from financials 1 8% Past project experience 9 75%

78 Performance evaluations 5 42% Claims history 1 8% Past illegal behavior 6 50% Other, please specify 0 0% 16. Please use this text box to elaborate on any of the above responses: 6 Responses 17. Please list those aspects of your administrative contractor prequalification program that you think are particularly effective. 21 Responses 18. Point of contact for clarification to question 17 question, if required (name/email/phone): Performance-Based Contractor Prequalification Policy Information: 19. Does your organization require performance bonds from construction contractors? Yes 41 85% No 7 15% Total 48 100% 20. If the answer to question 19 is “Yes,” do you view the contractor’s ability to furnish the requisite performance bond as ade- quate to demonstrate its qualifications to successfully complete the project? Yes 21 44% No 27 56% Total 48 100% 21. Has your organization ever required less than the full contract amount to be bonded? Yes 7 18% No 31 82% Total 38 100% 22. If the answer to question 21 is “Yes,” how did that impact the number of qualified bidders for those projects? Increased the number 0 0% No effect 3 33% Decreased the number 0 0% Don’t know 3 33% Other, please specify 3 33% Total 9 100% 23. Does your organization require some form of performance-based contractor prequalification? If you answered “No,” then scroll to the bottom of this page and click submit to get to question 28. Yes 14 35% No 26 65% Total 40 100% 24. Please check those prequalification factors that are used in your organization’s performance-based contractor prequalifica- tion program. Check all that apply. Financial capability 7 44% Calculated capacity factor from financials 5 31% Bonding capacity 4 25% Surety statements 2 12% Detailed financial analysis 4 25% Bank statements 2 12% Managerial ability 7 44%

79 Resumes for key personnel 8 50% Insurances 3 19% Equipment and plant 8 50% Technical ability 10 62% Calculated ability factor from financials 2 12% Performance evaluations 6 38% Claims history 6 38% Past illegal behavior 8 50% Past project experience 10 62% Timely completion of past projects 5 31% Quality of material and workmanship 7 44% Workman’s compensation modifier 3 19% Quality assurance plans 6 38% Safety plans 6 38% Environmental plans 5 31% Traffic control plans 3 19% Level of subcontracting 1 6% Use of DBEs 4 25% Other, please specify 0 0% 25. Please use this text box to elaborate on any of the above responses: 12 Responses 26. Please list those aspects of your performance-based contractor prequalification program that you think are particularly effective. 9 Responses 27. Point of contact for clarification on question 26, if required (name/email/phone): Contractor Performance Evaluation Policy Information: 28. Does your organization develop post-project performance evaluations for construction contractors? If you answered “No,” scroll to the bottom of this page and click submit to get to question 38. Yes 31 65% No 17 35% Total 48 100% 29. Please check those factors that are used in your organization’s post-project contractor performance evaluation program. Check all that apply. Level of effort displayed on the job 19 61% Proper maintenance of traffic 28 90% Impacts to the traveling public 19 61% Timely and complete submittals 28 90% Timely project completion 31 100% Environmental compliance 29 94% Coordination/cooperation with agency 29 94%

80 Coordination/cooperation with property owners 18 58% Coordination/cooperation with third-party stakeholders 16 52% Conformance with contract documents 29 94% Quality assurance program effectiveness 26 84% Safety program effectiveness 24 77% DBE utilization 18 58% Mitigate cost overruns 10 32% Mitigate time overruns 12 39% Timely punchlist completion 14 45% Responsiveness to warranty call-backs 7 23% Other, please specify 3 10% 30. Please use this text box to elaborate on any of the above responses: 8 Responses 31. When the post-project contractor performance evaluation is completed, what is it used for? Check all that apply. Administrative prequalification 21 68% Performance-based prequalification 7 23% Release of retainage/final payment 0 0% It is reviewed and filed, but only used informally thereafter 6 19% I don’t know 1 3% Other, please specify 7 23% 32. Does the contractor receive a copy of the competed performance evaluation? Yes 25 81% No 6 19% Total 31 100% 33. Are you required to notify the contractor prior to submission if the performance evaluation is considered negative? Yes 12 39% No 19 61% Total 31 100% 34. Is there an appeals process for a contractor that receives a negative performance evaluation? Yes 22 71% No 9 29% Total 31 100% 35. How long do performance evaluations remain on the record? 1 year or less 1 3% 2 years 2 6% 3 years 7 23% More than 3 years 15 48%

81 Don’t know 4 13% Other, please specify 2 6% Total 31 100% 36. Does a negative performance evaluation automatically disqualify a contractor? Yes 1 3% No 30 97% Total 31 100% 37. Please use the text box to elaborate on any of your responses in this section: 8 Responses Legal and Political Barriers to Implementing Contractor Prequalification: 38. Does your organization operate under laws that prohibit construction contractor prequalification? Yes 0 0% No 39 83% Don’t know 3 6% If yes, please explain 5 11% Total 47 100% 39. Does your organization operate under procurement regulations that prohibit construction contractor prequalification? Yes 1 2% No 39 83% Don’t know 5 11% If yes, please explain 2 4% Total 47 100% 40. Does your organization have internal policies that prohibit construction contractor prequalification? Yes 1 2% No 39 83% Don’t know 3 6% If yes, please explain 4 9% Total 47 100% 41. Is the construction industry in the area where you operate OPPOSED to: Top number is the count of respondents selecting the option. Bottom number is percent of the total respondents selecting the option. Yes No Don’t know Administrative contractor prequalification? 6 31 9 13% 67% 20% Performance-based contractor prequalification? 5 15 26 11% 33% 57% Post-project performance evaluation? 4 24 18 9% 52% 39%

82 42. What is the single largest barrier to implementing performance-based contractor prequalification by your organization? There isn’t one; we are already doing it 7 15% State/local laws 1 2% Organizational procurement regulations 1 2% Internal organizational policies 5 11% Industry opposition 4 9% Public opposition 0 0% Political opposition 1 2% Don’t know 14 30% Other, please specify 14 30% Total 47 100% 43. In your opinion, how does the construction industry in your area of operations view performance-based contractor prequalification? They support it 4 9% They are neutral to it 9 19% They oppose it 13 28% No opinion 21 45% Total 47 100% 44. In your opinion, how do the subcontractors in the construction industry in your area of operations view performance-based subcontractor prequalification? They support it 3 6% They are neutral to it 5 11% They oppose it 9 19% No opinion 30 64% Total 47 100% General Questions: 40. Does your organization have internal policies that prohibit construction contractor prequalification? Yes 30 70% No 13 30% Total 43 100% 46. If the answer to question 45 is “Yes,” do you use a performance-based prequalification process in conjunction with the con- tractor acceptance testing program? Yes 7 22% No 25 78% Total 32 100% 47. If the answer to question 45 is “No,” would you use it if you could prequalify contractors and/or their quality assurance per- sonnel on a performance basis? Yes 11 41% No 16 59% Total 27 100%

83 48. Regardless of your experience with contractor prequalification, in your opinion, what impact would performance-based con- tractor prequalification have on the following project aspects? Top number is the count of respondents selecting the option. Bottom number is percent of the total respondents selecting the option. Better No change Worse No opinion Number of bidders 3 16 18 5 7% 38% 43% 12% Material quality 21 17 0 5 49% 40% 0% 12% Workmanship quality 30 6 0 7 70% 14% 0% 16% Safety 23 10 1 9 53% 23% 2% 21% Maintenance of traffic 18 16 1 7 43% 38% 2% 17% Level/amount of agency inspection required 10 25 2 6 23% 58% 5% 14% Timely project completion 21 15 0 6 50% 36% 0% 14% Timely construction submittal completion 18 17 0 8 42% 40% 0% 19% Timely punchlist completion 19 16 0 8 44% 37% 0% 19% Personnel experience 14 21 0 6 34% 51% 0% 15% Personnel competence 19 17 1 6 44% 40% 2% 14% Number of contractor initiated change order requests 9 25 0 9 21% 58% 0% 21% Number of claims/disputes 13 22 0 8 30% 51% 0% 19% Responsiveness on warranty call-backs 16 14 0 13 37% 33% 0% 30% Achievement of DBE goals 13 18 0 12 30% 42% 0% 28% Environmental compliance 16 19 0 8 37% 44% 0% 19% Contractor cooperation with agency 23 13 1 6 53% 30% 2% 14% Contractor cooperation with property owners 13 23 0 6 31% 55% 0% 14% Contractor cooperation with third-party stakeholders 14 22 0 7 33% 51% 0% 16% Contractor cooperation with public concerns 15 20 1 6 36% 48% 2% 14%

84 49. Please use the text box to elaborate on any of your above responses as necessary: 3 Responses 50. Do you have an experience with either the success or failure of performance-based contractor prequalification that the researcher could use to develop a case study? Yes 3 7% No 40 93% Total 43 100%

85 APPENDIx B ADMINISTRATIVE PREQUALIFICATION AND PROJECT-SPECIFIC REQUEST FOR QUALIFICATION CONTENT ANALYSIS Administrative Prequalification Form Content Analysis The findings of the formal content analysis of the administrative prequalification form for each state are shown below in six sections. Each section contains three to five questions. SECTION 1: ADMINISTRATIVE PREQUALIFICATION FORM CONTENT ANALYSIS State Obtain Prequalification Renewal of Prequalification Type of Business Alabama Yes Requested Alaska Register with the state as busi- ness and contractor 50% Payment Bond 50% Performance Bond 5% Bid Bond Arizona Yes Every year or significant change in organization Requested Arkansas Yes Every year California Register with the state as well as online survey Colorado Yes Yearly; any organizational changes Requested/DBE Connecticut Yes Requested Delaware Yes Florida Yes Requested Georgia No Hawaii Register with state, and division of labor and industrial relations, cert. of good standing w/ department of commerce and consumer affairs, tax clearance Idaho Yes Yearly Requested Illinois Yes Requested Indiana Yes Iowa Yes Every 2 years Requested Kansas Yes Kentucky Online Louisiana Register with the state 100% Payment Bond 100% Performance Bond

86 State Obtain Prequalification Renewal of Prequalification Type of Business Maine Yes Yearly Requested Maryland Doesn’t qualify at all Massachusetts Yes Michigan Yes Yearly Minnesota Doesn’t qualify at all Mississippi Register with the state Missouri Yes Requested Montana Register with state Every 2 years Requested Nebraska Yes Yearly Requested Nevada Being mailed to me New Hampshire Yes Requested New Jersey Yes Yearly Requested New Mexico Yes Yearly Requested New York Yes; post qualify letter Yearly; any organizational changes Requested North Carolina Yes Requalify every 3 years; renew yearly North Dakota Yes Yearly; before July 1 Requested Ohio Yes Oklahoma Yes Requalify every 2 years; renew yearly Requested Oregon Yes Yearly Requested Pennsylvania Yes Requested Rhode Island Yes; post qualify letter Submit docs 5 days after bid opening South Carolina Yes Needed but not found Requested South Dakota Yes Requested Tennessee Yes Yearly Requested Texas Yes Yearly Requested Utah Yes Requested Vermont Yes Yearly Virginia Yes Yearly Requested Washington Yes Yearly Requested West Virginia Yes Yearly Wisconsin Yes Requested Wyoming Yes Yearly Requested

87 SECTION 2: ADMINISTRATIVE PREQUALIFICATION FORM CONTENT ANALYSIS State Financial Statement Work Classifications/ Type of Work Sought Conflicts of Interest Any Subsidiaries? %? Alabama Required by CPA Requested Alaska Arizona Requested Requested Requested Arkansas Requested Requested California Colorado Requested Requested Requested Connecticut Requested Requested Requested Delaware Requested Requested Florida Requested Requested Requested Requested Georgia Hawaii Idaho Requested Requested Requested Illinois Requested Requested Requested Indiana Requested Requested Requested Iowa Gross quarterly wages Kansas Requested Requested Requested Kentucky Louisiana Maine Requested Requested Maryland Massachusetts Optional Requested Michigan Requested Requested Requested Minnesota Mississippi Missouri Requested Requested Montana Nebraska Requested Requested Requested Requested Nevada New Hampshire Requested Requested Requested Requested New Jersey Requested Requested Requested New Mexico Asks for annual gross receipts of firm Requested Requested

88 State Financial Statement Work Classifications/ Type of Work Sought Conflicts of Interest Any Subsidiaries? %? New York Requested Requested North Carolina Requested Requested Requested Requested North Dakota Requested Requested Requested Ohio Oklahoma Requested Requested Requested Oregon Requested Requested Pennsylvania Requested Requested Requested Requested Rhode Island Requested South Carolina Requested South Dakota Requested Requested Requested Tennessee Requested with gross annual receipts Requested Requested Requested Texas Requested Business Classification/ Work Category Requested Requested Utah Requested Requested Requested Vermont Requested Requested Requested Requested Virginia Requested Requested Requested Washington Requested Requested Requested West Virginia Requested Requested Wisconsin Requested Requested Requested Requested Wyoming Requested Requested Requested Requested

89 SECTION 3: ADMINISTRATIVE PREQUALIFICATION FORM CONTENT ANALYSIS State Workman’s Compensation Modifier Previous Experience with Organization State Licensure Experience as a Contrac- tor (prime and sub) Alabama Requested Requested Alaska Arizona Requested Requested Requested Arkansas In lieu of organiza- tional and financial information Requested; both California Colorado Requested; both Connecticut Requested Delaware Required Florida Requested; both Georgia Hawaii Idaho Requested; both Illinois Requested Indiana Requested Requested; both Iowa Requested Yes Kansas Requested; both Kentucky Louisiana Maine Maryland Massachusetts Michigan Requested Requested Minnesota Mississippi Missouri Requested In the last five years Requested; both Montana Requested Nebraska Requested Requested; both Nevada New Hampshire Requested Requested Requested; both New Jersey Requested; both New Mexico Requested Requested Requested; both New York Requested North Carolina Requested North Dakota Requested; both Ohio Oklahoma Oregon Requested Requested; both Pennsylvania Requested

90 State Workman’s Compensation Modifier Previous Experience with Organization State Licensure Experience as a Contrac- tor (prime and sub) Rhode Island South Carolina EMR requested Requested Registered with Requested; both South Dakota Requested; both Tennessee Requested; both Texas Utah Requested Required Requested; both Vermont Requested; both Virginia Washington Requested; both West Virginia Requested Required Requested; both Wisconsin Requested; both Wyoming Requested; both

91 SECTION 4: ADMINISTRATIVE PREQUALIFICATION FORM CONTENT ANALYSIS State Any Project Failures by Company or Principals Any Illegal Behav- ior, Disbarment, or Suspension for You or Principals? Major Projects for X Years? Projects Currently Working? Quality Control Plan Alabama Requested 3 Requested Alaska Arizona Requested All Arkansas Requested 3 Requested California Colorado Denial of prequal. in last 5 years; failures requested too Requested 5 Connecticut Requested; 5 years Requested; for 5 years 5 Delaware Florida Requested Requested Requested Georgia Hawaii Idaho Requested All Illinois Requested 3 Indiana Requested Requested 3 Requested Iowa Kansas Requested 2 Kentucky Louisiana Maine Requested; for 5 years Requested; for 10 years Last 6 projects All Requested Maryland Massachusetts Requested Requested; Last 3 years Last 3 projects Michigan Requested Minnesota Mississippi Missouri Requested Requested 5 Requested Montana Nebraska Requested Nevada New Hampshire Requested Requested 3 Requested New Jersey Requested Requested 4 New Mexico Requested; last 10 years Requested New York Requested Requested Last 10 projects All North Carolina Requested 3 All North Dakota Requested Requested 5 All

92 State Any project Failures by Company or Principals Any Illegal Behav- ior, Disbarment, or Suspension for You or Principals? Major Projects for X years? Projects Currently Working? Quality Control Plan Ohio Oklahoma Requested Requested 2 All Oregon Requested Requested 5 Pennsylvania Requested Requested 5 All Rhode Island South Carolina Requested Requested; past 6 years Last 10 projects South Dakota Requested Requested; past 10 years 2 Tennessee Requested Requested; 5 years All Requested Texas Requested All Utah Requested 5 Vermont Requested Requested 5 Virginia Requested 3 Requested Use a formula w/this and safety Washington Requested Requested 3 All West Virginia Requested 5 Wisconsin Requested 3 Wyoming Requested 3 All

93 SECTION 5: ADMINISTRATIVE PREQUALIFICATION FORM CONTENT ANALYSIS State Safety Program Experience of Prin- cipal Individuals List All Equipment List of Banks Used for the Last X Years? List Places of Credit for X Years? Alabama Requested Requested as part of financial statement Requested; 3 years Requested; 3 years Alaska Arizona Requested Requested Arkansas Requested California Colorado Requested Requested Connecticut Requested Requested Delaware Florida Requested Requested Georgia Hawaii Idaho Requested Requested Requested Illinois Requested Requested Indiana Requested Requested Iowa Kansas Requested Contained in finan- cial submittal Kentucky Louisiana Maine Requested; really in-depth Requested Maryland Massachusetts Optional Requested Primary bank Michigan Requested Requested Minnesota Mississippi Missouri Requested Requested Montana Nebraska Requested Nevada New Hampshire Requested Requested Primary bank New Jersey Requested Requested Current Current New Mexico Last 10 years; OSHA ratings New York Any OSHA violations North Carolina Submit NCDOT Safety Index Rating Requested Requested Requested Requested North Dakota Requested Requested Ohio

94 State Safety Program Experience of Prin- cipal Individuals List All Equipment List of Banks Used for the Last X Years? List Places of Credit for X Years? Oklahoma Requested Requested Oregon Requested Requested Pennsylvania Requested Requested Requested Rhode Island Requested Requested South Carolina Requested Requested South Dakota Requested Requested Credit amount Tennessee Requested Requested Texas Requested List 6 largest Utah Requested Vermont Requested Letter of credit Virginia Use a formula w/ this and quality Requested Washington Requested Requested West Virginia Requested Requested Line of credit letter Wisconsin Requested Requested Loan capacity Wyoming Requested Requested Note: OSHA = Occupational Safety and Health Administration.

95 SECTION 6: ADMINISTRATIVE PREQUALIFICATION FORM CONTENT ANALYSIS State List All Authorized Signers List All Surety Compa- nies for the Last X Years Bonding Agent and Amount Bonded for Affidavit Alabama Requested Requested; 3 years Requested Alaska Arizona Requested Arkansas Requested Requested California Colorado Requested Connecticut Requested Requested Delaware Requested Florida Requested Requested Georgia Hawaii Idaho Requested Not called affidavit but has to be notarized Illinois Guaranty agreement Affidavit for equipment availability; affidavit for total prequal. form Indiana Requested Iowa Kansas Requested Kentucky Louisiana Maine Requested Requested Maryland Massachusetts List largest bond granted Requested Michigan Requested Requested Minnesota Mississippi Missouri Requested; surety bond Requested Requested Montana Nebraska Ever have to pay a loss to a surety Requested Nevada New Hampshire Requested; 100% contract bond New Jersey Requested New Mexico Requested Requested New York Has surety been called in the last 5 years Requested range North Carolina Requested Noncollusion affidavit and debarment cert. North Dakota Requested

96 State List All Authorized Signers List All Surety Compa- nies for the Last X Years Bonding Agent and Amount Bonded for Affidavit Ohio Oklahoma Requested Ever defaulted surety Requested Oregon Requested Performance and pay- ment; total and agent Requested Pennsylvania Requested Rhode Island Original letters needed and total amount South Carolina Requested Current Requested South Dakota Cert. of surety Requested Requested Tennessee Requested Maximum amount Requested Texas Lost a bond in last year Requested Utah Requested Vermont Requested Both company and limit Requested Virginia Requested Requested Requested Washington Requested Requested West Virginia Requested Requested Wisconsin Requested Requested; both Requested Wyoming Requested

97 PROJECT-SPECIFIC RFQ CONTENT ANALYSIS The findings of the formal content analysis of the administrative prequalification form for each state are shown below in two sections: transportation projects and nontransportation projects. SECTION 1: TRANSPORTATION PROJECTS R F Q # A ge nc y P ro je ct T yp e P as t P er fo rm an ce K ey P er so nn el E xp er ie nc e M an ag em en t P la ns Su bc on tr ac ti ng P la n Q A /Q C P la n C ur re nt W or kl oa d Sa fe ty R ec or d F in an ci al B on di ng 1 Alaska DOT Glenn - Parks Interchange DB x x x x x x x x 2 Florida DOT Bridge Replacement DB x x 3 Colorado DOT/RTD Southeast Corridor Multi- Modal DB x x x x x x x x x 4 Arizona DOT Cortaro Interchange DB x x x x x x x x 5 Nevada DOT Reno Trans Rail Access Corridor DB x x x x x x x x 6 Minnesota DOT T.H.100 Duluth ST. DB x x x x x x 7 Washing-ton DOT SR 500 Interchange DB x x x x x x 8 North Caro- lina DOT US 64 DB x x x x x x x x 9 Florida DOT Bridge Replacement DB x x x x x 10 USACE Lock & Dam DBB-BV x x x x x x x x 11 USACE Range Road Upgrade DBB-BV x x x x x x 12 USACE Range Roads DB x x x x x x x 13 USACE Airfield Improvement DB x x x 14 NAVFAC Storm Drainage DB x x x 15 FHWA EFLHD DC Highways DB x x x x x x x 16 FHWA EFLHD Parkway DB x x x x x 17 USACE Dam DBB-BV x x x x x x x 18 North Caro- lina DOT I-380 DB x x x x x x x x 19 City: James- town, VA Shore Protection DB x x x x x x x x 20 USACE Railroad Extension DB x x x x x x x x 21 USACE Runway Extension DB x x x x x x x x x 22 USACE Paving DBB-BV x x x x x x x 23 NAVFAC Runway Extension DB x x x x x x 24 USACE Paving DBB-BV x x x x x x x

98 R F Q # A ge nc y P ro je ct T yp e P as t P er fo rm an ce K ey P er so nn el E xp er ie nc e M an ag em en t P la ns Su bc on tr ac ti ng P la n Q A /Q C P la n C ur re nt W or kl oa d Sa fe ty R ec or d F in an ci al B on di ng 25 USACE Pave Aprons DB x x x x x x x x 26 USACE Railroad Complex DB x x x x x x x 27 USACE Runway Extension DB x x x x x x 28 USACE Floodway Extension DB x x x x x 29 USACE Water line DBB-BV x x x x x 30 City: Berke- ley, CA Water line DB x x x x x 31 City: Raleigh, NC Street Improve- ment DB x x x x x x x x 32 Utah DOT Paving DB x x x x 33 City: Raleigh, NC Storm Drainage DB x x x x x x x 34 Denver Intl. Airport Regional Jet Facility DB x x x x x 35 Vancouv-er, BC Transport- Authority Bridge DB x x x x x x 36 Minnesota DOT TH 10/32 DB x x x x x x x 37 Utah DOT SR 201 DB x x x x x x 38 Minnesota DOT TH 52 DB x x x x x x x x 39 City: Phoe- nix, AZ Traffic System DB x x x x x x x 40 Alberta MOT Highway Maintenance Contract DB x x x x x x 41 Nevada Upgrade DB x x x x 42 Indiana Toll Road Fiber Optic System DB x x x x x x 43 Mass DOT Improve- ment DB x x x x x 44 Missouri DOT I-64 DB x x x x x x 45 Texas DOT SH45 South- east Turnpike DB x x x x x x 46 Virginia DOT Road Improve- ment DB x x x x x x x 47 Virginia DOT Rest Area DB x x x x x x x 48 Virginia DOT Mobility Corridor DB x x x x x x

99 R F Q # A ge nc y P ro je ct T yp e P as t P er fo rm an ce K ey P er so nn el E xp er ie nc e M an ag em en t P la ns Su bc on tr ac ti ng P la n Q A /Q C P la n C ur re nt W or kl oa d Sa fe ty R ec or d F in an ci al B on di ng 49 Virginia DOT Rte 713 Bridge DB x x x x x x x x 50 Utah DOT SR 265 DB x x x x x x 51 City: Tempe, AZ Multi-Modal Facility DB x x x x x x 52 City: Lake Oswego, OR Storm Drainage DB x x x x x x 53 Florida DOT Miami Inter- Modal Center CMR x x x x x x x x x 54 City: Oak- land, CA Intermodal Center Development DB x x x x x 55 Corpus Christi Port, [ST?] Terminal Development DB x x x x x x 56 West Vir- ginia DOT WV2 Upgrade DB x x x x x x x 57 Mississippi DOT US 90 Bridge Replacement DB x x x x x x x x 58 Washington DOT Ferry Terminal DB x x x x x 59 Port of NY & NJ Wireless System PPP x x x x x x 60 Utah DOT Precon/ Con- struction services PPP x x x x x 61 City: Tempe, AZ Tempe Trans Center CMR x x x x x x 62 South Caro- lina DOT Bridge Replacement DB x x x x x Total 62 56 41 34 36 16 30 59 60 Percent of Total 100% 90% 66% 55% 58% 26% 48% 95% 97%

100 SECTION 2: NONTRANSPORTATION PROJECTS R F Q # A ge nc y P ro je ct T yp e P as t P er fo rm an ce K ey P er so nn el E xp er ie nc e M an ag em en t P la ns Su bc on tr ac ti ng P la n Q A /Q C P la n C ur re nt W or kl oa d Sa fe ty R ec or d F in an ci al B on di ng 1 NAVFAC FY02 MCON PROJECT P4 DB x x x x x x x x 2 USACE Laboratory Building Extension DB x x x 3 University of Nebraska Project Num- ber TBD DB x x x x x x 4 NAVFAC Moderniza- tion Naval Station DB x x x x x x 5 NAVFAC Replace 70 Units Hale Moku DB x x x x x 6 University of Nebraska Walter Scott Engineer Center DB x x x x x x 7 FT. Lauder- dale County Pet Crematory DB x x x 8 NAVFAC FY 2001 Dormitory DB x x x x x x 9 City: Wheat Ridge, [ST?] Monument DB DB x x x x x 10 University of Colorado Steam and Chilled Water Plant DB x x x x x 11 City: Santa Monica, CA Bus Campus Expansion DB x x x x x 12 Federal Bureau of Prisons USP Tucson DB x x x x x x 13 Dept. of Vet- erans Affairs VA Medical Center Projects DB x x x x x 14 GSA Post Office/ Courthouse DB x x x x x x 15 USACE Pentagon Renovation DB x x x x x 16 Colorado School of Mines Dormitory DB x x x x x x x 17 University of California Dormitory DB x x x x x x x 18 University of North Carolina Dormitory DB x x x x x x 19 Madison County, Texas Courthouse DB x x x x x

101 R F Q # A ge nc y P ro je ct T yp e P as t P er fo rm an ce K ey P er so nn el E xp er ie nc e M an ag em en t P la ns Su bc on tr ac ti ng P la n Q A /Q C P la n C ur re nt W or kl oa d Sa fe ty R ec or d F in an ci al B on di ng 20 State of Florida Radio Station Renovation DB x x x x x 21 City: Mesa, AZ Warehouse DB x x x x x 22 Plumas County, CA Courthouse CMR x x x x x x x 23 USACE Housing FL DB x x x x x x x 24 USACE Lab NY DB x x x x x x 25 USAF Housing Tx DB x x x x x x 26 USACE Training Facility KS DB x x x x x x x x x 27 USACE School NM DB x x x x x 28 USACE Range complex ME DB x x x x x x 29 USACE Building Texas DBB-BV x x x x x x 30 USACE Arizona Airfield Building DB x x x x x 31 USACE Support Facility CA DB x x x x 32 USACE Dormitory AZ DBB-BV x x x x x x x 33 USAF Building HI DBB-BV x x x x x x x 34 USAF Dormitory NC DB x x x x x 35 GSA Building Office DB x x x x x x x 36 USACE Building Tx DBB-BV x x x x x x 37 USACE Army Depot DBB-BV x x x x x x x x x 38 USACE Launch Complex NM DB x x x x x x x x 39 USACE Building Ft. Bliss, Tx DBB-BV x x x x x x x x 40 USACE Construction Services Tx DBB-BV x x x x x x x x 41 USACE Bexar County Con- struction Services DBB-BV x x x x x x x x 42 USACE LA Construction Services DBB-BV x x x x x x x x x 43 Virginia State Steam Plant Renovation DBB-BV x x x x x x x x 44 Texas Tech Dormitory CMR x x x x x x x x 45 East Carolina University Dormitory CMR x x x x x x x Total 45 36 34 20 24 16 16 45 45 Percent of Total 100% 80% 76% 44% 53% 36% 36% 100% 100%