Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 23
11 chances for success and ease of implementation. Appendix E contractors work with current cost data obtained from their contains a copy of the advisory board survey. The results of subcontractors and suppliers. Therefore, the second disad- the advisory board survey are presented at the end of this vantage lies in this disconnect between the owners' and con- chapter, and comments from discussions with the advisory tractors' estimating systems. board members have been integrated into the critical analy- Life-cycle cost parameters' main advantage is that they per- sis in this chapter. mit the owner to compare the long-term advantages of com- peting proposals using an engineering economic analysis. State agencies will usually have the funding to complete new 2.4 Analysis of Best-Value Concepts construction projects because most state statutes require A thorough examination of the literature, case studies, and funds to be available for public sector contracts. For federal- solicitation documents allowed the research team to further aid contracts, the state DOT signs a project agreement certi- define and critically evaluate the best-value concepts and gen- fying that state funds will be available for the non-federal erate a series of advantages and disadvantages for each cate- share of construction costs [23 U.S.C. Section 106(b)(1)]. gory. The next sections will detail that analysis for the major However, they all have huge maintenance backlogs due to components including parameters, evaluation criteria, rating insufficient operations and maintenance funding (ASCE systems, and award algorithms. 2001, Ashley et al. 1998). Thus, it is quite logical for an agency to be willing to pay a marginally higher initial cost in exchange for reduced annual maintenance costs, extended Parameters design lives, or both. The difficulty in using life-cycle cost parameters lies in adequately defining the economic analysis Cost and developing a relatively simple set of life-cycle cost input Best-value cost parameters generally include two options: variables. Selecting arbitrary values for such important vari- initial capital costs of construction and life-cycle costs ables as the discount rate or the analysis period can have incurred after construction is complete. While best-value con- unintended consequences on the validity of the output. Thus, tracting seeks to award a project on a basis of other than low an owner who intends to use life-cycle cost parameters should bid alone, cost usually plays an important part, if not the most first complete an exhaustive analysis of its algorithm to ensure important part, of the overall decision. In effect, the non-cost that it will produce the mathematically unbiased, reliable parameters are used as a way for the owner to measure the output needed to truly make a best-value award. value of qualifications, schedule, quality, and design alternates. These must then be compared with the cost parameters to Time determine whether an increase in the project's construction cost is justified by the enhanced value brought to the project Best-value time parameters not only include direct by a particular set of non-cost parameters. It may be possible contractor-proposed schedule systems such as A+B bidding, to measure the impact of schedule, quality, and design alter- but also those methods that use lane rental and traffic control nates on the project's post-construction life-cycle cost of oper- plans to indirectly influence the contractor's proposed schedule. ations and maintenance, and thus use the other type of cost Best-value time parameters can be objectively assessed based on parameter as the performance metric to assess the long-term cost by converting a time saving to user delay costs. However, value of a particular proposal. Some of the non-cost parame- these conversions are not yet universally accepted. The major ters cannot be measured on either a capital cost or life-cycle advantage of best-value time parameters is allowing the con- cost basis, but the owner will include them based on the tractor to establish a schedule that is complementary to the plan owner's perception of value to the project. for executing the construction. These parameters also reward a Cost parameters' greatest advantage in the best-value deci- contractor who proposes an aggressive schedule by making sion is that they are inherently objective. Often, the proposed the final best-value award on a combination of both price bid price can be used to determine the contractor's under- and time, thereby allowing the price to rise as the schedule is standing of the magnitude of the actual scope of work. Thus, reduced. Both lane rental and traffic control systems permit the an unrealistically low bid, while appearing to be a real bargain, owner to communicate the need to minimize a project's impact may in fact result from the bidder's lack of competence to on the traveling public during construction. These parameters successfully complete the given project. This may also be cost create an incentive toward innovative management of conges- parameters' greatest disadvantage in that public owners must tion in work zones and reductions in detour lengths and times have great justification to reject a bid that is unrealistically low. by rewarding the proposal that minimizes impact on traffic flow Additionally, public owners usually work with historic cost during construction. Their disadvantage is in the selection of data such as statewide bid averages, whereas construction lane rental rates and other factors to price user construction
OCR for page 24
12 costs. If the state highway agency is not careful when estab- The disadvantage associated with qualifications parameters lishing rates for these variables, a bias can unintentionally be mainly concerns the possibility of creating barriers to contrac- created, sacrificing construction product quality to avoid oner- tors who wish to participate in the competition but who cannot ous lane rental charges if a planned activity gets behind sched- meet the narrow or unrealistically restrictive qualification ule. One alternative (or supplement) to this approach is to requirements. This leads to potential accusations of favoritism, consider the proposer's plan for reduction of traffic impacts as bid protests, and possible political difficulties during construc- part of the proposal evaluation. tion. However, these concerns can be minimized by making the qualifications parameters match the project's specific require- ments and ensuring that the best-value award system is pub- Qualifications lished and totally transparent to industry (Parvin 2000). Best-value qualifications parameters allow the public owner to obtain some of the benefits from the historically Quality accepted practice of a Brooks Act, Qualifications-Based Selec- tion (QBS) used for procurement of design profession con- The major advantage of using best-value quality parameters tracts. The common criticism of the traditional is the ability to review and rate contractor quality management design-bid-build award to the low bidder, whether justified or plans before the contract is awarded. This has the potential to not, is that any contractor that can produce a bid bond can change the whole dynamic of quality management from an bid on a project, and anyone who can post a performance adversarial, compliance-based system to a competitive, award- bond can perform the contract regardless of past perform- to-the-best-plan system. Coupling this with some form of ance and professional qualifications. State agencies often use warranty or performance-based acceptance indicator creates a general past performance and experience criteria in their pre- situation where the focus of the proposal is toward delivering qualification procedures to determine whether a contractor is quality. Contractors will have an incentive to deliver the quality qualified to bid. By using specific qualifications parameters in as promised if they will likely be judged on this performance in the selection process, the public agency can filter out unqual- future projects. Some of the case studies actually put an ified contractors and can consider the contractor's past per- extended warranty pay item in the bid form, thus creating an formance record, thereby increasing the probability that the environment that communicates the owner's willingness to pay project will be completed successfully (Gransberg and Ellicott for the desired level of quality. One concern regarding this 1996). However, the key to public sector application of qual- approach is that various factors may affect the functional abil- ifications parameters in a bid is the use of these parameters in ity of the owner to enforce an extended warranty after con- the selection process. Their application must be justifiable struction is complete.Warranties will be formed with restrictive and defensible. and exclusionary language that the owner's facility operators Public agencies have used a broad range of evaluation must understand to avoid unintentionally invalidating the war- criteria that fall within the best-value qualifications param- ranty through some error or omission. However, quality param- eter. The first advantage of using best-value qualification eters included in the RFP and enhancements included in the parameters is the ability to restrict competition to contrac- contractor's proposal would become part of the final construc- tors who have a proven track record of successfully complet- tion contract and are therefore enforceable through standard ing a specific type of highway construction project, ensuring contractual procedures. However, it should be noted that some that all bidders will have the technical skills and experience agencies prefer not to include the proposal as a contract docu- to produce a high quality product. Additionally, by not forc- ment, because of concerns that it may not fully meet all RFP ing bidders to compete with less qualified contractors, the requirements. This concern can be addressed through appro- owner will also receive a bid price that accurately reflects the priate contract language making it clear that the RFP prevails in scope of work and adequately compensates the contractor the event that the proposal is noncompliant. If the proposal is for assuming the project risk. This reduces the probability of not included in the contract documents, it is highly advisable to a bid error and its attendant repercussions with respect to make sure that any features of the proposal that were the basis quality and timely completion. The third advantage is the of the selection decision are incorporated into the contract, so ability of the owner to influence the general contractor's that the contracting agency obtains the benefit of the bargain. subcontracting plan by elevating the importance of small business participation. Thus, a contractor may increase its Design Alternates potential to win the best-value contract by teaming with small business subcontractors. The final advantage is the Design criteria are a component of many best-value pro- ability to review and rate contractor project management curements, particularly when highway agencies are soliciting plans before the contract is awarded. bid alternates under design-bid-build or using a design-build
OCR for page 25
13 delivery method. Design alternates have advantages and The success of bid alternates depends on the use of proven disadvantages, depending on the delivery method. designs specified by the owner that can be evaluated for life- The major disadvantage of using best-value design alternate cycle costs on a reasonably equivalent basis. parameters for design-bid-build projects relates to design In summary, public agencies can create a set of potential liability considerations. In design-build, the owner sheds most variations on the theme of best value that is equal to the num- of the design liability and transfers it to the design-build ber of statistical combinations that can be developed using contractor who becomes a single point of responsibility for two or more of the above best-value parameters. Looking at both design and construction issues. However, when an owner the case study projects, it is apparent that agencies have been only allows a narrow amount of contractor-determined design experimenting with these variations in recent years. The case scope, the responsibility for coordinating the contractor- studies provide examples of agencies applying anywhere from proposed elements of work with the rest of the owner- two to eleven best-value parameters to a procurement designed construction project becomes less clear. from any or all of the best-value parameter categories. One One advantage of requesting design alternates is that it opens conclusion emerging from this experience with best-value the door to potentially innovative design solutions for a specific parameters is that the owner should customize the parameters design problem. Sometimes the design alternate could be a for the needs of the given project rather than strive to find a one- better material or a more efficient construction process. At size-fits-all standard system. To do otherwise would probably other times, it could take advantage of a drop in the cost of a reduce the effectiveness of the project delivery system and desirable material or system. In both cases, the construction create a procurement environment where minimal value, if contractor who is aware of the latest developments in materials any, could be accrued. In this vein, public agencies should also and technology in its section of the industry will usually be in a keep in mind that in many cases the tried and true design- better position to turn a design alternate into a timely advantage bid-build and low-bid award system may indeed be the best for a public agency's project. delivery method for a specific project. Highway agencies have experimented with alternate bids for specific materials, construction items, or pavement types with Evaluation Criteria some success and evaluated the value received in terms of life- cycle cost analysis. The State of Missouri experimented with five After defining the best-value parameters for a project, the competitively bid pilot projects in 1996 using portland cement agency must create an evaluation and award plan. This eval- concrete and asphaltic concrete pavement alternates. The spec- uation plan will involve determining best-value evaluation ifications for these projects included an adjustment factor added criteria from the previously mentioned parameters, defining to each asphalt concrete bid to reflect higher future reha- evaluation criteria rating systems, and defining a best-value bilitation costs during the chosen 35-year design period. For award algorithm. example, based on historical records, the asphalt pavement Best-value evaluation criteria include those factors, in addi- would need rehabilitation at 15 and 25 years versus 25 years for tion to price, that add value to the procurement. Evaluation concrete. Certain assumptions were made regarding the design criteria vary on each project as illustrated in the detailed case life (35-year analysis period), future construction and mainte- studies in Appendix D. In addition to the detailed case nance costs, salvage values, and the discount rate, to calculate the studies, the research team summarized the best-value and eval- life-cycle costs for each alternative for an equivalent analysis uation criteria from 50 RFPs as shown in Table 2.2, which illus- period. Of the five projects let, the low bidders used asphalt for trates the additional information gleaned from the analysis of three projects and concrete for two projects (Missouri 1994). best-value RFPs collected during Phase I of this study. Those The findings reported by Missouri indicated that solicitation documents included both vertical (building) projects and horizontal (transportation/utility) projects. The population · Alternate bids were in line with comparable projects and concentrated on design-bid-build/best-value RFPs specifically, engineering estimates, and provided a savings through but as best-value contracting is in its infancy in highway con- increased competition. struction, the population also looked at design-build projects to · The asphalt and concrete industry questioned the assump- find those types of evaluation criteria that would easily be trans- tions made regarding the expected design life, maintenance lated to design-bid-build/best-value contracts. The vertical proj- expenditures, pavement thicknesses, and rehabilitation needs ects were surveyed for the same reason. It can be seen that most to create a level playing field between the two alternates. of the criteria fit into one of the best-value parameter definitions. · The state determined that life-cycle costs for the pavement Public agencies also must include regulatory evaluation criteria alternatives need to be further refined to ensure that com- to comply with their local procurement law constraints. parisons are made on an equivalent basis and all future Additionally, the team conducted interviews, surveys, and costs are taken into account. case studies associated with the International Construction
OCR for page 26
14 Table 2.2. Summary evaluation criteria as identified with best-value parameter from case study project population. Evaluation Criteria Number of Contracts Using Evaluation Criteria (Total = 50) (1) Best-Value Parameter (2) Price Evaluation A.0 42 Low Bid A.0 7 Life-Cycle Cost A.1 2 Project Schedule Evaluation B.0 19 Traffic Maintenance B.2 3 Financial & Bonding Requirements P.0 35 Past Experience/Performance Evaluation P.1 44 Safety Record (or Plan) P.1 25 Current Project Workload P.1 17 Regional Performance Capacity (Political) P.1 4 Key Personnel & Qualifications P.2 41 Utilization of Small Business P.3 30 Subcontractor Evaluation/Plan P.3 29 Management/Organization Plan P.4 31 Construction Warranties Q.0 11 Construction Engineering Inspection Q.2 1 Construction Methods Q.3 1 Quality Management Q.4 27 Proposed Design Alternate & Experience D.0 26 Mix Designs & Alternates D.0 2 Technical Proposal Responsiveness D.1 37 Environmental Protection/Considerations D.1 25 Site Plan D.1 5 Innovation & Aesthetics D.1 5 Site Utilities Plan D.1 1 Coordination D.1 1 Cultural Sensitivity D.1 1 Incentives/Disincentives I/D 4 Management Scan. This information is not shown in Table 2.2, alternates. The other 11 projects were design-bid-build proj- but is incorporated into the analysis that follows the table. ects where the agency asked for a design alternate for evalua- Looking at Table 2.2, one can see that cost and qualifications tion. Proposed environmental protection measures were also criteria are used most in all types of best-value contracts. Cost a popular aspect of design information that public agencies and qualifications criteria are used most in the international wanted to evaluate. Finally, among the design-related evalua- projects as well. Past performance, qualifications of key per- tion criteria, proposal responsiveness was the preeminent sonnel, and subcontracting/small business plans are the most criterion as would be expected. popular of the qualifications parameter criteria. Of the six The crux of communicating the requirements and selecting international projects reviewed, five used past project per- the best-value parameter for a project is in the owner's devel- formance and six considered qualifications of key personnel. opment of definitive evaluation criteria. These criteria articu- In the quality parameter group, evaluation criteria for quality late the quality, cost, schedule, and qualifications requirements management planning and warranties led the category. In the for a given project. These criteria are the basis of the best-value design parameter, criteria specifying an evaluation of techni- procurement and become the foundation for the final contract. cal proposals were used in the majority of the RFPs. The The evaluation criteria that were identified in the best-value heavy use of this criterion must be compared with the use case studies have been placed into four categories: of the "proposed design alternates" criterion to understand the amount of design detail the agencies were willing to allow · Management the contractor to apply to the project. Fifteen of the case study · Schedule projects using the "proposed design alternates" criterion · Cost were design-build projects requiring evaluation of proposed · Design Alternate
OCR for page 27
15 Each of these evaluation criteria categories corresponds to individuals who could perform the work competently even the parameters discussed in Section 2.4. Keep in mind, how- though their level of experience may be short of the arbitrary ever, that the management category includes both qualifica- mark set in the solicitation. tions and quality parameters. The past performance of the organizations is a criterion often used in prequalification and in most best-value solicitations--this is understandably the case because one of Management the reasons owners are interested in a best-value approach is A strong argument can be made that the success of the to ensure that they can select the best contractor for the job. best-value project depends on the people and organizations However, there are a number of issues associated with this cri- that are selected to execute it. This is because a well-qualified terion, and the contracting authority must carefully consider construction team with highly experienced team members how to implement it such that it is accurate and unbiased and can probably sort out the post-award technical issues regard- should evaluate the pros and cons when making the decision less of the quality and clarity of the technical requirements to use past performance in the evaluation. The federal gov- in the solicitation. Management criteria come in three gen- ernment and a number of state agencies have for many years eral varieties: maintained a database of contractor evaluations on past proj- ects and often use this resource as a means to measure the · Qualifications of the individual personnel contractor's track record. Despite certain drawbacks, this · Past performance of the organizations on the best-value appears to be the best means of assessing past performance as team it allows contractors the opportunity to appeal negative rat- · Plans to execute the project ings. However, this type of system has been accused of being resource intensive, overly subjective or biased, and subject to Many public owners include schedule in the management- challenge. Owners that do not have such systems in place may planning portion of their best-value solicitations, but because decide to address past performance by asking for evaluations it is a unique and overarching feature of the project environ- from project owners for similar projects completed by the ment, it will be dealt with individually in the next section. contractor in the recent past, often asking for specific data Individual qualifications can generally be placed into to relating to schedule, cost, and claims performance on those two broad categories. The first category is the professional specific projects. The use of these metrics can be controver- credentials held by the individuals, that is, personal creden- sial due to concerns relating to due process because the con- tials that qualify an individual to perform a specific function tractors do not have the opportunity to object to negative on a team. One obvious requirement is proper licensure in the ratings and because of concerns regarding the validity of the state in which the project will be built. This and certain other information obtained. Careful consideration should there- qualifications requirements are mandated by law and would fore be given to a decision to use such a process to ensure that have to be met even if not specifically articulated in the solic- appropriate questions are asked and that the results are both itation. However, to avoid potential misunderstandings, it is fair to the contractor and useful to the owner. good practice to publish evaluation criteria that are at least The Ontario Ministry of Transportation (MTO) in minimally responsive to legal requirements. In certain cases, Canada has developed a system to rate consultants' and con- it may be advisable to include requirements that exceed the tractors' past performance, which it began to implement in minimum legal standards. 2001. The Registry, Appraisal and Qualification System The next category of qualifications is specific experience (RAQS) is used to prequalify consultants and contractors requirements. It is critical to the success of a project for and is also used in what would be considered best-value the key members of the contractor's team to have experience selection in this report (Ministry of Transportation 2004). In building similar projects. However, in developing evaluation addition to measuring financial status, the RAQS uses criteria for personal experience, owners must not be arbitrary performance appraisals and infraction reports at the end of in setting the performance standard. For example, a require- each project (no interims) to establish an overall perform- ment for the project superintendent to have 20 years of expe- ance rating. The rating is maintained on a 3-year rolling rience working on a particular type of project or on projects average basis. Penalty adjustments are made for poor with a particular agency would probably exclude many indi- performance through an infraction process and contractor viduals who would be qualified for the job. In setting the performance rating system. The MTO's use of RAQS has experience requirements, agencies should also keep in mind enhanced their prequalification process and has allowed that seniority requirements will drive up the personnel them to completely eliminate performance bonding require- costs while reducing the competitive field of qualified candi- ments for all construction contracts--saving approximately dates, and that high seniority requirements may exclude $2 million per year (Minchin and Smith 2001).
OCR for page 28
16 The MTO's use of the performance rating is demon- they can be more readily determined in an objective manner. strated by how they rate consultants to perform construc- Experience requirements can readily be defined with refer- tion administration. These consultants are selected on a ence to years of experience, number of similar successful proj- combination of price, performance, and quality, at assigned ects, or a similar measure (Vacura and Bante 2003). Owners percentages of 20%, 50%, and 30%, respectively. The system may also establish criteria for past joint performance or expe- they have developed for conducting this assessment is called rience of the various members of the contractor's team such the Consultant Performance and Selection System (CPSS), as major subcontractors and specialty consultants. which yields a Corporate Performance Rating (CPR). The The final category of management evaluation criteria that following is a description of the consultant selection process is typically included in a best-value procurement deals with taken from the CPSS Procedures Guide (Ontario Ministry the contractor's management plans to execute the project. of Transportation 2003; See RAQS website
OCR for page 29
17 Table 2.3. Case study management evaluation criteria. Management Evaluation Criteria Best-Value Parameter (1) (2) Financial & Bonding Requirements P.0 Past Experience/Performance Evaluation P.1 Safety Record (or Plan) P.1 Current Project Workload P.1 Regional Performance Capacity (Political) P.1 Key Personnel & Qualifications P.2 Utilization of Small Business P.3 Subcontractor Evaluation/Plan P.3 Management/Organization Plan P.4 Construction Engineering Inspection Q.2 Construction Methods* Q.3 Quality Management Q.4 * Owner's specialized means and methods to achieve desired quality levels. The Ontario Government requires a quality management 3. People plan in the previously mentioned RAQS. To be qualified to 4. Partnering bid construction contracts that require a qualification rating, 5. Processes contractors are required to submit a declaration showing that 6. Internal resources they have a Quality Management System (QMS). A QMS replaces traditional quality control (QC) plans as the "quality The following website contains detailed information on component" of the MTO's qualification requirements. Con- the implementation of the CAT process and how each of these tractors who wish to remain qualified, or become qualified, to attributes are scored: bid for major MTO construction contracts must choose one http://www.highways.gov.uk/roads/705.aspx of the following approaches: Schedule · Alternative 1: Annual declaration that the QMS meets MTO's minimum requirements Developing schedule evaluation criteria for the best-value · Alternative 2: Annual declaration that the Company is cer- selection is more than just setting a contract completion date. tified to ISO 9001 quality management standard Anything that the owner knows that might have a material impact on the schedule must be disclosed in the solicitation. If the schedule is an item of competition (i.e., the owner To assist the qualifications based procurement, the High- allows the offerors to propose the schedule), definitive evalu- ways Agency in England has recently developed the CAT ation criteria must be established against which the proposal (Highways Agency 2003). The CAT is a system for contrac- evaluation panel can rate the various proposals. Schedule cri- tors' self assessment of their capabilities, which are com- teria can be categorized in four general forms: bined with a past performance rating to develop a qualification based score for procurement. The system relies · Completion criteria heavily on a company's strategic management and quality · Intermediate milestone criteria management plans to establish the ratings. The CAT is a very · Restrictive criteria structured qualifications assessment tool that was developed · Descriptive criteria in consultation with industry. The CAT was developed using principles that underpin a number of business excellence Developing completion criteria is quite straightforward. If models. The CAT considers what companies need to do to the proposal date is set, the RFPs could simply provide that be effective. The CAT relies on the following capability submittal of a proposal constitutes a commitment to com- attributes: plete by the stated date, or they could include a pass/fail requirement, such as the following statement: 1. Direction and leadership The proposal shall include a commitment to complete the 2. Strategy and planning project no later than [date].
OCR for page 30
18 However, if the owner wants to ask the proposers to that routinely produce noise levels in excess of XX decibels. consider whether it will be possible to accelerate project mile- Those activities may not take place during normal business hours of 8:00 a.m. to 5:00 p.m., Monday through Friday nor late at stones or project completion and take into account commit- night on any day of the week between the hours of 10:00 p.m. and ments to accelerate the schedule as part of the best-value 6:00 a.m. Additionally, the proposal will contain a calendar that evaluation, the RFPs for proposals will need to communicate shows those periods in which loud activities will be planned. the owner's wishes to the proposers. In addition, the evalua- Those proposals that show the fewest number of days that exceed tion plan and rating system must give schedule an appropri- the prescribed noise limit will be preferred. ate weight among all other rated categories. One way to communicate this concept is as follows: Descriptive schedule requirements are used to establish a uniform format for the proposal's schedule-related submit- Offerors shall submit their proposed completion date and a tals. The underlying concept is to put all proposals on a level critical path schedule that supports a completion no later than playing field and thus facilitate equitable evaluation. In devel- (date). Completion before that date is highly desirable, and pro- oping these criteria, the owner should seek to minimize the posals with an early completion will be given preference. "bells and whistles" on the schedule submittals reducing the submittal requirement to a stark, easy to analyze document. Intermediate milestone criteria are called for if the owner One way to do this is as follows: needs to control the pace of the project. Often these criteria can be applied to those aspects of the project's progress that The critical path schedule shall be displayed as a bar chart with are not completely controlled by either the owner or the con- no more than 50 activities. The following major milestones shall tractor, such as the need to obtain permits from outside be shown on the chart along with their associated completion agencies. Another example would be a requirement to com- date: (list of milestones such as major submittal completions, plete a portion of the project to be placed in service in construction phase completions, final acceptance, etc.). advance of completion of the entire project or to require cer- tain work to be completed before proceeding with other Additionally, the owner can include recommendations in the work, a process commonly called "phased construction." RFP to influence the approach the contractor takes to the An example of this type of performance requirement is as scheduling of the project. Table 2.4 lists different approaches to follows: schedule evaluation found in the case study data collection effort. Table 2.2 shows the public agencies that constitute the The critical path schedule shall show completion of all Phase I case study population have frequently used best-value procure- construction including receipt of all digging permits by (date). ment to accelerate completion through an A+B formula and No Phase II work will proceed until Phase I work and permits have also evaluated different approaches to traffic maintenance. have been inspected and accepted by the owner. Constraints that would prevent the contractor from being Cost able to complete as fast as possible must be disclosed and are Properly written proposal submittal requirements give the required to be included in the schedule. Items such as work owner an opportunity to obtain cost information from pro- hour restrictions, prohibition on performance of work dur- posers allowing the owner to understand the best-value con- ing specified periods of time, limitations on work on holidays, tractor's thought process in developing the proposal and to and security precautions might all be addressed. The owner obtain a competitive break-down of project costs to use may request maintenance of traffic plans as part of the pro- later in change order negotiations. Often, cost information posal and evaluate them in determining best value. An exam- required to be included in the proposal can help communi- ple of RFP language dealing with noise restrictions follows: cate the relative importance of cost in the best-value award The contractor shall minimize the use of construction means decision. Cost information can range from a simple require- and methods that require the production of loud noise levels. ment to provide a lump sum amount to a complex require- The critical path schedule shall highlight in green those activities ment to provide detailed elements of a build-operate-transfer Table 2.4. Case study schedule evaluation approaches. Schedule Evaluation Criteria Best-Value Parameter (1) (2) Project Schedule Evaluation (A+B) B.0 Project Completion B.0 Traffic Maintenance B.2
OCR for page 31
19 financing scheme. Generally, three types of cost information per site. The proposal shall contain a narrative describing the requirements and associated evaluation were found: details of the proposed landscape plan for a typical area. · Cost limitations Thus, the owner in this example is effectively telling the · Cost breakdowns contractor the price payable for a specific feature of work and · Life-cycle costs asking to be told how much quality will be provided in exchange for that fixed amount of money. Specifically, the Cost limitations include cost constraints applicable to the contractor will be competing with other bidders to furnish as project as well as cost-related goals for the project. Many much landscaping as possible for the target price. solicitations contain only a single cost criterion: the proposed Cost breakdown criteria establish a means for the owner to price. The following is a list of typical cost limitation criteria better understand the basis of the contractors' price propos- set by the owner: als and help establish the foundation on which the cost of change orders and contract modifications will be negotiated. · Maximum price Under typical unit priced contracts for highway construction, · Target price this cost breakdown is essentially provided in the bid form. As · Funds available previously stated, the price proposal is one mechanism that · Public project statutory limits the owner has to evaluate the contractor's understanding of · Type of funding the scope of work. Typically, the owner will have conducted Multiple fund sources its own estimate and will use this as a yardstick to measure the Fiscal year funding quality and completeness of each price proposal. (For federal- aid contracts, the owner's estimate will be reviewed as part of A maximum price criterion is a cost constraint that defines the price reasonableness analysis conducted for such proj- the allowable cost ceiling for the project. This criterion creates ects.) The owner may also use cost breakdown criteria to eval- a constraint on the technical scope of work. In essence, the uate the realism and reasonableness of each feature of work's proposal must be developed within the limits established by value. An example of this is shown as follows: the cost constraint, and the final proposal must not only com- ply with all the technical and schedule performance criteria The price proposal shall be broken out as shown on the Price but it must also be able to be delivered at or below the maxi- Proposal Form. To be deemed responsive, the value of each fea- mum allowable price to the owner. Thus, if the owner is pro- ture of work shall not fall outside the range of ± 5% of the inde- viding the project design, measures must be taken to ensure pendent estimate for that feature of work. If any item does, the contractor will be so informed during discussions and asked to that the design is consistent with the budget ceiling. This type justify its proposed price in greater detail in its final proposal. of criterion would be used in the fixed price-best proposal best-value award algorithm. The following is an example of this type of requirement: Best-value procurement also allows an owner to take a longer look at the project's ultimate costs and consider The final firm fixed price shall not exceed $XX,XXX. including life-cycle costs in the evaluation process, in addition to initial capital cost. Life-cycle cost criteria can be addressed A target price criterion operates in much the same man- through design alternates such as asking the contractor to ner as the maximum price criterion but is less restrictive. It propose the type of pavement it will use or through require- conveys the level of overall quality the owner desires using ments such as pricing of extended construction warranties financial rather than technical terms. Target price criteria are that "lock in" future costs of maintenance and rehabilitation. often stated as unit prices rather than lump sum amounts. Research has shown that the calculation of project life-cycle The owner uses these criteria to constrain the proposed cost is a relatively straightforward application of engineer- design alternates to proper cost levels and to help guide the ing economics (FHWA 1993). However, additional work is contractor's proposal development and to ensure that the needed to form the algorithm by which a fair and equitable proposed solution will be one that fits the owner's intent. decision can be made as to the accuracy of the calculation. These criteria all serve to make these cost limitations a part When using life-cycle cost criteria, the public owner must of the final contract. For instance, requirements relating to a be aware of the actual ability of the offerors to guarantee a target price criterion using a lump sum amount would be as specific life-cycle cost for a given project. With the tools avail- follows: able at this writing, the only means by which an owner can "lock in" a discrete value for annualized life-cycle costs is The landscaping around bridges, interchanges, and rest areas, to award a contract that includes long-term operations including sodding, trees, and plantings shall cost $XX,XXX ± Y% and maintenance, long-term maintenance, or a long-term
OCR for page 32
20 warranty. Such contracts are not common for highway proj- extended warranties. Most of the cases used some form of ects and are most likely to be used for revenue generating price evaluation beyond comparing low bids. projects such as toll roads. Proposals for a Design-Build- Maintain highway contract would include the price for the Design Alternates initial capital improvements, annual maintenance costs, and the costs of capital asset replacement necessary to ensure that Bidding of design alternates on highway construction proj- the project will meet the specified standards at the end of the ects is not a new concept, but it is not a common practice in maintenance period prior to transfer of maintenance respon- the United States. sibility to the owner. The owner would evaluate the technical Nevertheless, traditional highway construction projects and price proposals, determine which proposer offered the often contain limited requirements for design alternate com- best value based on the criteria specified in the RFPs, and ponents such as contractor-furnished/DOT-approved asphalt would award the contract to the proposer offering the best and concrete mix designs within owner established limits that value. To a significant extent, the risk that the actual costs will are created as construction submittals, and such projects can exceed their contract values is transferred to the contractor. be reviewed to determine how to factor design alternates into Such contracts typically provide for certain types of costs to a best-value procurement. In addition, there is an extensive be passed back through to the owner; contractors are gener- body of knowledge relating to evaluation of design alterna- ally opposed to accepting a total transfer of the risk except in tives for design-build projects. The only real difference the context of public-private partnerships where the private between use of design alternates for design-bid-build high- sector is granted a franchise to collect revenues. This way projects and use of design alternates for design-build approach has the advantage of tying the best-value contrac- highway projects lies in the scope of the proposed design tor financially to the actual success of the project after con- work. In the arena of best-value competitive sealed bidding, struction is complete. Thus, construction decisions will be contractors will only be asked to propose design solutions made in the context of operability and maintainability rather for a very narrow, discrete portion of the contract scope or a than merely minimizing construction cost while delivering "pre-engineered" component. The amount of design work the specified standard of quality. involved does not affect the process to be followed in evaluat- The other method that an owner can use to ensure a proj- ing the merits of the design proposal, and as a result, knowl- ect's life-cycle cost after construction completion is using edge gained in review of design-build proposals should be extended warranties, maintenance bonds, or both. The first directly transferable to evaluation of design alternatives in approach requires the contractor to come back to the project connection with competitive sealed bid procurements. to repair any defects in the project; the second would give the Table 2.6 shows typical design alternate evaluation criteria owner the right to call on the bond if project operation and that were found in the case study population. maintenance costs exceed those promised in the winning pro- posal. It should be noted that the current surety market will Best-Value Evaluation Rating Systems not support bonds longer than 5 years. Also, as time passes the owner's ability to call on either a warranty or a maintenance Public owners have used a variety of evaluation (scoring or bond will be subject to the defense that the defect was caused rating) systems. Many are quite sophisticated and some are by the owner's failure to maintain or improper use. The quite simple. All can generally be categorized into the follow- owner bears the risk in both cases that the winning contrac- ing four types of systems (see Figure 2.2): tor or surety may have gone out of business by the time a claim is made. · Satisficing (more commonly called "Go/No-Go") Table 2.5 synopsizes the cost evaluation criteria that were · Modified Satisficing found in the case study population. Life-cycle cost criteria · Adjectival Rating were only found in two of the cases, and eleven cases used · Direct Point Score Table 2.5. Case study cost evaluation criteria. Cost Evaluation Criteria Best-Value Parameter (1) (2) Price Evaluation A.0 Low Bid A.0 Life-Cycle Cost (of alternatives) A.1 Construction Warranties Q.0
OCR for page 33
21 Table 2.6. Case study design alternate evaluation criteria. Design Alternate Evaluation Criteria Best-Value Parameter (1) (2) Proposed Design Alternate & Experience D.0 Mix Designs & Alternates D.0 Technical Proposal Evaluation D.1 Environmental Protection/Considerations D.1 Site Plan D.1 Innovation & Aesthetics D.1 Site Utilities Plan D.1 Coordination D.1 Cultural Sensitivity D.1 Satisficing the number of alternatives to be evaluated. On the other hand, satisficing would not be an appropriate evaluation Satisficing is the simplest and easiest evaluation system to methodology for alternatives where the project owner wishes understand for evaluators and bidders. To use it, the evalua- to take value-added features into account. tion planner must establish a minimum standard for each and every evaluation criterion against which the proposals can be measured. This is relatively simple for certain kinds of Modified Satisficing criteria such as qualifications standards. Satisficing is often referred to as "Go/No-Go" by the industry. Modified satisficing recognizes that there may be degrees According to U.S. Army Materiel Command, the definition of responsiveness to any given submittal requirement. As a of evaluation standards is "a baseline level of merit used for result, the range of possible ratings is expanded to allow an measuring how well an offeror's response meets the solicita- evaluator to rate a given category of a proposal across a vari- tion's requirements. Standards are usually a statement of the ety of degrees. Thus, a proposal that is nearly responsive can minimum level of compliance with a requirement which must be rated accordingly and not dropped from the competition be offered for a proposal to be considered acceptable." Given due to a minor deficiency. Additionally, an offer that exceeds these minimal values, the evaluators decide whether or not the published criteria can be rewarded by a rating that indi- alternatives are acceptable. Because of its strong intuitive appeal, cates that it exceeded the standard. Modified satisfied systems satisficing has long been used as an assessment technique (Mac- usually differentiate between minor deficiencies that do not Crimmon 1968). With the satisficing method, it is possible to eliminate the offeror from continuing in the competition and successively change the minimal requirements and hence to major or "fatal" deficiencies that cause the proposal to be successively reduce the feasible set of alternatives. Numerical immediately rejected. It is important for owners to include information about values is unnecessary, but can be used just as the definition of a fatal deficiency and its consequences in the easily if the information happens to come in numerical form. solicitation. The simplest of the forms of modified satisficing Satisficing is an "all or nothing" process, thus it is not crit- that are currently in use is the "red-amber-green" system with ical to determine an accurate value for alternatives. An alter- the definitions for each rating are as follows: native is either acceptable or not acceptable. An alternative that exceeds the minimum would merely be considered · Green--fully responsive to the evaluation criteria acceptable, regardless of the amount of value added. The · Amber--not responsive, but deficiency is minor main advantage of satisficing is that it can be used to reduce · Red--not responsive due to fatal deficiency Satisficing Modified Adjectival Direct Point Satisficing Rating Scoring Simple Complex Quick Requires Analysis Bimodal Outcome Array of Outcomes Assessment Accuracy not Critical Assessment Accuracy Critical Figure 2.2. Best-value evaluation rating system continuum.
OCR for page 34
22 Table 2.7. Modified satisficing examples (USAED, New York 2002; U.S. Air Force 2001). Army Air Force Definition Definition Rating Rating Proposal meets the minimum Exceeds specified minimum performance SOLICITATION requirements for this item Dark Blue Blue or capability requirements in a way and has salient features that offer significant beneficial to the Air Force. advantages to the Government. Proposal meets the minimum SOLICITATION requirements for this item Purple N/A N/A and has salient features that offer advantages to the Government. Meets specified minimum performance Proposal meets the minimum Green Green or capability requirements necessary for SOLICITATION requirements for this item. acceptable contract performance. Proposal meets most of the minimum Does not clearly meet some specified requirements for this item, but offers weak minimum performance or capability Yellow area or mimics SOLICITATION language Yellow requirements necessary for acceptable rather than offering understanding of the contract performance, but any proposal requirements. inadequacies are correctable. Fails to meet specified minimum Proposal meets some but not all the minimum performance or capability requirements. Red requirements for this item or does not address Red Proposals with an unacceptable rating are all required criteria. not awardable. The next step in the modified satisficing evaluation process rating systems are an extension of modified satisficing. They is to roll-up the individual ratings for each evaluation crite- recognize that a more descriptive rating system is in order and rion and arrive at an overall rating for each proposal. Table 2.7 that the rating system should be continuous rather than dis- shows the approach used in solicitations from two military crete. Table 2.8 illustrates how one owner developed a series of best-value projects. One notices that both agencies use color adjectival criteria to rate different components of a proposal. coding to make it easier to identify the areas in which a par- There are three important elements of an adjectival rating ticular proposal offers advantages to the government. The system: Army distinguishes between proposals that offer advantages to the government and those that offer significant advantages, · Definitions while the Air Force provides only one category for proposals · Performance indicators that exceed the minimum requirements. · Differentiators The reader should note that the examples of modified sat- isficing evaluation systems in Table 2.7 are not examples of Each adjectival rating must have all three. The definition the standard for all projects in either of the two military must be both clear and relevant to the specific factor being departments. They were pulled from solicitations that were evaluated. It should portray to the evaluators the essence of developed specifically for the projects for which they were what the evaluation plan writer intends to be identified and written. They do furnish excellent examples of how two dif- rated. In the example provided in Table 2.8, the definition ferent owners defined the ratings that were used on two typ- provided for "Proposal Risk" indicates that evaluators are to ical projects. Additionally, the reader should note that the assess and rate the "weaknesses and strengths associated with definitions shown in Table 2.7 were published in the respec- the proposed approach as it relates to accomplishing the tive RFPs. Thus, the contractors were cognizant of the evalu- requirements of the solicitation." Following along with this ation scheme and could craft their proposals accordingly. It example, the rating will take the form of one of three adjec- should also be noted that the definition of each rating is clear tives, "high," "moderate," or "low." Each of these adjectives and offers a standard against which the evaluators can meas- is then defined in terms of a performance indicator that is ure each individual proposal. cogent to the factor that is being evaluated. The evaluators will use the indicator as a marker with which to determine the appropriate rating for the evaluated element. Again looking Adjectival Rating to the example provided in Table 2.8, the performance indi- Adjectival rating systems use a specific set of adjectives to cators associated with proposal risk include the potential to describe the conformance of an evaluated area within a disrupt the schedule, increase costs, or degrade performance. proposal to the project's requirements in that area. Adjectival To assist the evaluators with those proposals that seem to
OCR for page 35
23 Table 2.8. Example adjectival rating for three different evaluated areas (U.S. Air Force 2001). Evaluated Area Evaluation Plan Definition Adjectival Rating Proposal risk relates to the identification and assessment of the risks, PROPOSAL RISK weaknesses and strengths associated with the proposed approach as it relates to accomplishing the requirements of the solicitation. Likely to cause significant disruption of schedule, increased cost, or High degradation of performance. Risk may be unacceptable even with special contractor emphasis and close government monitoring. Can potentially cause some disruption of schedule, increased cost, or Moderate degradation of performance. Special contractor emphasis and close government monitoring will probably be able to overcome difficulties. Has little potential to cause disruption of schedule, increased cost, or Low degradation of performance. Normal contractor effort and normal government monitoring will probably be able to overcome difficulties. More recent and relevant performance will have a greater impact on the Performance Confidence Assessment than less recent or relevant effort. PERFORMANCE RECORD A strong record of relevant past performance will be considered more advantageous to the government. Exceptional Based on the Offeror's performance record, essentially no doubt exists High Confidence that the Offeror will successfully perform the required effort. Very Good Based on the Offeror's performance record, little doubt exists that the Significant Confidence Offeror will successfully perform the required effort. Satisfactory Based on the Offeror's performance record, some doubt exists that the Confidence Offeror will successfully perform the required effort. Neutral/Unknown Confidence No performance record identifiable. Based on the Offeror's performance record, substantial doubt exists that Marginal the Offeror will successfully perform the required effort. Changes to Little Confidence the Offeror's existing processes may be necessary in order to achieve contract requirements. Unsatisfactory Based on the Offeror's performance record, extreme doubt exists that No Confidence the Offeror will successfully perform the required effort. Past projects will be compared with the solicitation and those that RELEVANCY OF PAST PROJECTS involved features of work that are similar in size, scope, and technical complexity will be considered relevant. The magnitude of the effort and the complexities on this contract are Highly Relevant essentially what the solicitation requires. Some dissimilarities in magnitude of the effort and/or complexities Relevant exist on this contract, but it contains most of what the solicitation requires. Much less or dissimilar magnitude of effort and complexities exist on Somewhat Relevant this contract, but it contains some of what the solicitation requires. Performance on this contract contains relatively no similarities to the Not Relevant performance required by the solicitation. straddle two adjectival grades, differentiators are also pro- precision to evaluations, providing an appearance of objec- vided to further distinguish between the grades. In the exam- tivity even though the underlying ratings are inherently sub- ple, a "low" proposal risk is described as one for which jective. Evaluators assign points to evaluation criteria based difficulties will probably be overcome through normal con- on some predetermined scale or the preference of the evalu- tractor effort and normal government monitoring. In con- ator. Case Study 14: Maine DOT Bridge, in Appendix D, illus- trast, a "moderate" proposal risk suggests that special trates the direct point scoring system through the use of a contractor emphasis and close government monitoring will percentage defining a raw score definition as follows that is likely be needed to overcome difficulties. then translated into the final point allocation. Raw Score Definition Direct Point Scoring Direct point scoring evaluation allows for more rating lev- 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% els and thus may appear to give more precise distinctions of merit. However, point scoring may lend an unjustified air of Marginal Average Exceptional
OCR for page 36
24 Table 2.9. Direct point scoring example from Case Study 19. Standard Service Delivery Level Mark Very high standard Proposals likely to exceed all delivery targets 10 Good standard Proposals likely to meet all delivery targets and exceed some 8-9 delivery targets Acceptable standard Workable proposals likely to achieve all or most delivery 5-7 targets Poor standard Significant reservations on service delivery targets but not 1-4 sufficient to warrant exclusion of bid Not acceptable Bid excluded from further consideration 0 Some agencies use adjectival ratings as the basis of direct Proposal to meet the minimum criteria to be considered to point scoring systems. These should still be considered direct be Good, it must be determined to have strength(s), even point scoring methods, but the adjectival ratings are used to though minor and/or significant weaknesses exist. The narrow down the scoring to within ranges. Case Study 19: minimum score for Good is 70%. The greater the signifi- Forth Road Bridge Toll Equipment, in Appendix D, provides cance of the strengths and/or the number of strengths, and a simple example of a direct point scoring system that is based the fewer the minor or significant weaknesses will result in on adjectives shown in Table 2.9. a higher percentage, up to a maximum of 79%. The Pro- The Washington State DOT I405 Kirkland Stage I HOV poser demonstrates a reasonable probability of meeting the Design-Build RFP provides a more detailed direct point scor- requirements of the RFP. ing system that is based on adjectival ratings. · Non-responsive (069%): The Proposal demonstrates an approach that contains minor and/or significant weak- · Excellent (90100%): The Proposal demonstrates an nesses and no strengths. The Proposal is considered not to approach that is considered to significantly exceed the RFP meet the RFP requirements and may be determined to be requirements/objectives in a beneficial way (providing non-responsive. advantages, benefits, or added value to the Project) and provides a consistently outstanding level of quality. In The direct point scoring system or variations of it are used order for the Proposal to meet the minimum criteria to be by many transportation agencies. However, federal agencies considered to be Excellent, it must be determined to have a do not typically use such a system because the use of numer- significant strength and/or a number of strengths and no ical rating systems in conjunction with specific percentage weaknesses. The minimum score for Excellent is 90%. The weightings for the factors requires the source selection greater the significance of the strengths and/or the number authority to convert the decision-making process to a for- of strengths will result in a higher percentage, up to a max- mula without knowing what will be offered. Such a process imum of 100%. There is no risk that the Proposer would allows virtually no discretion to the selection official. fail to meet the requirements of the RFP. Direct point scoring evaluation is probably the most com- · Very Good (8089%): The Proposal demonstrates an plex best-value evaluation method. One of its weaknesses is approach that is considered to exceed the RFP require- the variation that is induced by evaluators who are assigning ments/objectives in a beneficial way (providing advantages, numerical scores to the same category. Even if the evaluators benefits, or added value to the Project) and offers a gener- are restricted to using integers, each individual will have his ally better than acceptable quality. In order for the Proposal or her own methodology for arriving at a point score. Thus, to meet the minimum criteria to be considered to be Very it becomes difficult for the owner to ensure that the evalua- Good, it must be determined to have strengths and no sig- tion system is both fair and uniformly applied to all propos- nificant weaknesses. The minimum score for Very Good is als. Fundamentally, two engineers looking at the same thing 80%. The greater the significance of the strengths and/or can probably agree on whether or not it is satisfactory or the number of strengths, and the fewer the minor weak- unsatisfactory (i.e., an adjectival rating), but getting them to nesses will result in a higher percentage, up to a maximum agree on exactly how many points a given category should be of 89%. There is very little risk that the Proposer would fail awarded will be much more difficult. For the evaluators, this to meet the requirements of the RFP. presents a psychological issue rather than a technical issue, · Good (7079%): The Proposal demonstrates an approach which is sometimes dealt with by resolving outlier scores that is considered to meet the RFP requirements/objectives through the use of adjectival ratings that are then converted and offers an acceptable level of quality. In order for the to numbers.
OCR for page 37
25 Table 2.10. Meets technical criteria--low-bid example. Technical Score (60 maximum) Offeror (40 minimum) Price Proposal 1 51 $1,400,000 2 53 $1,200,000 3 44 $1,100,000 4 39 NR Direct point scoring evaluation's greatest strength is the are evaluated before any cost proposals are reviewed. The flexibility of the scale on which each proposal is rated. If the price proposal is opened only if the technical proposal is owner does not require its evaluation panel to achieve con- found to have met the minimum requirements. The tech- sensus, but rather chooses to use an average of the individual nical proposal review can be done on a pass/fail basis or scores, direct point scoring in effect becomes an "expert sys- using numerical ratings with a predetermined minimum tem" in every sense of the computer-related definition of that score required for the proposal to be considered respon- term of art. This becomes valuable in those projects where the sive. If the proposal does not meet the minimum stan- salient aspects of the project are hard to quantify. Direct point dards, it is deemed non-responsive and the associated price scoring evaluation allows the average numerical ratings to act proposal will not be opened. The price proposals associ- as the collective expert. However, an averaging approach has ated with responsive technical proposals are then opened, the potential for allowing a single evaluator with a bias to affect often publicly, and the contract is awarded to the proposer the outcome. Some agencies eliminate the high and low scores offering the lowest price. See the following generic in order to reduce the likelihood of this type of problem. algorithm and Table 2.10. Case Studies 9, 10, and 12 in Appendix D also provide examples of the meets technical criteria--low-bid algorithm. Best-Value Award Algorithms Best-value award algorithms define the steps that owners Algorithm: If T > Tmin, Award to Pmin take to combine the parameters, evaluation criteria, and evalu- If T < Tmin, Non-responsive ation rating systems into a final award recommendation. Seven Tmin = Determination that proposal meets best-value award algorithms have been found through a com- minimum technical requirements prehensive analysis of the literature, case studies, and project P = Project Price procurement documents. Building, water/wastewater, indus- trial, and highway projects from both the public and the private sector were analyzed. The seven algorithms are as follows: Adjusted Bid The adjusted bid algorithm requires use of numerical scor- · Meets technical criteria--low bid ing (or adjectival ratings converted to numbers). Price pro- · Adjusted bid posals are opened after the technical proposals are scored. · Adjusted score When the price proposal is opened, the project price is · Weighted criteria adjusted in some manner by the technical score, typically · Quantitative cost--technical tradeoff through the division of price by a technical score from 01 or · Qualitative cost--technical tradeoff 0100. The adjusted bid is used only for project award. The · Fixed price--best proposal contract price will be based on the amount stated in the price proposal. The offeror with the lowest adjusted bid will be A description of each of these procedures follows. The awarded the project. See the following generic algorithm and algorithms are described through formulas and illustrated Table 2.11. Case Study 14 in Appendix D also provides an through generic examples. Case studies illustrating each of example of the adjusted bid algorithm. the algorithms can be found in Appendix D. Algorithm: AB = P/T Award ABmin Meets Technical Criteria--Low Bid AB = Adjusted Bid In the meets technical criteria--low-bid algorithm, the P = Project Price final award decision is based on price. Technical proposals T = Technical Score
OCR for page 38
26 Table 2.11. Adjusted bid example. Offeror Technical Score Price Proposal Adjusted Bid 1 0.85 $1,200,000 $1,411,765 2 0.95 $1,250,000 $1,315,789 3 0.90 $1,150,000 $1,277,777 4 0.70 $1,100,000 $1,571,429 Adjusted Score Algorithm: TS = W1S1 + W2S2 + ... + WiSi + W(i+1)PS Award TSmax The adjusted score algorithm also requires use of numeri- TS = Total Score cal scoring (or adjectival ratings converted to numbers). The Wi = Weight of Factor i price proposals are opened after the technical proposals are Si = Score of Factor i scored. The adjusted score is calculated by multiplying the PS = Price Score technical score by the total estimated project price and then dividing by the price proposal. Award is made to the offeror with the highest adjusted score. See the following generic Quantitative Cost-Technical Tradeoff algorithm and Table 2.12. Case Study 11 in Appendix D also The quantitative cost-technical tradeoff algorithm also provides an example of the adjusted score algorithm. requires use of numerical scoring (or adjectival ratings con- Algorithm: AS = (T x EE)/P verted to numbers). It involves calculating the technical score Award ASmax and the price score increment and then examining the differ- AS = Adjusted Score ence between the incremental advantages of each. The incre- T = Technical Score ment in the technical score is calculated by dividing the EE = Engineer's Estimate highest technical score by the next highest technical score P = Price Proposal minus one multiplied by 100%. The increment in price score is calculated by dividing the highest price score by the next Weighted Criteria highest price score minus one multiplied by 100%. The award is made to the offeror with the lowest price, unless the higher The weighted criteria algorithm also requires use of priced offers can be justified through a higher technical value. numerical scoring (or adjectival ratings converted to num- This justification is made by determining whether the added bers). The technical proposal and the price proposal are increment of price is offset by an added increment in techni- evaluated individually. A weight is assigned to the price and cal score. See the following generic algorithm and Table 2.14. each of the technical evaluation factors. The sum of these val- Case Study 13 in Appendix D also provides an example of the ues becomes the total score. The offeror with the highest total quantitative cost-technical tradeoff algorithm. score is selected. See the following generic algorithm and Algorithm: Order offers by increasing price proposals Table 2.13. Case Studies 4, 5, and 8 in Appendix D also pro- TIncrement = [(Tj/Ti) 1] x 100% vide examples of the weighted criteria algorithm. Table 2.12. Adjusted score example. Technical Score* Calculations (Engineer's Estimate Adjusted Offeror (1,000 maximum) Price Proposal = $10 million) Score* 930 x 106 1 930 $10,937,200 85 10,937,200 890 x 106 2 890 9,000,000 99 9,000,000 940 x 106 3 940 9,600,000 98 9,600,000 820 x 106 4 820 8,700,000 94 8,700,000 * Note: TechnicalScore --(Sumof Tec hnical Sco re f or al l evaluation fac tors); Adjuste Scd ore =(T echnical Score x 1, 00 0, 0 0 Pr 0 oposal )/Price($ )
OCR for page 39
27 Table 2.13. Weighted criteria example. Technical Score* Calculation of Price Score Calculation of Total Score Offeror (60 maximum) Price Score (40 maximum) Total Score (100 maximum) $1,000,000 x 40 1 51 33 51 + 33 = 84 $1,200,000 $1,000,000 x 40 2 53 32 53 + 32 = 85 $1,250,000 $1,000,000 x 40 3 44 36 44 + 36 = 80 $1,100,000 $1,000,000 x 40 4 39 40 39 + 40 = 79 $1,000,000 * Note: Sum of technical scores for all evaluation factors defined in the technical review evaluation plan. PIncrement = [(Pj/Pi) 1] x 100% Appendix D provide examples of the qualitative cost-technical If TIncrement < Increment, Award Proposali tradeoff algorithm. If TIncrement > PIncrement, Retain Proposalj for The tradeoff analysis is not conducted solely with the rat- possible award and repeat with Proposalj+i ings and scores. The selection official must analyze the differ- Repeat Process until TIncrement > PIncrement ences between the competing proposals and make a rational T = Technical Score decision based on the facts and circumstances of the specific P = Price Proposal acquisition. Although different selection officials may not necessarily come to the same conclusion, the same criteria In this example, because the difference between the low must be met in all cases. Specifically, the decision must and second low price proposals is 8%, the difference in the weighted scores of the two proposals should be greater than · Represent the selection official's rational and independent 8% to justify expending the additional increment of cost. In judgment, this case, the 33% difference in weighted scores and corre- · Be based on a comparative analysis of the proposal, and sponding 8% increase in price indicates that Proposal #2 is a · Be consistent with the solicitation evaluation factors and better value than Proposal #1. This is not the case when com- subfactors. paring Proposal #2 to Proposal #3--the 3% increase in cost is not justified by the 1% increase in technical score. Thus, the best value in this example is Proposal #2. Fixed Price--Best Proposal The fixed price--best proposal algorithm is based on the Qualitative Cost-Technical Tradeoff premise that the project owner will establish either a maxi- mum price or a fixed price for the project. Each Offeror must The qualitative cost-technical tradeoff is used by many fed- submit a technical proposal accompanied by an agreement to eral agencies under the FAR. This method relies primarily on perform the work within the specified pricing constraints. the judgment of the selection official to determine the relative The award is based only on the technical proposal evaluation. advantages offered by the proposals following a review of the The offeror that provides the best technical proposal will be evaluation ratings and prices (Army 2001). The final decision selected. See the following generic algorithm and Table 2.15. consists of an evaluation, comparative analysis, and tradeoff Case Study 6 in Appendix D also provides an example of the process that often require a subjective judgment on the part of fixed price--best proposal algorithm. the selecting official. Figure 2.3 depicts the qualitative cost- Algorithm: Award Tmax, Fixed P technical tradeoff algorithm as described in the Army Source T = Technical Rating Selection Guide (Army 2001). Case Studies 1, 2, and 3 in P = Project Price Table 2.14. Quantitative cost-technical tradeoff example. Proposal Price Weighted Score Price Increment Score Increment 1 $4 .0 M 300 -- -- 2 $4.3 M 400 + 8% + 33% 3 $4.4 M 405 + 3% + 1%
OCR for page 40
28 Lowest priced Award to proposal is the superior YES lowest priced proposal in terms of offeror non-cost proposal NO Proposals are essentially equal in YES terms of non-cost factors NO Award to Conduct offeror that tradeoff represents analysis the best value Figure 2.3. Decision model of determining the successful offeror using qualitative cost-technical tradeoff (Army 2001). Industry Applications of Best-Value Award Algorithms Comparison of Award Algorithms Table 2.15 illustrates the additional information gleaned Ultimately, no matter which algorithm is selected, the from the analysis of best-value RFPs collected during the first owner must have a result that allows it to differentiate a less phase of this study. The following case study summary is competent contractor with a low bid from a more highly based on the same 50 cases previously presented in Table 2.1 competent contractor whose proposal adds value to the proj- in the best-value parameter section of this chapter. As shown ect. The next step is to differentiate between those apparently in Table 2.16, it is very simple to classify the various agency competent and valuable proposals to determine which pro- best-value methodologies into the seven generic best-value posal is the optimum combination of price and non-price award algorithms proposed in this study. factors that delineate the true best value. All seven of the best-value award algorithms are represented Meets technical criterialow bid (cost) is defined as any in the case studies. The generic classification of the award algo- selection process where the eventual award will be made to rithms provides a baseline for comparison among agencies. Fig- the lowest priced, fully qualified and/or responsive bidder. ure 2.4 depicts the frequency of use for the award algorithms. This category includes the processes named "equivalent The qualitative cost-technical tradeoff and the weighted cri- design/low bid" and "meets criteria/low bid" and the FAR teria algorithms are the most frequently used and make up one- method named "fully responsivelowest price" as well as half of the sample population. The adjusted score, adjusted bid, other variations on this theme. and meets technical criterialow-bid algorithms are approxi- As a general rule, the low-bid approach was preferred on mately equal in number and constitute 44% of the sample. The projects where the scope was very tight and clearly defined, and quantitative cost-technical tradeoff and the fixed-pricebest innovation or alternatives were not being sought. This might proposal algorithms represent only 6% of the sample. include highway projects with a specified type of pavement, Table 2.15. Fixed price--best proposal example. Technical Score Offeror (100 maximum) 1 91 2 93 3 84 4 79
OCR for page 41
29 Table 2.16. Best-value award algorithm case study summary. Best-Value Award State/Agency Agency Terminology Remarks Algorithm Alaska DOT Criterion Score Divide Technical Score by Price Adjusted Score Arizona DOT Quality Adjusted Price Percentage system used to adjust bid Adjusted Bid Ranking price for technical score Colorado DOT Pre- Low Bid, Time Adjusted Multi-parameter bid with qualifications Meets Technical 1999 Criteria--Low Bid Colorado DOT Best Value May use weighted criteria to arrive at an Adjusted Score Post-1999 adjusted score Delaware DOT Competitive Proposals Design Alternates, Qualifications, Weighted Criteria Scheduled, and Price scored District of Best Value Adds owner contract administration Adjusted Score Columbia DPW costs to price Florida DOT Adjusted Score May also include time adjustment Adjusted Score Georgia DOT Low Bid, Prequalified Short list by qualifications Meets Technical Criteria--Low Bid Idaho DOT Weighted Selection Cost 51%; Qualifications/Past Weighted Criteria Experience 49% Indiana DOT Low Bid, Fully Qualified Minimum technical score to be found Meets Technical qualified Criteria--Low Bid Maine DOT Overall Value Rating Divide Price by Technical Score Adjusted Bid Mass Highway Best Value Included life-cycle cost criteria Weighted Criteria Michigan DOT Low Composite Score Divide Price by Technical Score Adjusted Bid Minnesota DOT Low Bid, Fully Qualified Short list by qualifications Meets Technical Criteria--Low Bid Missouri DOT Low Bid + Additional Additional costs include life-cycle cost Meets Technical Cost calculation Criteria--Low Bid New Jersey DOT Modified Low Bid Included design costs Meets Technical Criteria--Low Bid North Carolina Quality Adjusted Price Percentage system used to adjust bid Adjusted Bid DOT Ranking price for technical score Ohio DOT Low Bid Includes design costs Meets Technical Criteria--Low Bid Oregon DOT Best Value Combine technical with cost by weights Weighted Criteria South Carolina Low Composite Score Divide Price by Technical Score Adjusted Bid DOT South Dakota DOT Best Value Divide Price by Technical Score Adjusted Bid Utah DOT Best Value Combine technical with cost by weights Weighted Criteria Virginia DOT Two Step Selection Qualifications/Experience in Step 1 and Weighted Criteria Price and Technical in Step 2 Washington DOT High Best-Value Score Divide Technical Score by Price Adjusted Score Alberta, Canada, Value Index Divide Technical Score by Price Adjusted Score Ministry of Highways Alameda Lowest Ultimate Cost Add Price to Authority's Costs Meets Technical Criteria Transportation Associated with Proposal Low Cost Corridor Agency (continued) geometric design, and minimal ancillary works. It also is used Its thrust is to logically modify the price in a manner that reflects on building projects where the owner has completed most of the value of the underlying proposed qualitative factors. Its the design development and the contractor only needs to com- selection as an award algorithm indicates that price is an impor- plete the final construction documents. If the "cost" element is tant consideration but that some other aspects of the project added to the selection process, it can also be used for more must be included in the algorithm to determine best value. This complex projects where different proposals impact life-cycle is in effect a unit pricing of quality (Gransberg et al. 1999). costs, right-of-way expense, or other costs incurred by the proj- Adjusted score is the mathematical reciprocal of adjusted ect owner. bid. In this case, some function of the technical score is The adjusted bid algorithm is identified by the act of divid- divided by the proposed price to give an index in the units of ing the price by some factor related to the technical evaluation. technical points per dollar. It would follow that the adoption
OCR for page 42
30 Table 2.16. (Continued ) Best-Value Award State/Agency Agency Terminology Remarks Algorithm City of Reno, Best Value Qualifications & Past Performance Weighted Criteria Nevada equal to Price City of Santa RFP Process Requires Guaranteed Maximum Price Qualitative Cost- Monica, California and life-cycle criteria Technical Tradeoff City of Wheat RFP Process Uses Weighted Criteria approach to Fixed Price/Best Design Ridge, Colorado arrive at technical score District of Best Value Responsiveness check for qualifications, Meets Technical Columbia Schools experience & subcontracting plan Criteria--Low Bid Award to lowest, fully responsive bid Federal Bureau of Best Value Uses Weighted Criteria approach to Qualitative Cost- Prisons arrive at technical score Technical Tradeoff Federal Highway Best Value Adds owner contract administration Quantitative Cost- Administration costs to price. Uses Adjusted Score Technical Tradeoff formula to differentiate between bids Fort Lauderdale Selection/Negotiation Requires Guaranteed Maximum Price Weighted Criteria County, Florida General Services Best Value Uses Weighted Criteria approach to Qualitative Cost- Administration arrive at technical score Technical Tradeoff Los Alamos Best Value Two phase selection Weighted Criteria National Laboratory Maricopa County, Quality Adjusted Price Uses Weighted Criteria approach to Adjusted Bid Arizona Ranking arrive at technical score. Then computes a "$-value" of technical proposal and subtracts from price Naval Facilities Best Value Uses Weighted Criteria approach to Qualitative Cost- Engineering arrive at technical score Technical Tradeoff Command Nashville County, Competitive Sealed Qualifications, Management Plan and Adjusted Score Tennessee Proposals Price plus Warranty National Best Value Uses Weighted Criteria approach to Qualitative Cost- Aeronautics and arrive at technical score Technical Tradeoff Space Administration National Institute of Best Value Uses Weighted Criteria approach to Qualitative Cost- Standards and arrive at technical score Technical Tradeoff Technology National Park Best Value Uses "technically acceptable" approach Qualitative Cost- Service to arrive at technical score Technical Tradeoff Pentagon Best Value Uses Weighted Criteria approach to Qualitative Cost- Renovation arrive at technical score; includes Technical Tradeoff Program Office incentive clauses Seattle Water Best Value Uses Weighted Criteria approach to Cost-Technical Tradeoff Department arrive at technical score University of Best Value Qualifications/Experience in Step 1 and Weighted Criteria Colorado Price and Technical in Step 2 University of Best Value Qualifications/Experience in Step 1 and Weighted Criteria Nebraska Price and Technical in Step 2 (continued) of this approach would signal that the owner is less con- treatment plants where the owner wants to evaluate alterna- cerned about cost than quality. The adjusted score approach tive treatment processes. seems to work well when overall outcomes can be clearly The definition of weighted criteria is the broadest defini- defined and a number of alternatives exist which could pro- tion of all best-value algorithms. The weighted criteria algo- vide the desired outcomes. This could include public build- rithm is selected when innovation and new technology are to ings where the owner has some design constraints but is be encouraged or specific types of experience are required to open to innovative solutions within the constraints. It has obtain the desired outcome. This approach may also be used also been used in highway projects where alternative geo- when a fast track schedule is required or when constructabil- metric designs and material types are acceptable or water ity is inherent to the successful execution of the project. The
OCR for page 43
31 Table 2.16. (Continued ) Best-Value Award State/Agency Agency Terminology Remarks Algorithm U.S. Army Corps of Best Value Uses Weighted Criteria approach to Qualitative Cost- Engineers arrive at technical score Technical Tradeoff U.S. Customs Best Value Uses Weighted Criteria approach to Qualitative Cost- Service arrive at technical score. Requires Technical Tradeoff Guaranteed Maximum Price U.S. Department of Best Value Uses Weighted Criteria approach to Cost-Technical Tradeoff Energy arrive at technical score U.S. Forest Service Best Value Uses Adjusted Bid formula to Quantitative Cost- differentiate between bids Technical Tradeoff U.S. Postal Service Best Value Uses Weighted Criteria approach to Qualitative Cost- arrive at technical score Technical Tradeoff Utah Dept. of Value Based Selection Combine technical with cost by weights Weighted Criteria Natural Resources weighted criteria algorithm has the advantage of distinctly The quantitative cost-technical tradeoff best-value communicating the owner's perceived requirements for a algorithm uses the classic industrial engineering "Defender- successful proposal through the weights themselves. For Challenger Analysis" (Riggs and West 1986) to structure the instance, if a project owner is very concerned about the archi- comparison of price and all other non-price criteria. This tectural appearance of the project, a disproportionate weight algorithm starts by ranking the proposals from lowest to high- can be given to the evaluation criteria that directly define the est based on price. Then, it uses an incremental analysis of the ultimate aesthetic appeal. On the other hand, if an owner is percentage increase in price versus the percentage increase in concerned that the project's program might exceed the avail- technical score. If the technical incremental increase is greater able budget, price can be given a weight of greater than 50% than the price incremental increase, then the higher priced of the total. Thus, best-value bidders will be encouraged to proposal is preferred. This analysis is continued proposal by propose design alternates that will reduce the price or will proposal until the relative amount by which the score goes up only cause a minimal price increase. is less than the relative amount by which the price goes up. The Next, both qualitative and quantitative cost-technical best-value proposal is the highest rated proposal with an incre- tradeoff are algorithms that include the federally mandated mental analysis showing that the increase in price is justified variations of best-value award and those jurisdictions where by the increase in technical rating. technical and price must be evaluated separately (USACE 1994, Finally, fixed cost--best proposal is a relatively recent addi- NAVFAC 1996). The qualitative cost-technical best-value algo- tion to the best-value award discipline. In design-build proj- rithm could be the most subjective of all the award algorithms. ects, it is sometimes called "Design-to-Cost." This method In essence, the owner compares the value of the various features stipulates a fixed or maximum price and uses project scope, of the technical, schedule, and organization against the pro- qualifications, schedule, and other non-cost factors instead of posed price, and, using professional judgment, determines if bid price. This method has the advantage of immediately the aspects of a given proposal justify its price and whether the allowing the owner to determine if the required scope is real- additional positive attributes of a higher bid are worth more istically achievable within the limits of a tight budget. It also than the attributes contained in the low bidder's proposal. reduces the best-value decision to a fairly straightforward analysis of proposed design alternates and other non-cost fac- tors. Lastly, it truly is responsive to the efficient use of capital Meets Technical by committing virtually all available funding up front and Quantitative Cost- Criteria Low Bid Adjusted Bid Technical Tradeoff using the quantity and quality of project proposals to deter- 16% 14% 4% Adjusted Score mine the most attractive offer. Fixed Price - Best 14% Thus, given the previous discussion, it is now possible to Proposal classify each of the existing best-value award algorithms into 2% the proposed seven general categories. It is believed that by Qualitative Cost- doing so, much confusion about the details of the various Weighted Criteria Technical Tradeoff 24% selection methods can be eliminated. 26% Each algorithm brings strengths and weaknesses to the best- Figure 2.4. Frequency of use for the award algorithms. value contract award process. Meets technical criterialow bid
OCR for page 44
32 (cost) is by far the simplest and mechanically the closest to the Weighted criteria allows significant flexibility to the proj- existing design-bid-build/low-bid award process. As such, it is ect owner in determining the best-value proposal. It preserves probably the easiest to implement by an agency that has no the ability to tailor the evaluation plan to the specific needs of previous best-value experience. It is also the algorithm that each project and rate the qualifications of all bidders. It pro- will probably face the least opposition to its use for two rea- vides a method for including price as only one of several eval- sons. First, the concept of short-listing design firms on a qual- uation areas and permits the agency to adjust the weights of ifications basis is well accepted. Therefore, extending that each rated category as required to meet the needs of the par- concept to determining a short list of the best qualified con- ticular project. Its greatest drawback is the complexity of the struction contractors should be fairly direct. Secondly, award- evaluation planning. To properly implement the weighted ing to the lowest priced proposal from the list of prequalified criteria algorithm, a great deal of up-front investment in time firms is not very different from the typical public agency low- and human resources must be made during the development bid paradigm, and factoring owner costs into the equation of the RFP and its evaluation plan. involves a minimal change to that paradigm. This approach The cost-technical tradeoff algorithms preserve the would fit into the "competitive sealed bidding" category of the owner's option to award based on a qualitative (possibly, sub- ABA Model Procurement Code even though it allows the jective) comparison of the value of higher priced proposals or owner to consider certain elements in addition to the bid to make the cost-technical tradeoff decision quantitatively as price. It is likely to be acceptable in those states that still require shown in the U.S. Forest Service Highway Case Study. It also both qualifications-based selection of designers and low-bid furnishes the most robust method with which to make the award for construction (Wright 1997). The greatest weakness best-value decision. The use of a cost-technical tradeoff of meets technical criteria--low bid is its focus on price alone, forces the owner to relate the price and the value of the other which eliminates one of best-value procurement's greatest evaluated factors in a way that highlights the best features of benefits: the ability to compare different construction solu- best-value award algorithms. Additionally, the cost-technical tions to the same problem. The addition of cost elements helps tradeoff is mandated for federal projects (FAR 2004). It is to solve this problem, allowing a process to be used that is very probably best used when the owner anticipates a very com- close to low bid, where the differences between the proposals petitive set of proposals submitted by a sophisticated, can be converted into future out-of-pocket expense to the well-qualified group of competitors. It furnishes an avenue to project owner. step back after the evaluation and contemplate the relative Adjusted bid and adjusted score, on the other hand, allow desirability of the various combinations of qualifications, competition between varying design alternates, construction design approach, and price. Finally, it should be noted that management approaches, and contractor qualifications if this discussion assumes the technical evaluation is conducted appropriate for the project. This encourages innovative using a methodology such as the weighted criteria algorithm. approaches by industry while preserving the ability to rate The fixed-price best proposal award algorithm is similar to the qualifications of the contractors. The major drawback to all of the algorithms that assign a technical score to the best- these methods and other technical score-related algorithms value offer, but the price is fixed for all offerors. This award is the reliance on the evaluator's ability to develop an accu- algorithm should only be considered when the bidding of rate technical score for a proposal. Evaluators often have dif- design alternates is being entertained. This is an excellent sys- ficulties translating evaluation criteria into points and end tem when owners have a fixed budget and would like to get up trying to equate a dollars-per-point for each evaluation more construction for their money. However, the engineer's decision. Additionally, by mathematically combining the estimate for the scope of the project must be sound. An esti- technical and the proposed contract price, there is a poten- mate that is too high may result in offerors adding unneeded tial to create an environment where construction contractors scope to win the project. Even worse, an estimate that is too may be tempted to play games with the numbers to increase low could result in offerors proposing scopes that do not meet their adjusted bids or scores. The adjusted bid system appears technical standards. to be useful on projects where funding is constrained but The selection of an algorithm requires determining which where some qualitative feature of the project, such as a fast approach is the most appropriate for the project in question. track schedule or external factors such as traffic disruption The goal of best-value contract award is to devise a system or innovative environmental protection, is also very impor- that maximizes the probability of selecting a contractor who tant to the owner. Adjusted bid seems to be most appropri- will successfully complete the project. In many cases, the tried ate for projects where innovation is encouraged but where a and true low-bid price only method will in many cases be the high degree of price competition is desired. Adjusted score is most appropriate method. Therefore, a careful analysis of the more appropriate where the technical content is more project must be made before deciding on a project award important than the price. algorithm and its associated criteria.