Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program (2010)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

54 It’s unwise to pay too much, but it’s also unwise to pay too lit- tle. When you pay too much you lose a little money, that is all. When you pay too little, you sometimes lose everything, be- cause the thing you bought was incapable of doing the thing it was bought to do. The common law of business balance pro- hibits paying a little and getting a lot——it can’t be done. If you deal with the lowest bidder, it is well to add something for the risk you run. And if you do that, you will have enough to pay for something better. —John Ruskin (1819–1900) English author, artist, scientist, poet, philosopher The cost of conducting an availability or disparity study is a significant concern for state DOTs. Limited resources and competing priorities mean that costs must be justified and managed to deliver the best value. A study’s true value, how- ever, comes in its ability to withstand strict scrutiny in court. While studies were expensive, as a percentage of total state DOT contract awards over 5 or 6 years, the cost of a disparity or availability study is still very small.178 In many ways, a dis- parity study is like an insurance policy for the DBE Program: the cost is a small fraction of the value of what is being insured. We interviewed several state DOTs to obtain information about the price of their studies, the internal resources neces- sary to manage the process, and advice for other agencies considering studies. The following issues emerged that state DOTs may want to consider when they evaluate the costs of procuring studies and conducting studies. As with the other challenges of conducting studies, the costs must be weighed against the litigation risk that accompanies a less comprehen- sive study, or no study at all, and the benefits of a constitu- tionally sound DBE Program. Sources of Funds Most state DOTs used state funds to pay for their studies, regardless of type. At least one state DOT expressed concern that the FHWA would want to direct the study process if the state DOT used federal funds. One state DOT we interviewed had successfully used FHWA funds, Airport Improvement Program grant funds (the airport was included in the study), and contributions from the other local partners. Management of the Study Process Most state DOTs stated that it was critical to include senior management in the study process as early as possible, includ- ing ensuring that they understood the study’s legal and statis- tical challenges. This helped to ensure that the process was adequately funded and realistic timelines were developed. A few agencies have hired outside consultants to assist with drafting the RFP or managing the study process. One state DOT paid $20,000 to a supportive services consultant to assist with the drafting process and to respond to the DBE commu- nity’s desire to participate in the study. That consultant sur- veyed stakeholders about their preferences for what to include in the study and incorporated those concerns into the RFP. A multi-agency statewide study that included the state DOT hired an outside study manager for approximately $100,000 over 18 months. All agencies strongly stressed that a (close to) full-time study manager was necessary to ensure smooth and timely completion. If an outside study manager is used, it is important that he/she be vested with the bureaucratic author- ity to obtain responsiveness from other state DOT personnel and prime contractors. An outside manager who is unfamiliar with key actors may encounter significant difficulties securing their assistance with the study. One DBE director suggested that the state DOT also con- sider hiring a consultant to assist it to communicate with the department’s outside constituencies about the study. While C H A P T E R 4 Study Resource Issues 178 The average Federal-aid highway fund apportionment to state DOTs in FFY 2007 was about $600 million—or about $3 billion per state DOT over a 5-year period. See Table 1. If a disparity study for the average state DOT will cost $1.5 million, which was actually the price for only the largest states, this would be 0.05% of the federal funds disbursed during the study’s projected coverage.

55 the study consultant can provide information to outside par- ties, it is important that the perception of the firm’s objectiv- ity not be compromised by “reporting” to groups or persons with vested interests in the study’s outcome. This state DOT reported that an excellent communications strategy would have helped to increase inclusion in the study’s anecdotal data-gathering efforts and stakeholders’ familiarity with the legal and administrative challenges of the process. No cost es- timate was available, as this approach was not in fact pursued by the department. However, one state DOT reported that it used internal personnel to conduct extensive community outreach and Web site maintenance. If no outside assistance is procured, the state DOT must “budget” internally for a study manager and communica- tions strategy. It would be prudent to allocate the time, if not the salary, of a full-time experienced employee to oversee the study process. In-House Studies versus Outside Consultants Despite the potential for reducing study costs, conducting disparity or availability studies using primarily state personnel is not recommended because of the possible appearance of bias in favor of the agency’s decisions. No state DOT conducted a study using its own personnel. While there were some public entities that produced studies in-house in the early days after Croson, governments quickly learned that outside consultants were needed to provide the type of independent statistical and economic expertise that courts would require. While it would obviously be less costly in the short run to conduct the study using state staff already on payroll, the risk of such a report being rejected by the courts as biased is equally obvious. State personnel have, however, been used to calculate DBE goals using the nonstudy methods listed in 49 C.F.R § 26.45 (c).179 Availability Studies versus Disparity Studies One critical factor in determining the cost of studies is which analytical elements are included. Availability studies, because they typically lack the compelling interest compo- nents, cost less. Forgoing disparity testing on the state DOT’s own contracts and subcontracts, eliminating the gathering of statistical evidence of economy-wide discrimination, not gathering anecdotal or other qualitative evidence of discrim- ination, and not including a review of the state DOT’s race- and gender-neutral policies will significantly reduce study costs. One state DOT outside the Ninth Circuit has success- fully relied solely upon an availability study.180 For these reasons, we found that availability studies generally cost under $400,000, and potentially less if complete subcon- tract data are available. According to our interviews, costs for disparity studies ranged from a low of approximately $405,000 for a smaller DOT to $1,500,000 for a large state DOT. Collection of Subcontracting Data A second significant cost element of studies is the collec- tion of data on first-tier subcontracts. This cost element plays a role in both disparity and availability studies since it is a pre- requisite for developing a defensible estimate of DBE avail- ability.181 One of the first tasks for a state DOT considering a study is to evaluate the state of its existing subcontract records. Often, records on non-DBE subcontracts are less complete than those on DBE subcontracts, and even absent in their entirety. A good disparity or availability study will reconstruct the miss- ing data for at least a sample of relevant contracts, and this re- construction is one of the most significant single elements in a study. A study that includes good subcontract records is less vulnerable if challenged than one that does not.182 If such subcontract data are complete, then a study cost re- duction of 15%–20% may be achieved.183 If such data are miss- ing or incomplete, before commissioning a study the state DOT itself can reconstruct the missing data.184 If done cor- rectly, similar cost savings can then be achieved when the state DOT commissions a study. However, all DBE directors we in- terviewed specifically mentioned problems with data collection related to subcontracts. One state DOT attempted to save money by gathering the missing data itself, but reported that the effort did not work out well and strongly urged others to have the consultant conduct all data gathering. 179 E.g., dividing the number of firms in a state DOT’s DBE Directory by the County Business Patterns data or the use of bidder’s list information. 180 Sherbrooke Turf, Inc. v. Minnesota Department of Transportation, 345 F.3d. 964 (8th Cir. 2003), cert. denied, 541 U.S. 1041 (2004). 181 Information on how the state DOT’s contract and subcontract dollars are dis- tributed across different industries and geographic locations is used to provide weighted availability statistics, which, in turn, are important to support asser- tions that an agency’s statistics are narrowly tailored. See supra at Chapter Three, Determination of Relevant Product Market. 182 E.g., Contractors Association of Eastern Pennsylvania, Inc. v. City of Philadel- phia, 91 F.3d 586, 601 (3rd Cir. 1996) (the subcontracting analysis was insuffi- cient because the review of records by a city employee was “cursory” and the study did not include subcontracting. There was not a “firm basis for inferring discrimination by contractors in the subcontracting market during the period,” and therefore insufficient evidence for a subcontracting goal). 183 Detailed cost data for different consultants could not be obtained. This esti- mate is based on our own experience in conducting studies. 184The particular issues involved in reconstructing missing or incomplete sub- contract records are discussed in detail in Appendix A.

In addition to the costs associated with reconstructing past subcontract records, state DOTs are well advised to take steps to ensure the collection of current and future data.185 Once procedures have been adopted for subcontract data collec- tion, appropriate software for maintaining and tracking this data can be developed or purchased. There are several con- tract compliance software products on the market that make ongoing data gathering easier and facilitate production of re- quired annual U.S.DOT submissions and other reports. Since disparity and availability studies must be repeated periodi- cally (every 5 to 6 years is recommended),186 investment in subcontract data collection procedures and software will gen- erate cost savings on future studies. Finally, a state DOT should provide complete and accurate information to potential proposers so that proposed cost es- timates will accurately reflect the costs of reconstructing missing data. To the extent that the department does not pro- vide full information, proposers are likely to assume the worst and price accordingly. Subrecipient Data State DOTs routinely pass-through a portion of their federal-assistance dollars to other state and local agencies. These dollars support activities such as construction, design, planning, and transit and airport operations. State DOTs should consult with their regional modal ad- ministrations to confirm which of their subrecipients, if any, are responsible for reporting DBE activity; this will ensure that those subrecipients will be included in the scope of any availability or disparity study. Further, the subrecipients’ contracting and subcontracting data should conform to the same requirements presented above for the state DOT’s di- rect contracting and subcontracting data. Any additional ef- fort required to acquire the subrecipients’ data and bring it up to the same quality level as the state DOT’s data should be factored into the estimated cost and timeframe of any study. Examples of Costs for Other Analytical Elements Verification of Business Owner Race and Gender As previously illustrated, the race and gender of business owners obtained from various directories and lists is often in error.187 It is therefore important to verify race and gender as- signments by contacting a sample of business owners directly. Such “misclassification” surveys, typically done by phone, are costly due in part to the sample sizes required for reliable re- sults.188 Studies that include such verification procedures will cost more than studies that do not. Studies that do not, how- ever, may prove more vulnerable to the charge that they are not sufficiently narrowly tailored. Nonrespondent Surveys Similarly, most studies conduct one or more mail surveys as part of their analyses. Such mail surveys typically achieve response rates of 5%–15%, due largely to the fact that partic- ipation is voluntary. To have confidence in any inferences drawn from such a survey, a competent social scientist will test whether the respondents were representative of the entire survey sample, that is, some comparison of respondents and nonrespondents is required. Such nonresponse testing is also typically conducted over the telephone and, like the misclas- sification surveys, is relatively costly. However, also like mis- classification surveys, studies that ignore nonresponse bias may find their mail survey results disregarded, if challenged. Multi-Jurisdiction Studies One potential source of costs savings is to conduct a study jointly with other government agencies. This approach is most cost-effective when there is significant overlap in the geo- graphic and product markets of the participating agencies. However, because state DOT contracting can differ sub- stantially from that of governments like cities and counties, geographic and product markets do not overlap as much as might be desired. In these cases, the savings will be less than would be the case with more compatible study partners. There are, however, significant potential cost savings in partnering with other compatible governments, particularly for disparity studies. Costs for building the availability data- base, conducting the economy-wide statistical analyses, and gathering the anecdotal evidence, can all be readily prorated among the participants. State DOTs must, however, weigh the inevitable delays and complications that will result from more participants in the process against the potential cost savings. One way to offset these is to reach an understanding be- fore issuing the RFP regarding the responsibilities of each participating agency to provide its available contracting and 56 185 See Appendix A, Methods for Collecting Subcontract Data in anticipation of Future Studies. 186 Five to six years is our recommendation as a rule of thumb based on the study period length documented in Table 5. A shorter period than this may lead to unnecessary expenditure of public funds on disparity or availability studies, while a period much longer than this runs the risk of a court, in a constitutional challenge to a DBE Program, finding that the data placed into evidence are “stale.” 187 See supra, Tables 7–10. 188 Samples must typically be stratified by at least race, gender, and industry group.

57 subcontracting data to the consultant in a timely manner. Delay on the part of one agency can easily slow the study for all. Large differences between agencies in the state of their subcontract records can have a similar impact. Potential part- ners are advised to investigate these differences before agreeing to collaborate on a study. Another tip when conducting multi-agency studies is to appoint a single person to manage the study on behalf of all participating agencies, as opposed to having a separate study manager at each agency. A single project manager can more easily streamline communications with the consultant and with personnel at each agency. Only one state DOT has directly collaborated with other government entities to conduct its study, which included the state’s largest city and the state’s railroad authority. Other de- partments reported that it was not feasible for political or ad- ministrative reasons to collaborate with other governments. For example, one Ninth Circuit state DOT with an existing availability study conducted extensive outreach to the state’s largest city, county, and airport and transit agencies to help pay for additional elements of disparity analyses, but was unable to reach an agreement.189 Multi-Agency State Studies Some state DOTs have participated in statewide studies that included all or most other state agencies. This has the ad- vantage of sharing costs among numerous state departments. However, it is critical in such statewide studies to ensure that the special requirements of Part 26 are met, such as the need to determine the agency’s spending with DBEs through race- neutral measures and the need to consider federally assisted spending separately. A separate report for the state DOT is also recommended. As with partnering for multi-jurisdictional studies, the state DOT must consider the likely increase in the time to conduct a multi-agency study and the diffusion of focus against the savings to the department. Several statewide studies that we reviewed for possible inclusion in this report were rejected because the statistical analyses of the state DOT did not include or did not identify its federally assisted contracts. Model Study Scope of Work We recommend that RFPs require the proposers to fully explain their approach to all components described in this re- port. Proposals should provide a complete, detailed method- ology that reflects a thorough understanding of the case law and presents sound economic analyses that will meet strict scrutiny and the regulatory requirements. For availability studies, the following major elements should be included: • An empirical assessment of the appropriate geographic market relevant to an agency’s contracting activity; • An empirical assessment of the appropriate product mar- kets relevant to an agency’s contracting activity; • An estimate of the fraction of businesses within the agency’s geographic and product markets that are owned by DBEs (i.e., “availability”); • To the extent necessary to implement Step 2, econometric analyses of DBEs’ success relative to non-DBEs’ (e.g., in business formation rates and in business owner earnings), and holding nondiscriminatory factors constant, in the market area surrounding the agency in question (i.e., “pri- vate sector disparity ratios”); • To the extent necessary to implement Step 2, econometric analysis of DBEs’ access to capital and credit relative to non-DBEs’, holding balance sheet and creditworthiness in- formation constant; and • Qualitative and/or quantitative analysis of the effectiveness of race-neutral measures to address low DBE participation in public contracting. For disparity studies, the following major elements should be included: • A legal review discussing Croson, Adarand, and subsequent case law and their impact; • An empirical assessment of the appropriate geographic market relevant to an agency’s contracting activity; • An empirical assessment of the appropriate product mar- kets relevant to an agency’s contracting activity; • An estimate of the fraction of businesses within the agency’s geographic and product markets that are owned by DBEs (i.e., “availability”); • An estimate of the percentage of all prime contract and subcontract dollars earned by DBEs (i.e., “public sector utilization”); • A statistical comparison of public sector utilization to availability (i.e., “public sector disparity ratios”); • Econometric analyses of DBEs’ success, relative to non- DBEs’ (e.g., in business formation rates and in business owner earnings), and holding nondiscriminatory factors constant, in the market area surrounding the agency in question (i.e., “private sector disparity ratios”); • Econometric analysis of DBEs’ access to capital and credit relative to non-DBEs’, holding balance sheet and credit- worthiness information constant; 189 Neither the airport nor the transit agency has conducted studies to date. They continue to implement all race-neutral DBE Programs.

• Qualitative evidence from DBEs and non-DBEs concerning experiences doing business or attempting to do business in the relevant marketplace, including experiences of institu- tionalized discrimination and/or individual disparate treat- ment, gathered through surveys, personal interviews, and/ or public hearings (i.e., “anecdotal evidence”); • Qualitative and/or quantitative analysis of the effectiveness of race-neutral measures to address low DBE participation in public contracting; and • Review of existing policies and procedures related to DBE participation, with recommendations for changes/revisions designed to improve the effectiveness of the program and increase legal compliance. Additionally, both types of studies should be supervised by professional social scientists (e.g., economists and/or statisti- cians) who can be qualified as expert trial witnesses under the exacting standards of the Federal Rules of Evidence to testify about the study’s data, methods, and findings. Tips for a Successful DBE Disparity or Availability Study RFP Process Drafting the RFP Document • Immediately and directly involve state DOT legal counsel in all phases of the RFP process, from drafting the RFP to selecting the consultant. We recommend that state DOTs consider placing the procurement and conduct of the study under the supervision of a senior attorney in the legal department. Not only will this ensure that critical legal support is provided throughout the process but also the attorney-client privilege may attach to many communica- tions. A senior attorney is recommended because of the complex and complicated nature of the legal and adminis- trative issues involved. • Develop a study team, whose members should be com- prised of the following agency functions: – Law, – Purchasing and/or contract administration, – Public works/engineering or equivalent, – Finance/accounts payable or equivalent, – Information technology, – M/W/DBE program administration, and – Communications and/or external affairs. This team may be responsible for evaluating proposals, as well as overseeing the study process. Additional state DOT personnel may be included. We caution against including persons external to the agency, such as representatives of con- tractor groups or community organizations, because of the highly specialized nature of the study and the perception of possible bias or conflicts of interest. • Keep the RFP simple. Ask for the proposers’ best profes- sional judgment, supported by the case law, of what con- stitutes a defensible study. • Do not commission a study unless sufficient funds are available to fund it properly. A poorly funded study will be of little or no value in supporting a state DOT DBE Pro- gram and could, in fact, do actual harm.190 • Do not separate the conduct of the study into “phases.” Many study approaches are integrated. An “availability” phase is not readily separable from a “disparity” phase, as both share many common elements (building project database, relevant market determination, etc.). Moreover, important types of statistical testing are done through re- gression analysis where disparities and their determining factors are assessed simultaneously. Asking for a “dispar- ity” phase and an “if there is a disparity, determine the causes of the disparity” phase therefore requires a separa- tion that is not possible with regression analysis. DBE di- rectors who used this approach strongly recommended against it because it just increased the time to engage the consultant and complete the study with no benefit to the study process or product. • Allow at least 12 months for the study to be completed. Allow more time in the case of multi-agency studies. The most time-consuming component is usually the collection of adequate contract and subcontract data. Unless there is active litigation, quality is more important than speed. • Do not mandate regular face-to-face meetings. They are costly and time consuming, and with the technologies avail- able, usually unnecessary. Requiring such meetings will in- crease the cost and the time required to perform a study. • Put all required attachments in one place in the RFP and include a complete list of all attachments required in the proposal. • Make the current DBE Program (and any programs for non-federal-aid contracts) available electronically. • Provide a due date for questions, compile all questions and answers, and email them to all who submitted questions. To ensure that answers are provided in sufficient time to address proposers’ concerns, allow at least one week be- tween the provision of answers and the proposal due date. • Include information in the RFP about the number of and dollar value of relevant agency contracts. Provide informa- tion regarding the format and availability of the contract data. Include as much information as possible regarding the number, dollar amounts, and types of contracts, as well as the availability (hard copy, electronic, etc.) of the con- tract data for prime contractors and subcontractors. 58 190 See W. H. Scott Construction Co., Inc. v. City of Jackson, Mississippi, 199 F.3d 206 (5th Cir. 1999) (City was not free to ignore the results of its disparity study).

59 • If a DBE goal is set for the study, ensure that it reflects a re- alistic assessment of the amount of work likely to be done by a local DBE that lacks direct study experience. Provide a list of relevant potential DBE subcontractors if the UCP Directory is not easily searchable. • We advise against limiting pages in the proposal. The issues about study methods are complex and do not lend themselves well to overly truncated presentations. The state DOT will be served better by receiving more in- formation rather than less about the approach and the experts. • Ask for and check consultant references. Contact agencies for which studies were conducted that are not listed as references. • Request at least one sample study and have the state DOT legal counsel review it for legal sufficiency. • Do not separate the process of procuring the study into phases. Do not first require a Statement of Qualifications and then later require a Proposal. In every instance we re- viewed that used this approach, the information requested of the bidders was, in general, the same in both documents. Requiring what amounts to double proposals is time con- suming and costly. • Proofread the RFP. Many questions and problems will be eliminated with careful proofreading from the perspec- tive of a proposer, including incorrect information, such as for the time frame for study and period for study perfor- mance; unnecessary requirements, such as attachments and information that are obviously for construction or engi- neering projects; and serious redundancy, such as requir- ing the same information be provided in more than one place in the proposal. This will also reduce the state DOT’s burden of responding to questions and evaluating proposals. Conducting the RFP Process • Directly notify the relatively small pool of consultants that performs disparity studies of the issuance of the RFP. It is not reasonable to expect consultants to register with every agency or search hundreds of public websites every day. Failure to disseminate the RFP will reduce the competition and breadth of choices for the state DOT. • Establish the due date for proposals on a Wednesday, Thursday, or Friday. Proposals will be submitted from consultants all over the country. This necessitates the use of overnight delivery services. Because unforeseen cir- cumstances may prevent packages from being delivered overnight, prudence counsels that the proposal be shipped two business days before the due date. If the due date is a Monday or Tuesday, this can essentially eliminate three days from the time the consultant has to work on prepar- ing its best offer. • Proposals should be due after 2:00 p.m. local time. This will lead to fewer problems with overnight delivery services. • Provide at least four weeks from the date of issuance for the response. • Include face-to-face interviews as part of the evaluation process for finalists. Not only is the state DOT hiring a study consultant but also potentially an expert witness for litigation. Face-to-face interviews permit the state DOT and its lawyers to evaluate the consultant’s ability to ex- plain the study data, methods, and findings and the team member’s experience and demeanor for litigation. • If you plan to conduct interviews, propose an anticipated timeline in the RFP and do your best to stick to it. This will help avoid scheduling difficulties later on. • If changes or amendments to RFP must be made within one week before the due date for proposals, then change the due date of the RFP. Give proposers sufficient time to revise their submissions. • Do not impose a mandatory pre-proposal meeting atten- dance requirement. Written questions and answers are the most effective way to address proposers’ concerns. • If state procurement and records laws permit, make evalu- ation documents available to bidders after the final selec- tion without filing Freedom of Information Act or similar requests. Doing this in a timely manner benefits the DOT because material errors have been discovered regarding the evaluation and award process once the evaluation docu- ments were made available to a bidder. Avoiding this type of situation is in the state DOT’s best interest.