Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program (2010)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

60 Introduction Twenty years after Croson, many public agencies with race- conscious subcontracting programs (including state DOTs) still do not systematically collect and maintain adequate sub- contracting data. Although state DOTs maintain records on their DBE subcontracting, many do not maintain compara- ble records regarding non-DBE subcontracting. Although non-DBE subcontract data are not necessary to produce the standard types of DBE utilization reports requested by U.S.DOT or the DOT’s executive leadership, which include, for example, only that share of construction dollars awarded during a given time to DBEs, such data are critical to produc- ing a high-quality and legally defensible availability or dispar- ity study. Non-DBE Subcontract Data Is Just as Important as DBE Subcontract Data In contrast to such limited reporting, a high-quality dispar- ity or availability study, among other things, should document the share of all construction dollars awarded by detailed indus- try during a given period. In addition, a disparity study should detail the share of dollars in each detailed industry awarded to DBEs. Several courts have warned defendants about the use of overly aggregated statistical comparisons in disparity studies because such aggregation can provide misleading results.191 This phenomenon is known as Simpson’s Paradox. Since many industries are only represented in state DOT projects through subcontracting, knowledge of non-DBE as well as DBE subcontract awards is imperative when building a proj- ect database for use in a disparity or availability study. An example may illustrate the point. Consider the con- struction contract shown in Table A1, below. The overall value of the contract is $12 million. The total value of the DBE subcontracts is $1 million and the total DBE share is therefore 8.3%. For a disparity study, however, the consultant would also want to know the dollar share accounted for by each NAICS code and the DBE share for each code. The former is needed in order to calculate properly weighted availability estimates. The latter is needed in order to calculate utilization and disparity statistics. These NAICS shares are shown in the last six rows of Table A1. This exercise should be repeated for each contract in the project database. Some studies have ignored the problem of missing subcon- tract data. For example, one study states bluntly: The utilization analysis of [M/WBEs] . . . does not include their utilization as sub-contractors or sub-consultants. . . . [We were] unable to secure sub-contracting/sub-consulting dollars from the actual contract files during the data collection effort, simply because the contract files do not contain the data relative to sub- contracting or subconsulting activities. . . . This . . . is striking in that . . . sub-contracting opportunities are often where much of the utilization of M/WBEs is found. Another study states: The data made available to [us] . . . contain primarily reports of subcontracts involving DBE participation, as the overall percent- age of subcontracting (less than 8 percent) is significantly lower A P P E N D I X A Importance of Comprehensive Subcontract Data Collection 191 See Engineering Contractors Association v. Metropolitan Dade County, 943 F. Supp. 1546, 1560 (S.D. Fla. 1996) (“The MRD study presents . . . data aggregated for all cap- ital construction contracts (i.e., SIC 15, 16 and 17 together) and disaggregated by SIC category. . . . The Court will focus primarily on statistical analyses of the disaggre- gated data because these data are more likely to reflect the realities of competition in the construction industry. Firms that build hospitals (SIC 15) do not compete for County contracts with firms that lay asphalt (SIC 16) or firms that install plumbing (SIC 17), therefore comparisons between these disparate entities would not produce a reliable portrait of County contracting trends.”). Simpson’s Paradox has also been used to criticize combining highway districts together into statewide statistics. See Phillips & Jordan, Inc. v. Watts, 13 F. Supp. 2d 1308, 1315 (N. D. Fla. 1998); Thompson Building and Wrecking Co. v. Augusta, 2007 U.S. Dist. LEXIS 27127, at * 8 (S.D. Geo. March 14, 2007) (criticizing combining MBE statistics with WBE statistics).

61 Table A1. Subcontract data collection example. Contractor Type Bid Package Dollar Amount NAICS DBE Prime $9,500,000 23621 Subcontractor $750,000 23621 Yes Subcontractor $300,000 23621 Subcontractor $50,000 23731 Subcontractor $50,000 23731 Subcontractor $200,000 23731 Subcontractor $100,000 23731 Yes Subcontractor $30,000 23811 Subcontractor $700,000 23821 Subcontractor $140,000 23821 Yes Subcontractor $10,000 23899 Yes Subcontractor $60,000 23899 Subcontractor $160,000 33599 TOTAL $12,000,000 NAICS Dollar Share DBE Share 87.6% 7.1% 3.3% 25.0% 0.2% 0.0% 7.0% 16.7% 0.6% 14.3% 23621 23731 23811 23821 23899 33599 1.3% 0.0% than subcontracting for most state Departments of Transporta- tion (between 40 and 45 percent). And, another states: [D]ata were missing [pertaining] to subcontract awards to Non- DBEs. This study disaggregated all contracting activity by prime and subcontracting categories. As such, it would have been helpful to have information on attainments by Non-DBE subcontractors. In the first study, no subcontract data were available from the client. In the second and third studies, the only data avail- able from the client were for prime contracts that happened to have DBE subcontractors. While it is apparent that all three studies recognized the importance of capturing complete subcontracting activity data, the problem remained unad- dressed in the final reports. Any statistical conclusions result- ing from such methods may be rejected by the courts since

the DBE program operates through the use of race-conscious contract goals for subcontracting. A subcontracting program requires subcontracting proof. In our view, disparity or avail- ability studies that ignore missing subcontract data are unlikely to withstand strict scrutiny. Therefore, agencies must educate themselves before com- missioning a study regarding their own data limitations and ensure that the data collection method to be implemented satisfactorily addressed the issue of missing subcontract data and that a realistic allocation of resources is included in the study budget to address the issue. Subcontract Data Allow Detailed Industry Statistics As illustrated by Table A1, without complete subcontract data it is not possible to carry out statistical analyses by detailed industry categories. For state DOTs, one of the most important uses of detailed industry statistics is to provide dollar-weighted estimates of DBE availability.192 As shown above in the sample contract in Table A1, far more dollars were expected to be spent in NAICS 23621 than in NAICS 23821 or 23811. Consequently, to set an overall DBE goal for this particular contract, the state DOT would weight DBE availability in NAICS 23621 more heavily than availability in NAICS 28321 or 23811. Exactly how much weight to give each detailed industry availability estimate would be determined by the proportional distribution of dol- lars across each relevant NAICS industry. Another, simpler example appears in the U.S.DOT’s DBE Program guidance: “if 90% of your contract dollars will be spent on heavy construction and 10% on trucking, you should weight your calculation of the relative availability of firms by the same percentages.”193 Despite some court rulings warning against the dangers of Simpson’s Paradox with respect to over-aggregation by indus- try, a substantial share of studies still fail to employ detailed industry weights. Of the 28 state DOTs that have performed or are currently performing a disparity or availability study, only three of the eight consultants involved used detailed industry data—i.e., three-digit NAICS/two-digit SIC or higher—to construct weighted availability estimates.194 Weighted DBE availability estimates are important because they allow state DOTs to meet the requirement that their annual DBE goals be narrowly tailored to their specific con- tracting circumstances. Weighted estimates are further impor- tant because narrow tailoring also requires DBE goals be set on a contract-by-contract basis.195 For example, a new road construction or a highway maintenance contract may have very different opportunities for DBE participation than an underground tunnel project. To set those goals, the agency needs to have reliable estimates of DBE availability for all detailed industry categories that may be involved in a project. State DOTs should require such detailed data be included and ensure that sufficient resources are allocated to the study to create it. Methods for Collecting Subcontract Data in Anticipation of Future Studies Compensating for missing subcontract data is one of the main factors affecting the cost of high-quality disparity and availability studies. One of the most important ways to reduce costs of future studies is to systematically collect and main- tain complete subcontract data. The most effective and least burdensome method is to require prime contractors and consultants to submit, either as part of their bid packages or at some other point prior to the agency issuing a Notice to Proceed, a standardized form listing all proposed first-tier subcontracts, both DBEs and non-DBEs. This form should clearly identify the prime con- tract and prime contractor and should include, at a mini- mum, the following information: • Full business name of the subcontractor; • Subcontractor street address, city, state, and zip code; • Subcontractor telephone number; • Subcontractor contact person and title; • Anticipated dollar amount of the subcontract; • Short description of the services to be performed and/or goods to be supplied; and • Race and gender of subcontractor ownership, if known. Additionally, the prime contractor should be periodically required during contract performance to certify that no mat- erial changes have been made to the proposed roster of sub- contractors or subcontract amounts.196 If change orders have been issued to any subcontractors, if new subcontractors have been added, or if original subcontractors have been dropped, then this should be noted and the pertinent details provided. At the final pay application, the prime contractor 62 192Detailed industry statistics are also important in calculations of utilization and disparity in disparity studies. 193U.S.DOT, OSDBU (n.d., Section II.F). 194Two of these three consultants, however, could not be evaluated because their studies are not yet complete and they have no other completed studies on record. 19549 C.F.R § 26.51(d) and (e); see also Chapter Two, Review of Existing Studies, supra. 196This could be done at each pay application or each quarter.

should be required to certify the final amounts actually paid to each subcontractor.197 Beyond this, the state DOT may wish to consider working with its IT personnel or engage an outside consultant to develop methods to maintain this data electronically for gen- eral program management use. The collected information can be integrated into existing data collection systems, or alternately, there are several specialized software products on the market designed to facilitate this process for DOTs and other public agencies. Methods for Addressing Missing Subcontract Data for Current Studies Obtain Missing Data Directly from Prime Contractors and Consultants If the state DOT has not collected and maintained this data, it will be necessary to either request the information directly from each relevant prime contractor or consultant or reconstruct the required information by other means. If all of the necessary subcontract data have not been maintained, it will probably not be cost-effective to recreate it all. It may be more feasible to recreate the missing subcontract information for a statistically representative sample of prime contracts. It is critical to ensure that the samples are properly drawn by trained social scientists. Once a sample is selected, the required subcontract infor- mation must be requested directly from the prime contrac- tors. State contracts usually include audit provisions allowing the state to obtain this information from the prime contrac- tor. This is very useful not only for conducting studies but also for program monitoring, so we suggest that all state DOT con- tracts include such provisions in the standard terms and con- ditions for construction and consulting contracts. Records retention requirements should apply for at least five years. Below is sample language from an actual contract: “AUDIT RIGHT AND RETENTION OF RECORDS AGENCY shall have the right, upon reasonable advanced notice and during ordinary business hours, to audit the books, records, and accounts of CONTRACTOR and its subcontractors that are related to this Project. CONTRACTOR and its subcontractors shall keep such books, records, and accounts as may be necessary in order to record complete and correct entries related to the Project. All books, records, and accounts of CONTRACTOR and its subcontractors shall be kept in written form, or in a form capable of conversion into written form within a reasonable time, and upon request to do so, CONTRACTOR or its subcon- tractor, as applicable, shall make same available at no cost to AGENCY in written form. CONTRACTOR and its subcontractors shall preserve and make available, at reasonable times for examination and audit by AGENCY, all financial records, supporting documents, statisti- cal records, and any other documents pertinent to this Agree- ment for the required retention period of the ____ Public Records Act, Chapter ____, ____ Statutes, as may be amended from time to time, if applicable, or, if the ____ Public Records Act is not applicable, for a minimum period of five (5) years after termination of this Agreement. If any audit has been initiated and audit findings have not been resolved at the end of the reten- tion period or three (3) years, whichever is longer, the books, records, and accounts shall be retained until resolution of the audit findings. If the ____ Public Records Act is determined by AGENCY to be applicable to CONTRACTOR and its subcon- tractors’ records, CONTRACTOR and its subcontractors shall comply with all requirements thereof; however, no confidential- ity or non-disclosure requirement of either federal or state law shall be violated by CONTRACTOR or its subcontractors. Any incomplete or incorrect entry in such books, records, and accounts shall be a basis for CONTRACTOR disallowance and recovery of any payment upon such entry. CONTRACTOR shall, by written contract, require its subcon- tractors to agree to the requirements and obligations of this Section.” Prime contractors are more likely to comply if the data request includes a letter from the Governor, the Director of Transportation, or other senior official explaining the impor- tance of the data collection effort to the state and encouraging prime contractors to cooperate. Follow-up telephone calls must be made to nonresponsive prime contractors, and they are most effective when they are made by the state. This is because the state has a pre-existing and in many cases an ongo- ing relationship with the prime contractor while the disparity or availability consultant does not. While the initial round of follow up can be made by the consultant if necessary, increas- ingly higher-level state DOT personnel must become involved for those prime contractors who fail to provide the requested information. Even under the best of circumstances, this entire process can be costly and time-consuming. The necessary staff resources to conduct this follow up should be planned for by the state if it is considering commissioning a disparity or avail- ability study and knows that it will need to supplement its exist- ing subcontract record. The combination of contract audit provisions, a letter, and follow-up communications will typically achieve a response rate accounting for 75% or more of the contract dollars in the sample. Extreme cases have required more stringent methods, such as auditing the prime contractor’s books or withholding progress payments on current work, to achieve cooperation. Finally, the longer the time from award, the more difficult it will typically be for the prime contractor to provide the 63 197If the state DOT makes local assistance grants to other public entities in the state, and the contracts funded with such grants are subject to the DBE Program then local recipients should be required to collect and maintain the same level of prime contract and subcontract data as the state DOT.

ting contract goals and the overall dollar-weighted DBE avail- ability estimates for use in annual goal setting. Case 2: Complete Pay Items. To determine the product dimension of the large state DOT’s construction contracting, we worked with agency staff to identify all construction con- tracts awarded during the study period. The agency then pro- vided detailed pay item-level cost estimates for each contract. Ultimately, more than 3,000 contracts consisting of almost 135,000 separate pay items and totaling almost $2.5 billion were included in the study. We matched each pay item number with its associated section from the agency’s Standard Specifications. Once all pay item numbers were linked with a specific section, we calculated a dollar-based weight to apply to each section. These weights were based on adjusted dollar amounts (i.e., original award amounts plus all change order amounts). Next, we assigned detailed industry codes to each section of the specification manual. Each section could be assigned multiple detailed industry codes, depending on that sec- tion’s scope of work. For sections assigned more than one detailed industry code, a second set of weights was applied to reflect the importance of each industry code in dollar terms within the section. These secondary weights were assigned with reference to published construction estimat- ing standards.199 As a check on the appropriateness of our assignment of detailed industry codes, we asked agency staff to review our selections, and we refined them further based on staff input. For example, we eliminated several industries that agency staff understood to have negligible opportunities for prime contracting or subcontracting. Input from agency staff resulted in several consolidations or expansions of our origi- nal industry code assignments. In this manner, as with the bid tabulation method, it was possible to produce the necessary detailed industry estimates of DBE availability and the overall dollar-weighted DBE availability estimates for use in annual goal setting. requested information. In our experience, response rates for contracts that are more than 5 years old are substantially lower than those for more recent projects. Where this sam- pling method must be employed, a study period longer than 5 years may not be feasible or supportable, and is probably not necessary. Impute Missing Data Using Bid Tabulations, Pay Items, or Similar Records If subcontract data are missing and obtaining the informa- tion directly from the prime contractors is not feasible, it is still possible to create a proxy for the missing information using bid tabulations, pay items, or similar records. However, these methods do not allow for the empirical determination of the geographic market area and are therefore inferior to and less defensible198 than obtaining data directly from the prime contractors. Moreover, this approach is problematic for disparity studies because of the uncertainty introduced into utilization and disparity statistics. We have constructed such proxies for two clients, using somewhat different methods based on the data that were available. Case 1: Representative Bid Tabulations. In collabora- tion with engineering staff from a major metropolitan tran- sit agency, we first identified the major types of contracts that were undertaken in a typical year: (1) building construction, (2) heavy elevated construction, and (3) signal construction. Next, the agency provided detailed cost estimates associated with nine upcoming representative construction projects— three building projects, three elevated heavy construction proj- ects, and three signal construction projects. We then worked with agency engineering staff to assign detailed industry codes to each major task and task dollars associated with each upcom- ing project. In this manner, it was possible to produce the necessary detailed industry estimates of DBE availability for use in set- 64 198 Cf. 49 C.F.R § 26.45(b), requiring that the recipient’s goal reflect its market, not the national goal. 199 See e.g., Richardson Engineering Services, Inc. (2000).