National Academies Press: OpenBook
« Previous: Chapter 2 - Designing Defensible DBE Programs
Page 29
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 29
Page 30
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 30
Page 31
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 31
Page 32
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 32
Page 33
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 33
Page 34
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 34
Page 35
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 35
Page 36
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 36
Page 37
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 37
Page 38
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 38
Page 39
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 39
Page 40
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 40
Page 41
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 41
Page 42
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 42
Page 43
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 43
Page 44
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 44
Page 45
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 45
Page 46
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 46
Page 47
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 47
Page 48
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 48
Page 49
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 49
Page 50
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 50
Page 51
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 51
Page 52
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 52
Page 53
Suggested Citation:"Chapter 3 - Model Disparity Study." National Academies of Sciences, Engineering, and Medicine. 2010. Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program. Washington, DC: The National Academies Press. doi: 10.17226/14346.
×
Page 53

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

29 The disparity and availability studies reviewed for this report differ not only in the elements included but also in their design and implementation. In this section, we compare the studies reviewed according to the key elements: the definition and use of geographic and product markets; the development of availability estimates; the analyses of state DOT contracting disparities; the analyses of economy-wide disparities; and the collection of anecdotal evidence. For each key element, we present the recommended approach judged by actual success or potential for success under strict scrutiny review. Determination of Relevant Geographic Market Area Recommended Approach. In high-quality disparity and availability studies, the relevant geographic market area iden- tifies those vendor locations that account for at least 75% of contract and subcontract102 dollar expenditures in the project database for the study period.103 The report should describe how the contract and subcontract data were used to make this determination and one or more tables should be presented showing the results. Location should be determined by linking the zip code of the contractor or subcontractor to the associated state and county. For multi-establishment firms, location does not have to be defined as the headquarters of the firm. If the firm has established a local presence, then it is appropriate to use that address for purposes of market determination. Since the two major contracting categories typically exam- ined in state DOT studies are construction and construction- related professional services,104 it is also advisable to make separate geographic market determinations for each category, as well as for a combined category.105 Review of Studies. Most of the studies followed the rec- ommended approach in most or all respects. When they did not, the following were the chief concerns: • Assuming, rather than empirically determining, the market area; • Not including subcontract data in the determination; • Using states rather than counties as the unit of analysis for the determination;106 • Not providing details or tables showing how the determi- nation was made; and • Not providing separate determinations for construction and construction-related professional services. Some studies went beyond the minimum recommended approach and provided separate geographic market calcula- tions by highway district as well as statewide. Determination of Relevant Product Market Recommended Approach. In high-quality disparity and availability studies, the relevant product market identifies the detailed industries that account for at least 75% of contract and subcontract dollar expenditures in the project database for the study period. The amounts accounted for by each industry should be listed by dollars and as a percentage of overall contract and subcontract spending.107 The report should describe how the contract and subcon- tract data were used to make this determination and one or more tables should be presented showing the results. C H A P T E R 3 Model Disparity Study 102To the extent that studies have not reconstructed or otherwise accounted for incomplete subcontractor or subconsultant data, the determination of the rele- vant geographic market area will be incomplete. 103A detailed description of the project database and the steps necessary to its assembly appears in Step A—Create a Database of Representative, Recent, and Complete State DOT Projects, infra. 104Sometimes also referred to as “architecture & engineering,” “design,” “pre- construction,” or just “consulting.” 105See, e.g., Sherbrooke, 345 F.3d at 973–74. 106This is less of a concern in very small states such as Delaware or Rhode Island. 107The percentage distribution by industry is used elsewhere in the study to calcu- late overall DBE availability as a dollar-weighted average of detailed industry level DBE availability. See infra, Carrying Out Final Step 1 Availability Calculations.

Detailed industry affiliation should be determined by assigning a four-, five-, or six-digit North American Indus- try Classification System (NAICS) code, as appropriate, to each contractor and subcontractor in the project database. For firms whose work qualifies under more than one NAICS code, the assignment should be made based on the firm’s pri- mary code unless there is enough information available to allo- cate the firm’s work by dollars across multiple industries.108 However, if multiple NAICS codes are assigned to firms, the study must be careful to avoid double-counting. We recommend using NAICS codes even if agencies use systems other than NAICS (such as Construction Specifica- tion Institute codes or other internal work code systems) to classify contract and subcontract work.109 This is because the data necessary to implement high-quality availability estimates are classified according to NAICS (i.e., Dun & Bradstreet data). Moreover, the courts are familiar with NAICS and Standard Industrial Classification (SIC, NAICS predecessor), but will be less so with other types of work classification systems. While all the studies examined provided results according to aggregate industry sectors (e.g., construction and construction- related professional services), only a few: • Calculated detailed industry weights to be used for produc- ing dollar-weighted overall availability estimates. • Provided detailed industry availability estimates and/or carried out utilization and disparity analyses by detailed industry. One study asserted that the lack of disaggregated data by industry was due to limitations of the state DOT’s contract and subcontract data: Some studies have provided for data at the . . . SIC . . . level and others have aggregated the data into business type categories such as architect & engineering; construction; professional services; and goods and non-professional services. The amount of data [dis]aggregation is generally limited by a governmental agency’s record-keeping format. The business type or procurement cate- gories identified above [architecture/engineering, construction, professional services, and goods and non-professional services] form the level of aggregation for this study.110 This is not a problem with the state DOT’s data but rather with the study’s methods. If the project database assembled for the study contains the name of the firm, its address, and a brief description of the project, then the study can assign a detailed SIC or NAICS code. Even without a project descrip- tion, lookups in Dun & Bradstreet, SelectPhone, InfoUSA, or even Google, will almost always turn up enough information to properly assign a detailed industry code to a firm. Without a dollar-weighted overall availability estimate, avail- ability in an industry with only $1,000 of contract and subcon- tract spending has as much impact on the overall availability estimate as availability in an industry with $1,000,000,000 of contract and subcontract spending.111 Without availability esti- mates by detailed industry, the state DOT is deprived of an extremely useful tool for conducting contract-level goal setting. Without utilization and disparity analyses by detailed industry, a study can be criticized for ignoring aggregation issues.112 Estimation of DBE Availability Recommended Approach. The recommended approach to estimating availability is a “custom census” designed to provide an accurate calculation of the current availability of DBEs in the relevant market.113 This is the only method under Part 26 that has received favorable judicial analysis.114 The custom census approach employs a seven-step analysis that (1) creates a database of representative state DOT projects, (2) identifies the appropriate geographic market for the state DOT’s contracting activity, (3) identifies the appropriate prod- uct market for the state DOT’s contracting activity, (4) counts all businesses in those relevant markets, (5) identifies listed minority-owned and women-owned businesses in the relevant markets, (6) verifies the ownership status of listed minority- owned and women-owned businesses, and (7) verifies the ownership status of all other firms. This method results in an overall DBE availability number that is a dollar-weighted average of all the underlying industry availability numbers, with larger weights applied to industries with relatively more spending and lower weights applied to industries with rela- tively less spending. The availability figure can also be sub- divided by race, ethnicity, and gender group, as well as by highway district, where appropriate.115 30 108If the state DOT’s contract data contain enough descriptive information about the nature of the work being performed, then this may be possible. Otherwise, the allocation becomes arbitrary. For example, should a firm working in two NAICS codes be classified as split between them 50/50, or 70/30, or 90/10? 109Commodities are sometimes classified according to the National Institute of Government Purchasing (NIGP) system. This system is commodity based rather than industry based. For studies that include commodities as a contracting cat- egory, we would recommend classifying according to NAICS. 110OH (2001, Ch. 3, n.p.). 111U.S.DOT “Tips for Goal-Setting in the Disadvantaged Business Enterprise (DBE) Program” (“Whenever possible, use weighting . . . to insure that your Step One Base Figure is as accurate as possible.”). 112Simpson’s Paradox states that outcomes observed when data are aggregated may not be observed when data are disaggregated, and vice versa. Some courts have criticized public agencies for ignoring this issue. See Appendix A, Non-DBE Subcontract Data is Just as Important as DBE Subcontract Data, for further dis- cussion of this phenomenon. 113See Northern Contracting III, 473 F.3d at 723; Sherbrooke, 345 F.3d at 973. 114Sherbrooke, 345 F.3d at 973; see Concrete Works IV, 321 F.3d at 966 (custom census was “more sophisticated” than earlier studies using Census data and bidders lists). 115Recommended Approach to Measuring DBE Availability, supra, provides an extended discussion of how to implement the “custom census” method.

Review of Studies: Different Approaches to Measuring Availability A variety of approaches to measuring availability appear in the 25 studies we reviewed. The recommended approach, dis- cussed in detail in Recommended Approach to Measuring DBE Availability, was used in seven of 25 studies. Another ten studies relied primarily on internal state DOT lists of contrac- tors and subcontractors, such as certified DBE directories, bidders lists, prequalified contractor lists, registered subcon- tractor lists, licensed contractor lists, plan-holder lists, or lists of winning contractors or subcontractors. Internal lists were sometimes supplemented with lists gathered from other sources. However, the internal lists remained primary. We will refer to this as the “bidders list approach.” The bidders list approach has appeal since these types of lists tend to be readily available and since it seems natural to base an availability measure on firms that are already work- ing or trying to work with the state DOT. Some have also posited that inclusion on the lists is a measure of whether DBEs are “ready, willing and able.”116 However, there are sev- eral drawbacks to this approach: • The remedial aspect of Congress’ intent in passing the DBE Program is easily lost because it limits the availability pool to only those firms with which the state DOT is already familiar or already does business. A broader measure is preferred.117 • To the extent that there is still discrimination against DBEs, they are likely to be underrepresented on such lists. This approach is therefore likely to lead to lower estimates of availability than are actually present in the relevant markets. • To the extent that economy-wide discrimination in the rel- evant markets channels DBEs into state DOT work because of the remedy of the DBE Program, DBEs may be overrep- resented on such lists. This approach is therefore likely to lead to higher estimates of availability than are actually present in the relevant markets. • To the extent that such lists are produced using different methods and criteria (e.g., criteria for DBE certification is different from criteria for prime or subcontractor prequal- ification), such lists mix “apples and oranges.” • The bidders list approach often calculates availability sepa- rately for prime contractors and subcontractors. First, firms may perform as subcontractors on some jobs and prime contractors on others, so assigning firms to separate cate- gories may not yield an accurate picture. This may be espe- cially true for state DOTs that are implementing aggressive race-neutral measures to reduce barriers for DBEs seeking to perform prime contracts (e.g., setting aside smaller con- tracts for bidding by small firms, providing bonding and financing support, reducing prequalification burdens). Further, if DBEs suffer discrimination in the relevant mar- kets, then their ability to progress from subcontractors to prime contracts is likely to have been affected.118 Creating separate availability measures for primes and subs serves to exacerbate rather than remedy this problem.119 • The bidders list approach rarely includes attempts to validate whether firms are correctly classified by race and gender. A variation on the bidders list approach was used in four studies. Three studies used one such variation and a fourth used another. We will refer to these collectively as the “bid- ders list approach with capacity adjustments.” This method seeks to test whether firms with lower rev- enues won fewer contracts and subcontracts while firms with higher revenues won more. The method was implemented by surveying state DOT bidders to collect data on revenues and then comparing that data to existing data on contract and subcontract awards using regression analysis. However, the three studies that employed this “capacity” test could not find a statistically significant relationship between revenues and contract awards.120 In part, the data did not fit because firms of given revenue sizes won a variety of contracts and subcontracts both large and small.121 Despite finding no empirical support for the hypothesized relationship between revenue size and propensity to win con- tract and subcontract awards, all three studies proceeded without explanation to eliminate firms from their availability calculations if they had revenue levels lower than those of sur- vey respondents that had actually won state DOT contracts or subcontracts. This resulted in an unjustified downward adjustment to most of the initial availability estimates. 31 116See, e.g., Concrete Works IV, 321 F.3d at 984 (discussing plaintiff’s argument that bidding data are the only measure of availability). 117Northern Contracting III, 473 F.3d at 723 (The “remedial nature of the federal scheme militates in favor of a method of DBE availability calculation that casts a broader net.”). 118See Northern Contracting II, 2005 U.S. Dist. Lexis at *74 (“The ability of DBEs to compete successfully for prime contracts may be indirectly affected by discrimina- tion in the subcontracting market, or in the bonding and financing markets. Such discrimination is particularly burdensome in the construction industry. . . .”). 119See Appendix B for further discussion. 120Moreover, none of these studies describes why a firm’s revenue can be appro- priately construed as a race-neutral explanatory variable. Additional discussion of the difficulties involved in using variables, such as revenue, that can be impacted by discrimination to explain success or failure in the award of contracts or subcontracts appears in Appendix B. 121See Concrete Works IV, 321 F.2d at 981 (“At trial, Denver introduced evidence that the median number of employees of all construction firms in the Denver MSA is three and presented testimony that even firms with few permanent employees can perform large, public contracts by hiring additional employees or subcontrac- tors and renting equipment. Additionally, the district court found that ‘most firms have few full-time permanent employees and must grow or shrink their perfor- mance capacity according to the volume of business they are doing.’ ”).

A fourth study also implemented a “capacity” adjustment with its bidders list approach to measuring availability. This adjustment, however, was implemented in a way that was not obviously biased by past discrimination against DBEs. The study performed a regression analysis of total dollars awarded to non-DBE contractors and subcontracts against the largest successful bid from each non-DBE contractor or subcontrac- tor. It then applied the statistical relationship implied by the non-DBE regression to DBEs and used the results to estimate “expected utilization” for DBEs. The expected utilization fig- ure was then compared to the actual utilization figure to form a disparity ratio. This approach is roughly analogous to the approach used to estimate potential business formation among minorities and women.122 As a potential measure of disparity, it is quite useful. As a measure of availability, however, it suf- fers from the same infirmities as other bidders list approaches to availability measurement.123 Moreover, from a practical standpoint, the necessary data on successful and unsuccessful bid values, particularly at the subcontractor level, are rarely available. This study did not describe how complete the data were for this exercise, although elsewhere there are indications that a significant amount of non-DBE subcontractor data were in fact missing. The study’s regression results do not report the number of DBE and non-DBE observations included. These omissions made it difficult to evaluate this approach’s litigation poten- tial, as an adverse party’s expert might be unable to duplicate the results from the study. Another method used to account for capacity appears in three other studies we reviewed. We will refer to this as the “custom census approach with capacity adjustments.” The three studies using this method started with Dun & Bradstreet data as their source for the availability denomina- tor. The Dun & Bradstreet data were then restricted to the rel- evant geographic market based on the location of contractors and subcontractors on federally assisted state DOT contracts. The data were further restricted to the relevant product mar- ket by matching the names and addresses of firms that had bid on state DOT contracts as prime contractors or first-tier subcontractors with the Dun & Bradstreet data to determine the relevant SIC or NAICS codes. All the firms identified in this manner were then surveyed by telephone. Firms that could not be reached were dis- carded, including firms that may actually have worked for the agency as prime contractors or subcontractors. Any firms that had not worked or attempted to work on construction or construction-related professional services contracts were also discarded. In one of these studies, for example, the original pool of 39,911 establishments was reduced by over 90%— leaving only 3,398 firms in the availability denominator. The process used in the other two studies was very similar. As part of the telephone survey, each firm was asked whether it was 51% or more minority or female owned. If it reported being minority owned, then the survey asked which race or ethnicity comprised the bulk of the minority ownership. In this way, 32% of the 3,398 firms identified in the survey were identified as minority owned and/or woman owned. The study examined five years of DOT contract and sub- contract data. From this, dollar-based weights were derived to account for the amount of contract and subcontract spending in each SIC or NAICS code. Although the study does not describe the method, additional weights were also derived to account for prime contractor versus subcontractor status, contract dollar size categories, and regions within the state. Based on this weighting procedure, the overall estimate of availability fell from 32% to under 18%. Finally, firms that were considered too large to meet U.S.DOT DBE certification guidelines were excluded from the availability measure. Any DBE construction firm that responded to the telephone survey and reported prior year gross revenues exceeding $10 million,124 or any DBE engineer- ing firm reporting more than $5 million, was dropped from the calculation. The final availability figure arrived at through this process was 13.5%. As with the “bidders list with capacity adjustments” approach, the “custom census with capacity adjustments” approach is biased downward. It reduces the availability per- centage by controlling for factors that are likely to be directly affected by the presence of discrimination in the relevant mar- kets. Whether firms have worked or attempted to work on state DOT projects have been awarded prime contracts or the size of those contracts should not be used to limit the DBE avail- ability measure.125 Not only is this a problem in its own right, but also it may hide the existence of discrimination because a downward bias in availability can lead to a conclusion of no significant disparity when, in fact, a disparity exists.126 32 122A business formation regression is calculated for nonminority males and the resulting model is applied to minorities and women to derive their expected rate of business formation. See Chapter Three, Economy-Wide Disparity Analysis for the Relevant Markets, infra. 123See Chapter Three, Estimation of DBE Availability, supra. 124Cf. 49 C.F.R. § 26.65, establishing the size limits for DBE eligibility as those imposed by the SBA under 13 C.F.R. Part 121. SBA limits at the time of the study ranged from $13 million for specialty trade contractors to $31 million for gen- eral and heavy construction firms. The current overall DBE Program cap is $20.41 million. No explanation is provided in the study of how the lower ceiling was determined or why it was imposed. 125See also the discussion of capacity in Appendix B. 126Based on the results of one of these studies, the state DOT has sought a waiver to exclude DBEs owned by Hispanic males and Subcontinent Asian males from its race-conscious measures, i.e., denying their eligibility for credit toward meet- ing subcontracting goals based on a finding of no significant disparity.

Disparity study authors, along with a number of courts, have wrestled with the concept of DBE or M/WBE “capacity.” Concerns regarding capacity arise not only in the context of availability measurement but also in the context of assessing disparities. Appendix B provides an extended discussion of “capacity” issues in the context of disparity and availability studies and related litigation. In summary, where “capacity,” whether measured by firm revenues, employment size, or some other metric, is influenced by the presence of discrimi- nation in the relevant markets, it is inappropriate to use such measures to “correct” or “adjust” Step 1 availability or Step 2 disparity statistics. Finally, one study calculated DBE availability six different ways yielding six different estimates. Such an approach can be a drawback in litigation as it provides a plaintiff an opportu- nity to cast doubt on the entire study by pointing out that the multiple estimates cannot all be correct.127 In fairness, how- ever, this particular study is one of the earliest that we reviewed, from a period when recipients were still striving to find an approach to availability measurement that the courts would find acceptable.128 Recommended Approach to Measuring DBE Availability Introduction. The determination of DBE availability is the cornerstone of an availability or disparity study. Accurate, comprehensive estimates of availability are critical. An expan- sive concept of availability is important because, as the courts have held, looking beyond the recipient’s contracting results helps to further Congress’ remedial intent. [T]he purpose of the overall goal—and, in fact, the DBE pro- gram, as a whole—is to achieve a “level playing field” for DBEs seeking to participate in federal-aid transportation contracting. To reach a level playing field, recipients need to examine their programs and their markets and determine the amount of par- ticipation they would expect DBEs to achieve in the absence of discrimination and the effects of past discrimination.129 Limiting the inquiry to agencies’ internal lists of firms can- not fully result in an annual goal that “reflect[s] . . . the level of DBE participation you would expect absent the effects of discrimination.”130 Such an approach reflects in part the cur- rent effects of past or current discrimination and so should not be used to limit the examination of how the market would look if it became discrimination free. Although 49 C.F.R. § 26.45(c)(1) permits the use of DBE directories and Census Bureau data to estimate availability, this is a less than optimal approach. It necessitates a compar- ison of “apples to oranges” because the methods used to identify and certify firms as DBEs are entirely different from the methods used by the Census Bureau to count businesses or business establishments for inclusion in CBP Survey of Business Owners (SBO) or other statistical databases. As a result, no defensible comparisons are likely to result from dividing figures from one of the latter sources by figures from the former. What has been termed the “custom census” approach to measuring DBE availability, when properly executed, is supe- rior to the other methods allowable under 49 C.F.R. § 26.45 for at least four reasons. First, it provides an internally con- sistent and rigorous “apples to apples” comparison between firms in the availability numerator and those in the denomi- nator. Second, by “casts[ing] a broader net” it comports with the remedial nature of the DBE Program. Third, a custom census is less likely to be tainted by the effects of past and present discrimination than the other methods. Finally, it has been upheld by every court that has reviewed it. The Tenth Circuit found the custom census approach to be “a more sophisticated method to calculate availability than the earlier studies.”131 Likewise, this method was suc- cessful in the defense of the DBE Programs for Mn/DOT132 and IDOT,133 as well as the M/WBE construction program for the City of Chicago.134 The following are the seven steps to the custom census approach: (A) Create a database of representative, recent, and complete state DOT projects; (B) Identify the contracting activity’s relevant geographic market; (C) Identify the contracting activity’s relevant product mar- ket for the contracting activity in question; (D) Count all businesses in the relevant markets; (E) Identify listed minority-owned and women-owned busi- nesses in the relevant markets; (F) Verify the ownership status of listed minority-owned and women-owned businesses; and 33 127Cf. Associated General Contractors of America v. City of Columbus, 936 F. Supp. 1363, (S.D. Ohio 1996) (discussing various measures of availability). 128Another of these early studies presented seven different availability measures. However, the agency ultimately chose just one (a bidders list approach) to use for disparity testing purposes. 12964 Fed. Reg. 5108. 130Ibid. 131Concrete Works of Colorado, Inc. v. City and County of Denver, 321 F.3d 950, 966 (10th Cir. 2003) (Concrete Works IV), cert. denied, 540 U.S. 1027 (2003). 132Sherbrooke Turf, Inc. v. Minnesota Department of Transportation, 345 F.3d. 964 (8th Cir. 2003), cert. denied, 541 U.S. 1041 (2004). 133Northern Contracting, Inc. v. Illinois Department of Transportation, 473 F.3d 715 (7th Cir. 2007). 134Builders Association of Greater Chicago v. City of Chicago, 298 F. Supp.2d 725 (N.D. Ill. 2003).

(G) Verify the ownership status of all other firms in the rele- vant markets. Each step is described in more detail below, in the context of estimating availability for construction and construction- related professional services—the two most commonly studied (and litigated) sectors in disparity and availability studies. The methods generalize to other industry sectors as well. Step A—Create a Database of Representative, Recent, and Complete State DOT Projects. The first step is to create a database of state DOT projects. This database provides the empirical basis for many of the statistical analyses in the study. Each project in the database should contain information on all relevant prime contracts and all associated first-tier agree- ments with subcontractors, subconsultants, or suppliers.135 It is imperative to obtain subcontract information for both DBEs and non-DBEs. If all of the necessary subcontract data have been maintained, then all of it should typically be included in the project database. If the state DOT has not collected and maintained this data, then it will be necessary to either request the information directly from each relevant prime contractor or consultant or reconstruct the required information by other means.136 The project database must be representative of the type of work usually undertaken by the state DOT. It is therefore advisable to study several years’ worth of contract and sub- contract data so that atypical projects do not unduly affect the statistical analysis.137 Most state DOT disparity and availability studies have relied on a study period of 5 years—typically the most recent five full fiscal or calendar years.138 Of the 28 state DOTs that have per- formed, or are currently performing, a disparity or availability study, we were able to obtain the number of years of contract and subcontract data studied for 25. This is shown in Table 5.139 Of these 23 studies, study periods ranged from 2 years to 14 years. A study period of only two years, in our view, is likely to be inadequate due to the smaller sample sizes yielded. For example, a significant share of projects awarded during that period may still not yet be complete. This could cause the resulting database to be biased by the exclusion of larger proj- ects with longer completion times. The median study period length was 5 years and the aver- age was 5.3 years. The disparity or availability studies intro- duced as evidence in the Sherbrooke and Northern Contracting cases covered 5 years. Because the types of subcontracting that occur early in a project are often different from those that occur later, we rec- ommend that only complete or substantially complete proj- ects be studied; that is, enough of the project has been finished (e.g., it is now open to traffic) so an accurate picture of all sub- contracting activity emerges. If payment data are being used in the analysis (as opposed to or in addition to award data), then including incomplete projects could lead to inaccurate conclusions about the relative weight of different subcontract- ing activities. In order to assist in the determination of the level of DBE participation that would be expected in a race-neutral con- tracting environment, it is important that the project database include contracts with and without DBE goals.140 In states without a M/WBE program for state-funded contracts, this can be achieved by including non-federally assisted projects in the database. For states without these data, more creative approaches can be applied. For example, in preparing for the Northern Contracting trial, IDOT initiated a “zero goals” experiment, where selected prime contracts that ordinarily would have been subject to a DBE goal based on the scopes of work and DBEs certified in those industries were solicited with a “zero goal.” The result was that DBEs received approximately 1.5% of the total value of these contracts. This evidence of the results of totally race-neutral measures was found to be pro- bative by the court.141 Several courts have likewise held that a large contrast between DBE participation on contracts with and without goals can be probative of the continuing need for race-conscious remedies.142 34 135Hereafter, unless otherwise indicated, we refer to these collectively as “subcon- tractors” or “subcontracts.” 136For additional discussion of the subcontract data collection issue, see Appendix A. 137In some cases, it can be appropriate to ignore or prorate unusual projects. For example, in Denver’s study, 1999 projects associated with the construction of the Denver International Airport were excluded from the main analysis, because this sort of project was not representative of typical Denver construction projects and was unlikely to be undertaken again for decades. In that same study, several large bond-funded public library construction projects that were likely to be under- taken only once per decade were prorated. Since the study period was 5 years, half of the value of these projects was included in the database. 138Typically, the federal fiscal year (which runs from October through Septem- ber) is used, although calendar years are sometimes used depending on the study’s scope and on organization of the agency’s data. 139Of the remaining three, one is ongoing (North Carolina), one could not be located (Florida), and one was never released (Tennessee). 140One objective of the DBE Program is to “assist the development of firms that can compete successfully in the marketplace outside the DBE program.” 49 C.F.R. § 26.1(f) (emphasis added). A disparity study can examine how DBEs are faring on contracts that are not federally assisted, which may provide information that is useful to the agency in this regard, as well as the need to continue to use race- conscious goals to meet the annual goal. 141Northern Contracting III, 473 F.3d at 719. 142See, e.g., Western States, 407 F.3d at 992 (Congress properly considered evi- dence of the “significant drop in racial minorities’ participation in the construc- tion industry” after state and local governments removed affirmative action provisions); Adarand VII, 228 F.3d at 1186 (evidence included “studies of local subcontracting markets after the removal of affirmative action programs”); Con- crete Works IV, 321 F.3d at 984–85.

35 Table 5. Study period length for state DOTs that are currently performing or have recently performed disparity or availability studies. State Consultant Years of Contract Data in Most Recent Study NC EuQuant ongoing FL MGT of America, Inc. not obtained TN Mason Tillman Associates, Ltd. not released LA D.J. Miller & Associates 2 NJ Mason Tillman Associates, Ltd. 2 CO MGT of America, Inc.a 3.5 ID BBC Research & Consulting 4 AK D. Wilson Consulting Group, LLC 5 CA BBC Research & Consulting 5 CO D. Wilson Consulting Group, LLCb 5 NV BBC Research & Consulting 5 NM BBC Research & Consulting 5 OH D.J. Miller & Associates 5 NE MGT of America, Inc. 5 NC MGT of America, Inc.c 5 VA MGT of America, Inc. 5 HI NERA Economic Consulting 5 IL NERA Economic Consulting 5 MD NERA Economic Consulting 5 MN NERA Economic Consulting 5 NY NERA Economic Consulting 5 WA NERA Economic Consulting 5 AZ MGT of America, Inc. 6 GA Boston Research Group 6 MT D. Wilson Consulting Group, LLCb 6 MO NERA Economic Consultingd 6.5 OR MGT of America, Inc. 8 SC MGT of America, Inc. 14 Notes: (a) Colorado’s current study is ongoing—this figure is for its 2001 study; (b) study is ongoing, study period based on consultant’s proposal; (c) North Carolina’s current study is ongoing—this figure is for its 2004 study; (d) figure is for construction contracts—only 3 years of data were available for construction-related professional services and local assistance contracts.

For each project in the database, the following are key prime contract fields that should be included:143 • Unique contract number for prime contract; • Brief description of the prime contract; • Department or subdepartment for which the project was performed; • Date of prime contract award; • Original dollar amount of prime contract; • Total dollar amount of prime contract (including all change orders); • Date of completion or substantial completion; • Total dollar amount paid through completion or substan- tial completion; • Business name of prime contractor or consultant; • Unique identification code for the prime contractor or consultant;144 • Street address, city, state, and zip code of prime contractor or consultant; • Telephone number of prime contractor or consultant; • Contact person name and title for prime contractor or consultant; • Certification status of prime contractor or consultant; • Race and gender of prime contractor or consultant ownership;145 • Brief description of prime contractor work specialties; • Indication of whether prime contract was federally assisted; • DBE goal for the prime contract; and • DBE goal for associated change orders.146 For each project in the database, the following key sub- contract fields should be included: • Unique contract number for prime contract; • Brief description of the subcontract; • Original dollar amount of subcontract; • Total dollar amount of subcontract (inclusive of all change orders); • Total dollar amount paid through completion or substan- tial completion; • Business name of subcontractor or consultant; • Unique identification code for the subcontractor; • Street address, city, state, and zip code of subcontractor; • Telephone number of subcontractor; • Contact person name and title for subcontractor; • Certification status of subcontractor; • Race and gender of subcontractor ownership;147 and • Brief description of subcontractor work specialties. Steps B & C—Identify the Relevant Markets. Markets have both a geographic and an industry dimension.148 Once the project database is assembled, the next step in determin- ing availability is to identify the geographic locations and the industries from which the state DOT draws the preponder- ance of its prime contractors and consultants and from which the state DOT’s prime contractors and consultants draw the preponderance of their subcontractors, subconsultants, and suppliers. The unit of analysis to define “preponderance” should be the number of contract dollars since subcontracting goals are set as a percentage of total dollars awarded and since contracts and subcontracts vary greatly by dollar size.149 36 143For projects procured using the construction manager or construction manager at-risk process, similar data should be included for the construction manager. 144The use of unique codes to identify vendors in contracting records is preferable to using the vendor’s name as an identifier, since vendor names can (and usually are) entered inconsistently. Reconciling this inconsistent information can be a huge, costly, and time-consuming task for a study consultant, especially in con- tract files containing thousands (or millions) of records. The use of unique ven- dor IDs eliminates the inconsistency inherent in the use of names as identifiers. It also facilitates the integration of contracting and contractor data into larger rela- tional database systems, which have many advantages, from an information man- agement perspective, over more traditional, or flat file database management systems. See, e.g. Wiley Publishing, Inc. (2008). 145If the contractor is publicly owned, this should be noted. In the absence of information to the contrary on a specific company, however, it is appropriate to treat publicly owned companies as non-DBEs. According to the Census Bureau (2008, Tables 2 and 5), of the total value of stock and mutual fund shares owned by U.S. households in 2002 (the latest data available), black households owned only 3.5%, Hispanic households 6.9%, Asian households 4.3%. Non-Hispanic white households owned 77.3%. 146We have observed many instances where DBE goals on awarded contracts have not been applied to change orders. Since change orders can often account for a large share of a construction project’s dollar value, this can lead to a significant dilution of DBE participation. 147If the subcontractor is publicly owned, this should be recorded in lieu of race and gender. See footnote 145. 148See, e.g., Spectrum Sports, Inc. v. McQuillan, 506 U.S. 447, 459 (1993). 149For example, data from WSDOT for FFY 1999–2003 [NERA Economic Con- sulting (2005)] included 624 federally funded prime construction contracts val- ued at $1.52 billion. These contracts ranged in value from a minimum of $54,000 to a maximum of $204 million, with a median of $961,000 and an average of $2.44 million. Of the 624 prime contracts, 182 (29%) were valued at $500,000 or less. However, these 182 contracts accounted for less than 3.5% of total dollars. Even if we exclude the largest prime contracts (e.g., those in excess of $20 million), which collectively account for 35% of all dollars, the share of total dollars accounted for by contracts of $500,000 or less is still only 5.0% of the total. A sim- ilar pattern is observed in subcontracts. In the WSDOT data, there were 4,998 subcontracts totaling almost $560 million. Subcontracts ranged in size from $100 to almost $15 million, with a median of $18,000 and an average of $111,000. Sub- contracts of $15,000 or less accounted for 46% of all subcontracts but less than 2.5% of all subcontract dollars. Once again, even if we ignore all subcontracts val- ued above $2 million (27% of all subcontract dollars), subcontracts of $15,000 or less account for just over 3.0% of all subcontract dollars. Finally, in the WSDOT data, more than 96% of all DBE participation occurred on contracts valued above $500,000. Similar patterns are observed in the construction-related professional services contracts data. Not only does this example demonstrate the wide varia- tion in prime contract and subcontract sizes, but it also casts in doubt a method that limits most or all statistical analyses to prime contracts of $500,000 or less. This approach will cause a very large share of its contract spending, including the DBE spending, to go unanalyzed. See, e.g., Texas Building and Procurement Com- mission (2007, 5–6).

Using the project database, the county in which each con- tractor is located can be identified using the firm’s zip code. Then, the percent of dollars awarded and/or paid to contrac- tors and subcontractors in each state/county unit can be cal- culated. Drawing on work in the antitrust field, a geographic market can be defined as that area in which the state DOT operates and as well as the area where it “can predictably turn” to obtain construction services and where “the vast bulk” of the agency’s contract and subcontract dollars are spent.150 Although there is no single numerical percentage that deter- mines “the vast bulk,” a figure of 75% is often employed as a reasonable approximation.151 Table 6, drawn from an actual availability study, provides an example of how the informa- tion from the project database can be arrayed by geography to assist in determining the appropriate geographic market. For that study, the geographic market was defined to be the state as a whole. In order to identify the product market, each contactor and subcontractor in the project database can be assigned a primary industry code, using the NAICS system. If enough detail is provided in the project database, the project name and description can be used to assign primary industry codes to prime contractors and the subcontract task descriptions can be used to assign codes to subcontractors. When the description of the task does not clearly indicate the appropriate NAICS code for a contract or subcontract, lookups can be performed using Dun & Bradstreet’s Market- Place, SelectPhone, InfoUSA, or other sources. Most of these sources still use SIC codes, the predecessor to the NAICS sys- tem, to classify firms. Crosswalk tables allowing conversion between SIC and the several editions of NAICS and vice versa are available online from the Census Bureau.152 In construction and construction-related professional services, a four-digit NAICS code is most comparable to a four-digit SIC code for general contracting categories.153 For specialty trades and pro- fessional services, a five-digit NAICS code is most comparable. Once NAICS codes are assigned to all firms in the project database, the percent of dollars awarded and/or paid to con- tractors and subcontractors in each detailed industry can be calculated to determine which ones are most relevant for a state DOT’s contracting activity. Some NAICS codes will have much larger percentage shares than others, as some industries account for a larger share of agency contract spending than others. These percentage shares are referred to as “product market weights” because they will be used to create an over- all availability figure that is a weighted average of all the individual four-digit and five-digit NAICS level availability estimates, with the weights being the percentage dollar shares of spending. Table 7, drawn from an actual availability study we performed, provides an example of how the information from the project database can be arrayed by industry (in this case using SIC rather than NAICS codes) to assist in deter- mining the appropriate product market. As Table 7 shows, 22 SIC codes were identified in this state DOT’s subrecipient contracting market. Ninety percent of such contracting and subcontracting activity occurred in just eight industries and that one industry, SIC 1611, accounted for 24% of all such activity. As with the geographic market, it is important that the dis- parity or availability study captures the “vast bulk” of an agency’s spending. Here too, the courts have not established any clear numerical boundary.154 Since one of the deliverables from a high-quality study should be detailed (i.e., four-digit or five-digit NAICS) estimates of DBE availability that can be used to assist in establishing DBE contract goals, the agency should have as many relevant NAICS codes included in its study as possible. On the other hand, it is not cost-effective to 37 Construction (%) Subrecipients (%) 82.1 17.9 86.7 Location Inside State Outside State Inside Metropolitan Statistical Area Outside Metropolitan Statistical Area 13.3 Consulting (%) 83.8 16.2 89.2 10.8 82.4 17.6 99.6 0.4 Table 6. Distribution of contract dollars by contract category. 150See, e.g., Tampa Electric Co. v. Nashville Coal Co., 365 U.S. 320, 367 (1963); United States v. Philadelphia National Bank, 374 U.S. 321, 359 (1963); also Areeda, Kaplow and Edlin (2004). 151This was the benchmark employed in the availability studies that were upheld in the Sherbrooke and Northern Contracting cases. 152See NAICS at http://www.census.gov/epcd/www/naics.html. 153Four digits is the most detailed classification available in the SIC system. 154See fn. 149; however, it is not advisable to structure a study so that only a small fraction of overall spending is captured.

38 Table 7. Product market for subrecipient contracts. SIC Code SIC Description Percentage Cumulative Percentage 1611 Highway and Street Construction 24.26 24.26 1622 Bridge, Tunnel, and Elevated Highway 22.10 46.35 1794 Excavation Work 14.69 61.04 1771 Concrete Work 11.43 72.48 1623 Water, Sewer, and Utility Lines 6.02 78.49 1542 Nonresidential Construction, n.e.c. 5.30 83.80 1629 Heavy Construction, n.e.c. 3.42 87.22 0782 Lawn and Garden Services 3.27 90.49 1731 Electrical Work 3.12 93.61 3273 Ready-Mixed Concrete 1.44 95.05 1521 Single-Family Housing Construction 1.08 96.13 5051 Metals Service Centers and Offices 1.05 97.18 1791 Structural Steel Erection 0.86 98.05 4212 Local Trucking without Storage 0.73 98.78 1541 Industrial Buildings and Warehouses 0.39 99.17 7359 Equipment Rental and Leasing, n.e.c. 0.30 99.47 8711 Engineering Services 0.22 99.69 3272 Concrete Products, n.e.c. 0.18 99.86 1721 Painting 0.06 99.92 5063 Electrical Apparatus and Equipment, Wiring Supplies, and Construction Materials 0.04 99.97 5172 Petroleum and Petroleum Products Wholesalers, except Bulk Stations and Terminals 0.02 99.98 3446 Architectural Metal Work 0.02 100.00 TOTAL $48,540,859

create availability estimates for NAICS codes in which the total spending over a 5-year study period is relatively negligi- ble; some balance must be struck. Our experience is that in using a well-constructed project database, it is usually possi- ble to analyze at least 80% of contract spending and in some cases as much as 95% or more.155 Steps D & E—Count All Businesses in the Relevant Mar- kets and Identify Listed Minority-Owned and Women- Owned Firms. Once the geographic and product markets have been defined, it is possible to define the “baseline” pop- ulation of relevant businesses for the study. In the custom census approach, Dun & Bradstreet’s MarketPlace database is used to define the baseline business population. MarketPlace is the most comprehensive and objective available micro- level database of U.S. businesses.156 MarketPlace contains over 14 million records, is updated continuously, and revised each quarter. Using MarketPlace, it is possible to purchase a list of all businesses within the geographic market area that have an NAICS code to which a product market weight has been assigned using the project database. While extensive, MarketPlace does not sufficiently identify businesses owned by minorities or women. That is, although these firms are included in MarketPlace, they are not always identified as being minority or women owned. Although many such businesses are correctly identified in MarketPlace, experi- ence has demonstrated that many are not. For this reason, the minority- and women-owned share of the baseline business population cannot be calculated directly from MarketPlace. Doing so would yield an availability estimate that was lower than the actual share of minority- and women-owned busi- nesses in the relevant markets. To compensate for this limitation, the custom census method supplements the existing MarketPlace race and gender identifiers with race and gender identifiers from other directo- ries and business listings gathered within and around the rele- vant geographic market. This requires conducting an “extensive and intensive” search for information on minority-owned and women-owned businesses in the relevant geographic market.157 Beyond the information already in MarketPlace, this includes obtaining the state’s Unified Certification Program directory as well as DBE listings from numerous other public and private entities in the relevant market. As an example, the following is a listing of directories obtained and/or agencies contacted as part of the WSDOT (2006) availability study: WSDOT certified DBE directory Associated General Contractors of Washington Bank of America Supplier Diversity Program Black Chamber of Commerce Pacific Northwest Boeing Company Supplier Diversity Program Boise Cascade Corp Supplier Diversity Program Business Research Services National Directory of Minority- Owned Businesses Business Research Services National Directory of Women- Owned Businesses Caltrans Central Contractor Registration database CH2M Hill Chevron/Texaco Supplier Diversity Program City of Bellevue City of Olympia City of Portland Sheltered Market Program City of Seattle Boost Program City of Seattle Vendor & Contractor Registration City of Spokane City of Tacoma City of Vancouver City of Olympia Coca-Cola Supplier Diversity Program Community Capital Development SMWBE list Conoco/Phillips Supplier Diversity Program Urban League of Metropolitan Seattle Diversity Information Resources Georgia-Pacific Supplier Diversity Program Howard S. Wright Construction Supplier Diversity Program Idaho Transportation Department King County Kroger Company Microsoft Supplier Diversity Program Montana Department of Transportation National Association of Minority Contractors National Association of Women Business Owners—Inland Northwest Chapter National Association of Women in Construction (various Chapters) National Center for American Indian Economic Development National Minority Business Council Nevada Department of Transportation Nike Supplier Diversity Program 39 155This can be done by excluding all prime contracts below a certain dollar threshold, say $25,000, or by cumulating total spending by NAICS code and then excluding contract spending in those NAICS codes above a certain percentage cutoff, say 90%, or by some combination of these two methods. See also the dis- cussion at fn. 149. 156“Micro-level” means a database where the individual firms within the database can be identified. Many Census databases, by contrast, simply provide counts of businesses in different categories. 157We say “extensive” because, as the list below illustrates, a wide variety and large number of organizations were contacted as potential sources of information on minority- and women-owned businesses. We say “intensive” because obtaining a given list in the necessary timeframe with the necessary information (i.e., race and gender identification in addition to standard data such as business name, address, and telephone number) and in a usable format often requires a great deal of patience and perseverance.

Nordstrom Department Stores Supplier Diversity Program Northwest Minority Business Council Northwest Native American Business Development Center Oregon Association of Minority Entrepreneurs Oregon Office of Minority, Women and Emerging Small Business Pepsico Port of Portland Port of Seattle Port of Tacoma Qwest Communications Raytheon Safeco Insurance Company Supplier Diversity Program Seattle Mariners Supplier Diversity Program Seattle Monorail Project Seattle/Washington State Minority Business Development Center Sound Transit Diversity Programs South Puget Sound Hispanic Chamber of Commerce Starbucks Supplier Diversity Program Tabor 100 (Northwest Association of African-American Businesses) Tacoma Housing Authority Thurston County U.S. Army Corps of Engineers University of Washington W.W. Grainger Co. Supplier Diversity Program Washington Mutual Washington State Hispanic Chamber of Commerce Washington State Office of Minority & Women’s Business Enterprise Wells Fargo Supplier Diversity Program Women and Emerging Small Business Women Business Owners of Puget Sound Women’s Business Enterprise National Council Xerox Corporation. Not every agency in the list above ultimately provided a directory or listing to the study team. Some entities use the lists of other agencies, some do not track race or gender infor- mation (rendering the list much less useful for disparity or availability study purposes), and some had lists but were unwilling or unable to cooperate with the study effort. All of the lists and directories obtained in this manner are combined in a standardized format. Duplicate entries are elim- inated and information from multiple records consolidated and reconciled. Obvious out-of-scope and non-M/W/DBE listings should be dropped. The result is a “master directory” of listed (or “known”) M/W/DBEs. The master directory is then merged with the correspon- ding Dun & Bradstreet MarketPlace database to enhance the identification of minority- and women-owned firms. How- ever, in order to maintain the “apples to apples” comparabil- ity between M/W/DBEs and non-M/W/DBEs, firms in the master directory for which there is no corresponding record in MarketPlace are not included in the analysis. Steps F & G. Verifying the Ownership Status of Firms in the Relevant Markets. If the listed DBEs158 identified above are, in fact, all DBEs and are the only DBEs among all the businesses identified, then the custom census estimate of DBE availability reduces simply to the number of listed DBEs divided by the total number of businesses in the relevant mar- ket. However, neither of these two conditions holds true in practice, and therefore this is not a complete measure of DBE availability. In order to derive a more accurate measure of availability, the possibility that some firms have been incorrectly classified as DBEs must be taken into account. This type of misclassifi- cation, if uncorrected, will lead to estimates of DBE availabil- ity that are upwardly biased. Similarly, the possibility that some firms not initially identified as DBEs are, in fact, DBEs must also be accounted for. This type of misclassification (which we refer to as “nonclassification” to avoid confusion), if uncorrected, will lead to estimates of DBE availability that are downwardly biased. In the custom census method, these two types of classifi- cation bias are corrected by a supplementary telephone sur- vey administered to a stratified random sample of firms in the baseline business population. These firms are contacted and asked directly about the race and gender of the firm’s primary owner(s). The results of the survey are then used to statistically adjust the estimates of DBE availability for mis- classification by race and gender. Tables 8 through 11 summarize the results from a misclas- sification survey we recently completed for a large municipal- ity in the southwest. Tables 8 and 9 show the fraction of firms originally classified as DBEs that were verified as such through the survey. Table 8 arrays the results by industry grouping and Table 9 by race and gender. Tables 10 and 11 show compara- ble results for the firms originally classified as non-DBE. As these four tables make clear, misclassification fractions are not insubstantial. Carrying Out Final Step 1 Availability Calculations. Once steps A through G are completed, final overall and detailed availability figures can be calculated. Below we define terms and provide the specific formula for estimating custom census avail- ability. We also provide a prose description of how the estimates are calculated in practice using the formula. 40 158As used here, “DBE” includes not only certified DBEs but also all other minority-owned and/or women-owned firms. See the discussion from the Northern Contracting case in Appendix C, Judicial Review of DBE Goal Setting Under Part 26.

41 Table 8. Listed DBE survey—amount of misclassification by NAICS code grouping. Table 9. Listed DBE survey—amount of misclassification by putative DBE type. Listed DBE By NAICS Code Grouping Misclassification (Percentage White Male) Number of Businesses Interviewed NAICS 236 24.0 104 NAICS 237 37.8 37 NAICS 238 20.2 252 NAICS 327, 332 25.0 12 NAICS 484 18.0 39 NAICS 42 31.8 129 NAICS 5413 19.5 174 Balance of NAICS Codes 12.2 557 All NAICS Codes 18.6 Percentage Actually M/WBE Owned 76.0 62.2 79.8 75.0 82.0 68.2 80.5 87.8 81.4 1,304 Note: NAICS 236—Building Construction, NAICS 237—Heavy Construction, NAICS 238—Special Trades Construction, NAICS 327—Nonmetallic Mineral Product Mfg., NAICS 332—Fabricated Metal Product Mfg., NAICS 484—Truck Transportation, NAICS 42—Wholesale Trade, NAICS 5413—Architecture, Engineering & Related Services. Putative Race/Gender Misclassi- fication (Percentage White Male) Misclassification (Percentage Other DBE Type) Percentage Correctly Classified Number of Businesses Interviewed African-American (either gender) 12.3 5.2 82.5 114 Hispanic (either gender) 15.5 4.4 80.1 401 Asian (either gender) 17.1 3.1 73.7 76 Native American (either gender) 47.6 26.2 26.2 42 White female 20.0 5.5 74.5 671 All DBE Types 18.6 N/A 81.4 1,304 Table 10. Nonclassified businesses survey— by NAICS code grouping. Listed DBE by SIC Code Grouping Percentage Actually White Male Owned Number of Businesses Interviewed NAICS 236 88.4 335 NAICS 237 88.6 140 NAICS 238 81.1 874 NAICS 327, 332 92.1 63 NAICS 484 67.6 145 NAICS 42 82.8 442 NAICS 5413 89.6 574 Balance of NAICS Codes 80.9 507 All NAICS Codes 83.6 Percentage DBE 11.6 11.4 18.9 7.9 32.4 17.2 10.4 19.1 16.4 3,080

42 Table 11. Nonclassified businesses survey—by race and gender. Verified Race/Gender Number of Businesses Interviewed Percentage of Total White male 2,575 83.6 White female 254 8.3 African-American 26 0.8 Hispanic 170 5.5 Asian 31 1.0 Native American 24 0.8 Total 3,080 100.0 All terms defined below are restricted to the relevant geo- graphic market. Individual industry categories are denoted by the sub- script i, which runs from 1 through n, where n is the total number of detailed industries included in the analyses, that is (i = 1, 2, . . . , n). In this particular example, n is 20. Di— Total number of “listed DBE” establishments in industry i. Ni— Total number of establishments in industry i. xi— Percentage of listed DBE establishments in industry i that aren’t actually minority or female owned—the “misclas- sification percentage.” yi— Percentage of establishments other than listed DBEs in industry i that aren’t actually majority male owned—the “nonclassification percentage.” 1-xi — Percentage of listed DBEs in industry i that are actu- ally minority or female owned. 1-yi— Percentage of establishments other than listed DBEs in industry i that are actually majority male owned. Dai— Total number of DBE establishments in industry i, adjusted for misclassification and nonclassification. Ai— Estimated percentage availability in industry i. wi— Percentage of total dollars spent in industry i, (“dollar- based industry weights”). Note: these weights sum to 1.0. A— Overall estimated percentage availability. The overall availability percentage, A, is then derived as follows:159 (F1) Total adjusted number of DBE establishments in industry i: D D x N D yai i i i i i= −( )+ −( ) ( ) 1 (F2) Availability percentage in industry i: (F3) Overall percentage availability: (F4) The full estimated availability formula is therefore: Below we provide a numerical representation of how this formula was used to derive the Step 1 DBE availability figure of 24.34% in a state DOT study we recently completed, along with a detailed explanation of the derivation. Table 12 pro- vides the actual numbers used for that study. Step 1 availability is calculated as a weighted average of the 20 individual estimated availability percentages (Ai). There is one individual availability estimate for each of the 20 SIC codes included in the calculation. These 20 individual availabil- ity percentages appear in column (9) of Table 12. For example, Ai for SIC code 1611 is 20.95%, for SIC code 8711 it is 24.51%, and so on. To derive the overall Step 1 availability estimate, A, of 24.34%, these 20 individual estimates must be averaged together, using Formula F3 above and the individual weights, wi, that appear in column (8) of Table 12. To calculate this weighted average, each individual availability estimate, Ai, from column (9) is multiplied by its corresponding weight, wi, from column (8). The result of this multiplication is shown in column (10). For SIC code 1611, for example, Ai equals 20.95%, wi equals 0.4293, and the product of the two is equal to 8.99, as shown in column (10). The sum of the 20 A D x N D y N i i i i i i = −( )+ −( ) ( )( )( )⎛⎝ ⎞⎠⎛⎝ ⎞⎠   1 100 wii=∑1 20 A A wi i i = ( ) = ∑  1 20 A D Ni ai i = ( )100 159Asterisk (*) indicates multiplication.

numbers in column (10) is 24.34%—the overall Step 1 DBE availability figure for the agency. The individual availability estimates, Ai, were calculated using formulas F1 and F2 above. Formula F2 says that each Ai is equal to the total number of listed DBE establishments in industry i, adjusted for misclassification and nonclassifi- cation (Dai), divided by the total number of establishments in industry i. The result is then multiplied by 100 to yield a percentage. Before we can carry out this calculation, however, we must calculate Dai. For that, we use formula F1 above, where Dai is equal to the number of “listed” DBE establishments in industry i, Di, multiplied by 1 minus the misclassification percentage, xi, plus the total number of remaining establish- ments in industry i, (Ni − Di), multiplied by the nonclassifi- cation percentage yi. That is, Dai = Di ∗ (1 − xi) + (Ni − Di) ∗ (yi). In SIC code 1611, for example, Di equals 78, 1 minus xi equals (1 − 0.238) = 0.762, Ni − Di equals 644 minus 78, or 566, and yi equals 0.13338. Therefore, Dai is (78 ∗ 0.762) + (566 ∗ 0.13338) which equals 134.93, as shown in column (7) of Table 12. Now that we know Dai is equal to 134.93 for SIC code 1611, we can calculate estimated availability for that industry, Ai, using formula (2) above. That is, we divide 134.93 by 644 and multiply the result by 100 to yield the estimated availability figure for SIC code 1611 of 20.95%, as shown in column (9) above. This process is repeated for each of the 20 SIC codes in the agency’s product market, which yields the 20 individual industry availability estimates, Ai, that appear in column (9) of Table 12. Finally, as explained above, each one of these Ai estimates is multiplied by its corresponding dollar-based industry weight, wi, to yield the Ai∗ wi figures in column (10). The 20 figures in column (10) are then added together to obtain the overall Step 1 DBE availability figure, A, of 24.34%. Below, we repeat formulas F1 through F4 and provide numerical tables below each formula showing the exact calcu- lations undertaken at each step for this particular example. (F1) Total adjusted number of DBE establishments in industry i: D D x N D yai i i i i i= −( )+ −( ) ( ) 1 43 Table 12. Numeric representation of Step 1 DBE availability calculation. (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) i SIC Code Di xi Ni yi Dai wi Ai Ai*wi 1 1611 78 0.238 644 0.13338 134.93 0.4293 20.95 8.99 2 8711 655 0.193 3582 0.11932 877.83 0.1243 24.51 3.05 3 1622 8 0.238 24 0.15458 8.57 0.0862 35.71 3.08 4 1541 33 0.253 307 0.12547 59.03 0.0736 19.23 1.42 5 5051 19 0.159 208 0.26484 66.03 0.0610 31.75 1.94 6 1731 331 0.231 3305 0.17129 763.94 0.0357 23.11 0.82 7 1771 131 0.231 1068 0.16293 253.40 0.0311 23.73 0.74 8 1791 24 0.231 114 0.18101 34.75 0.0239 30.48 0.73 9 1623 29 0.238 291 0.14372 59.75 0.0189 20.53 0.39 10 3273 9 0.159 70 0.26653 23.83 0.0179 34.04 0.61 11 1794 65 0.231 980 0.16959 205.16 0.0145 20.93 0.30 12 1799 278 0.231 3523 0.17113 769.08 0.0140 21.83 0.31 13 1721 330 0.231 2748 0.16933 663.21 0.0129 24.13 0.31 14 3569 2 0.159 29 0.26462 8.83 0.0123 30.44 0.37 15 782 202 0.159 2792 0.26543 857.35 0.0090 30.71 0.28 16 8748 2432 0.159 10449 0.27544 4253.52 0.0087 40.71 0.35 17 3531 2 0.159 45 0.25857 12.80 0.0076 28.45 0.22 18 1629 35 0.238 372 0.13153 70.99 0.0071 19.08 0.13 19 1741 87 0.231 1064 0.16877 231.79 0.0069 21.78 0.15 20 6512 90 0.159 1841 0.26450 538.82 0.0052 29.27 0.15 Overall Step 1 M/W/DBE Availability: 24.34 i SIC Code Dai = Di * (1–xi) + (Ni–Di) * yi 1 1611 134.93 = 78 * * * * * * * * * 0.762 + 566 * 0.13338 2 8711 877.83 = 655 * 0.807 + 2927 * *3 1622 8.57 = * 0.762 + 0.15458 4 1541 59.03 = 33 8 0.747 + 274 * 5 5051 66.03 = 19 0.841 0.841 0.841 0.841 0.841 0.841 + 189 * 6 1731 763.94 = 331 * + 2974 * 7 1771 253.40 = 131 * 0.769 0.769 + 937 * *8 1791 34.75 = 24 0.769 0.769 0.769 + 0.18101 9 1623 59.75 = 29 0.762 + 262 * *10 3273 23.83 = * + 16 90 61 0.26653 11 1794 205.16 = 65 9 2 2 0.769 + 915 * 12 1799 769.08 = 278 * + 3245 * 13 1721 663.21 = 330 * + 2418 * *14 3569 8.83 = * + 0.26462 15 782 857.35 = 202 * + 2590 * 16 8748 4253.52 = 2432 * + 8017 * *17 3531 12.80 = * + 27 43 0.25857 18 1629 70.99 = 0.762 + 337 * 19 1741 231.79 = 0.769 + 977 * 20 6512 538.82 = 35 87 90 0.841 + 1751 * 0.11932 0.12547 0.26484 0.17129 0.16293 0.14372 0.16959 0.17113 0.16933 0.26543 0.27544 0.13153 0.16877 0.26450

(F2) Availability percentage in industry i: A D Ni ai i = ( )100 Relationship Between Choice of Availability Measure and Resulting Annual DBE Goals The difference between the custom census method and other methods is evident when the annual DBE goals set by state DOTs during FFYs 2006, 2007, and 2008 are examined. Table 13 shows the overall DBE goals set by each state DOT during this period, as well as their race-conscious portions.160 Using the goal information in Table 13, we calculated some summary statistics to compare DBE goals derived from dis- parity or availability studies to those derived using bidders lists, prequalified contractor list, or other types of contractor lists or from DBE directories and Census Bureau data. The results of this analysis are shown in Table 14. Consistent with the concept of “casting a broader net,” it is clear from the top six rows of Table 14 that disparity or avail- ability studies tend to yield larger estimates of DBE availabil- ity than the other methods. Compared to the bidders list or prequalified contractors list approach, the disparity or avail- ability study approach yielded median overall DBE goals that were 3.8 percentage points higher on average, and mean over- all DBE goals that were 5.6 percentage points higher on aver- age. Compared to the method of DBE directories and Census Bureau data, the figures were 1.6 and 3.2 percentage points higher, respectively. The difference is even more pronounced when the race- conscious portion of the goal is examined. The disparity or availability study approach yielded median race-conscious DBE goals that were 4.5 percentage points higher and mean goals that were 5.8 percentage points higher, on average, compared to using bidders lists, prequalified contractor lists, or other contractor lists. Compared to using DBE directories and Census Bureau data, the race-conscious figures were 6.3 and 6.3 percentage points higher, respectively.161 It is possible that these differences do not arise because the availability estimation methods of disparity studies generally cast a broader net, but are instead due to larger M/WBE shares in those states that commissioned (and used) disparity or availability studies for their goal setting. To check for this possibility, we standardized the comparisons by dividing all the goals by the percentage of the construction business pop- ulation that was minority owned or women owned accord- ing to the 2002 SBO.162 The results appear in the last six rows 44 i SIC Code Ai = (Dai ÷ Ni) * 100 1 1611 20.95 = 134.93 ÷ 644 * 100 2 8711 24.51 = 877.83 ÷ 3582 * 100 3 1622 35.71 = 24 * 100 4 1541 19.23 = 59.03 ÷ ÷ 307 * 100 5 5051 31.75 = 66.03 ÷ 208 * 100 6 1731 23.11 = 763.94 ÷ 3305 * 100 7 1771 23.73 = 253.40 ÷ 1068 * 100 8 1791 30.48 = 34.75 ÷ 114 * 100 9 1623 20.53 = 59.75 ÷ 291 * 100 10 3273 34.04 = 23.83 ÷ 70 * 100 11 1794 20.93 = 205.16 ÷ 980 * 100 12 1799 21.83 = 769.08 ÷ 3523 * 100 13 1721 24.13 = 663.21 ÷ ÷ 2748 * 100 14 3569 30.44 = 8.57 8.83 29 * 100 15 782 30.71 = 857.35 ÷ 2792 * 100 16 8748 40.71 = 4253.52 ÷ 10449 * 100 17 3531 28.45 = 12.80 ÷ 45 * 100 18 1629 19.08 = 70.99 ÷ 372 * 100 19 1741 21.78 = 231.79 ÷ 1064 * 100 20 6512 29.27 = 538.82 ÷ 1841 * 100 (F3) Overall percentage availability for state DOT: A A wi i i = ( ) = ∑  1 20 i SIC Code Ai*wi = Ai * wi 1 1611 8.99 = 20.95 * 0.4293 2 8711 3.05 = 24.51 * 0.1243 3 1622 3.08 = 35.71 * 0.0862 4 1541 1.42 = 19.23 * 0.0736 5 5051 1.94 = 31.75 * 0.0610 6 1731 0.82 = 23.11 * 0.0357 7 1771 0.74 = 23.73 * 0.0311 8 1791 0.73 = 30.48 * 0.0239 9 1623 0.39 = 20.53 * 0.0189 10 3273 0.61 = 34.04 * 0.0179 11 1794 0.30 = 20.93 * 0.0145 12 1799 0.31 = 21.83 * 0.0140 13 1721 0.31 = 24.13 * 0.0129 14 3569 0.37 = 30.44 * 0.0123 15 782 0.28 = 30.71 * 0.0090 16 8748 0.35 = 40.71 * 0.0087 17 3531 0.22 = 28.45 * 0.0076 18 1629 0.13 = 19.08 * 0.0071 19 1741 0.15 = 21.78 * 0.0069 20 6512 0.15 = 29.27 * 0.0052 (F4) The full estimated availability formula is therefore, once again: A D x N D y N i i i i i i = −( )+ −( ) ( )( )( )⎛⎝ ⎞⎠⎛⎝ ⎞⎠   1 100 wii=∑1 20 160The race-neutral portion, of course, is obtained by subtracting the race- conscious portion from the overall goal. 161Similar differences are observed in both overall and race-conscious goals if state DOTs from the Ninth Circuit are excluded from the comparison. 162These percentages of minority-owned and women-owned businesses in each state were derived from U.S. Census Bureau (2006). The 2002 data are the most recent available.

45 State Method DBE Overall Goal 2006 DBE Race- Conscious Goal 2006 DBE Overall Goal 2007 DBE Race- Conscious Goal 2007 DBE Overall Goal 2008 DBE Race- Conscious Goal 2008 CA 3 10.50% 0.00% 10.50% 0.00% 13.50% 6.75% CO 3 12.19% 10.89% 13.76% 10.45% 12.80% 10.50% GA 3 12.00% 6.00% 12.00% 6.00% 12.00% 6.00% IL 3 22.77% 20.74% 22.77% 20.74% 22.77% 18.85% MD 3 16.10% 8.90% 24.34% 19.79% 25.30% 21.90% MN 3 9.57% 8.75% 6.27% 4.27% 15.30% 13.60% MO 3 13.50% 9.45% 13.37% 9.36% 13.34% 10.00% NC 3 10.40% 7.66% 9.90% 7.40% 10.10% 7.60% NV 3 3.30% 0.00% 3.00% 0.00% 5.70% 0.00% WA 3 12.70% 9.34% 18.77% 14.70% 18.77% 14.70% HI 1 11.00% 0.00% 9.00% 0.00% 7.50% 0.00% NJ 1 15.10% 14.10% 15.10% 14.50% 15.60% 0.38% NY 1 12.00% 9.20% 12.00% 9.50% 12.00% 5.00% AK 2 or 5 7.50% 3.50% 4.00% 0.00% 5.00% 0.00% AL 2 or 5 9.14% 4.32% 9.54% 5.34% * * AR 2 or 5 8.40% 6.40% 8.00% 6.10% 7.80% 5.00% AZ 2 or 5 10.50% 0.00% 9.67% 0.00% 9.91% 0.00% CT 2 or 5 15.50% 9.50% 13.40% 8.40% 13.60% 8.40% DC 2 or 5 32.84% 7.16% 32.70% 8.50% 26.89% 19.37% DE 2 or 5 11.02% 9.40% 10.01% 8.52% 11.73% 10.66% FL 2 or 5 7.87% 0.00% 8.12% 0.00% 8.07% 0.00% IA 2 or 5 4.80% 4.32% 4.50% 4.10% 4.60% 4.10% ID 2 or 5 11.00% 10.39% 11.00% 7.98% 11.00% 7.98% IN 2 or 5 9.85% 4.97% 8.80% 3.73% 9.90% 4.18% KS 2 or 5 9.19% 7.49% 10.02% 7.07% 10.19% 6.85% KY 2 or 5 7.00% 5.00% 7.00% 5.00% 7.00% 5.00% LA 2 or 5 10.00% 9.00% 10.00% 9.00% 10.50% 9.50% Table 13. FHWA annual DBE goals submitted by state DOTs, FFYs 2006–2008. (continued on next page)

46 * We were unable to obtain a copy of ALDOT’s FFY 2008 DBE goals. Notes: (1) Method 1 corresponds to 49 C. F. R. 26.45(c)(1) (DBE directories and Census Bureau data), method 2 or 5 corresponds to 49 C. F. R. 26.45(c)(2) or (c)(5) (bidders lists or other types of contractor lists), method 3 corresponds to 49 C. F. R. 26.45(c)(3) (disparity or availability study); (2) the race-neutral portion of the goal for each FFY is not presented due to space limitations. To calculate it, simply subtract the race-conscious goal from the corresponding overall goal; (3) goals were not necessarily approved as submitted to U.S.DOT; (4) Caltrans’ goals were set using method 1 in FFYs 2006 and 2007 and method 3 in FFY 2008; (5) Mn/DOT’s goals were set using method 3 in FFYs 2006 and 2008 and method 2 in FFY 2007; (6) NDOT’s goals were set using method 2 in FFYs 2006 and 2007 and method 3 in FFY 2008. MA 2 or 5 13.80% 8.00% 13.80% 9.00% 13.80% 10.00% ME 2 or 5 6.60% 1.60% 6.00% 1.00% 4.50% 1.00% MI 2 or 5 11.00% 8.50% 11.00% 8.50% 11.00% 8.50% MS 2 or 5 10.00% 4.00% 10.00% 5.00% 10.00% 4.00% MT 2 or 5 0.00% 0.00% 10.50% 0.00% 9.89% 0.00% ND 2 or 5 8.12% 3.25% 7.68% 3.88% 7.38% 3.76% NE 2 or 5 7.08% 4.32% 6.91% 4.44% 6.48% 4.53% NH 2 or 5 5.00% 1.00% 4.00% 1.00% 5.00% 1.00% NM 2 or 5 9.69% 0.00% 8.79% 0.00% 9.32% 0.00% OH 2 or 5 6.70% 5.60% 6.70% 5.90% 7.10% 5.80% OK 2 or 5 8.50% 6.50% 8.10% 5.10% 8.10% 4.52% OR 2 or 5 10.26% 5.89% 11.32% 0.00% 11.58% 0.00% PA 2 or 5 9.52% 6.88% 9.49% 5.37% 7.70% 4.52% RI 2 or 5 10.00% 7.00% 11.00% 6.20% 10.00% 7.04% SC 2 or 5 10.50% 7.50% 10.50% 7.50% 10.50% 7.50% SD 2 or 5 8.30% 3.59% 8.33% 3.64% 8.30% 3.94% TN 2 or 5 8.48% 6.96% 9.87% 6.63% 8.38% 5.04% TX 2 or 5 12.54% 6.00% 12.12% 6.00% 12.12% 6.00% UT 2 or 5 8.30% 4.40% 8.20% 4.00% 8.00% 3.80% VI 2 or 5 10.02% 10.02% 10.20% 10.20% 9.80% 9.50% VT 2 or 5 6.90% 0.00% 5.00% 0.00% 5.20% 0.00% WI 2 or 5 14.44% 11.20% 10.65% 7.15% 11.00% 8.80% WV 2 or 5 5.20% 4.00% 6.75% 5.46% 7.89% 5.15% WY 2 or 5 5.00% 0.00% 6.00% 0.00% 4.05% 0.00% State Method DBE Overall Goal 2006 DBE Race- Conscious Goal 2006 DBE Overall Goal 2007 DBE Race- Conscious Goal 2007 DBE Overall Goal 2008 DBE Race- Conscious Goal 2008 Table 13. (Continued).

47 Goal- Setting Method Summary Statistic Overall FFY DBE Goal Race-Conscious FFY DBE Goal 2006 2007 2008 2006 2007 2008 1 Median 11.5% 11.3% 12.0% 4.6% 4.8% 0.4% 2 or 5 Median 9.2% 9.5% 9.6% 5.3% 5.1% 4.8% 3 Median 12.4% 13.8% 13.4% 9.1% 10.5% 10.3% 1 Mean 12.2% 11.7% 11.7% 5.8% 6.0% 1.8% 2 or 5 Mean 9.6% 9.4% 9.4% 5.2% 4.7% 5.2% 3 Mean 13.7% 16.4% 15.0% 10.2% 12.6% 11.0% 1 Minimum 10.5% 9.0% 7.5% 0.0% 0.0% 0.0% 2 or 5 Minimum 3.3% 3.0% 4.1% 0.0% 0.0% 0.0% 3 Minimum 9.6% 9.9% 5.7% 6.0% 6.0% 0.0% 1 Maximum 15.1% 15.1% 15.6% 14.1% 14.5% 5.0% 2 or 5 Maximum 32.8% 32.7% 26.9% 11.2% 10.2% 19.4% 3 Maximum 22.8% 24.3% 25.3% 20.7% 20.7% 21.9% 1 No. Obs. 4 4 3 4 4 4 2 or 5 No. Obs. 38 39 36 38 39 36 3 1 2 or 5 3 1 2 or 5 3 No. Obs. 8 7 10 8 7 10 Note: Method 1 corresponds to 49 C. F. R. 26.45(c)(1) (used DBE directories and Census Bureau data); method 2 or 5 corresponds to 49 C. F. R. 26.45(c)(2) or (c)(5) (used bidders lists or other types of contractor lists); method 3 corresponds to 49 C. F. R. 26.45(c)(3) (used disparity or availability study). Std. Median 0.420 0.420 0.485 0.186 0.192 0.021 Std. Median 0.726 0.752 0.758 0.482 0.460 0.436 Std. Median 0.874 0.825 0.805 0.704 0.671 0.680 Std. Mean 0.465 0.457 0.492 0.289 0.298 0.074 Std. Mean 0.791 0.781 0.802 0.461 0.415 0.442 Std. Mean 0.881 0.978 0.922 0.680 0.751 0.698 Table 14. Summary statistics for DBE goals set by different methods.

of Table 14. As with the raw data, both the median and mean standardized DBE goals—overall and race-conscious—are larger among those state DOTs that used disparity or availabil- ity studies to help in the goal-setting process. It is also worth noting that for both FFY 2006 and FFY 2007, no state DOT that relied upon a disparity or availability study to assist in its DBE goal setting had an entirely race-neutral DBE goal.163 State DOT Utilization Analyses Recommended Approach. A high-quality utilization and disparity analysis requires a well-constructed project database detailing several years of past contract and subcon- tract activity. It is important that the database capture sub- contracting activity for both DBEs and non-DBEs.164 Utilization and disparity statistics are most informative when they are disaggregated along the following four key dimen- sions, including: • Race, ethnicity, and gender, as well as more aggregated groupings—minorities, white women, and all DBEs; • Contracts with DBE goals versus contracts without DBE goals;165 • Major procurement categories (i.e., construction and construction-related professional services); and • NAICS sectors (two-digit) and subsectors (three-digit), and industry groups (four-digit). Disaggregating by highway districts can also be useful, depending on the state DOT’s needs. All the disparity studies reviewed provided utilization statis- tics by race, ethnicity, and gender as well as for the aggregated groupings of MBE, WBE, and DBE.166 Most studies provided comparisons of federally assisted contracts to non-federally assisted contracts or otherwise attempted to provide compar- isons of DBE utilization on projects with goals to projects without goals. All of the studies examined utilization by major procurement categories. Most studies conducted separate utilization analyses for prime contracts versus subcontracts. In such cases, it is recom- mended that utilization also be calculated for prime contracts and subcontracts combined in order to provide a fuller picture of DBE participation relative to all contract and subcontract spending.167 Some studies did not have complete subcontractor data. The usefulness of their utilization (and disparity) statistics is correspondingly reduced. Few studies provided any utilization statistics by NAICS sector or subsector; omitting such detailed industry analy- sis is a practice that has been viewed negatively by some courts.168 State DOT Disparity Analyses Recommended Approach. A disparity analysis of state DOT spending is simply a comparison of DBE utilization to DBE availability. Therefore, the preceding discussions of mar- ket definition, availability measurement, and utilization sta- tistics are all relevant to the ability to produce a useful disparity analysis. The primary analysis should be conducted on proj- ects or business activities that were generally not subject to race-conscious contracting requirements. This means state DOT contracts without DBE goals, or the utilization of DBEs in the surrounding economy or, preferably, both. Proper test- ing for substantive and statistical significance must be per- formed in order to identify whether disparities are large and whether the observed disparities could have arisen due to ran- dom chance alone. Data Impacted by Race-Conscious Contracting Requirements As already explained, performing a disparity analysis on contracting and subcontracting dollars that were subject to race-conscious affirmative action requirements, as is often the case for state DOTs, is of limited value in strict scrutiny analysis. Much more informative is disparity analysis on con- tracting and subcontracting dollars that were not subject to affirmative action requirements, at the state DOT, economy- wide, or both. Some studies performed disparity analyses only on state DOT contracts and subcontracts that were subject to race- 48 163See line 9 in Table 14, above. This changed in FFY 2008 with the introduction of a completely race-neutral goal at NDOT subsequent to completion of its dis- parity study. 164See Step A—Create a Database of Representative, Recent, and Complete State DOT Projects, for an extended discussion of how to build such a database. Appen- dix A also provides an discussion regarding the collection of subcontract data. 165See Chapter Three, Economy-Wide Disparity Analysis for the Relevant Mar- kets, supra, for discussion of different methods for making this comparison. 166The one availability study that included a utilization analysis (NDOR, 2000) did not disaggregate by race, ethnicity, and gender. 167See 49 C.F.R. § 26.45(a) (goals are set on DOT-assisted contracts, not just the subcontract portion of DOT-assisted contracts); Northern Contracting, Inc. v. Illinois Department of Transportation, 2005 U.S. Dist. LEXIS 19868, *73 (Sept. 8, 2005) (Northern Contracting II) (“At no point do the Regulations limit the appli- cation of DBE goals to the subcontracting portion of contracts. To the contrary, the Regulations expressly provide that the goals requirements are imposed on prime contractors.”). 168See, e.g., fn. 191.

conscious contracting requirements.169 These studies included no comparison of DBE participation on projects with goals to DBE participation on projects without goals and also lacked any economy-wide analysis of business disparities. When studies such as these find “overutilization” in certain cate- gories, policy makers may be misled to conclude that there is an absence of discrimination. Several courts have recognized that this is incorrect. The plaintiffs in Concrete Works III argued that “overutilization” on projects subject to race-conscious contracting require- ments indicated an absence of discrimination. The court rejected this argument, concluding that the more pertinent inquiry should focus on M/WBE participation on projects without goals.170 Disparity analyses should strive to compare DBE partici- pation and DBE availability on projects that were not subject to race-conscious contracting requirements, either in the public sector, economy-wide, or both. This has been accom- plished in several ways, described below: • Comparing disparity ratios on contracts subject to goals to those on contracts not subject to goals. In states where there is no M/WBE program for state-funded projects, this can be accomplished by comparing disparities on federally assisted contracts with those on state-funded contracts. • Selecting certain contracts to be “control” or “zero goals” contracts and comparing disparity ratios on these con- tracts to those on contracts with goals. • Comparing disparity ratios on state DOT projects subject to goals to disparity ratios from other public entities in the relevant market that do not use goals. • Comparing disparity ratios on state DOT projects subject to goals to disparity ratios from private entities in the rele- vant market. Importance of Significance Testing Another important aspect to disparity analysis is significance testing. There are two dimensions to assessing significance. The first is “constitutional” or “substantive” significance.171 This type of significance addresses the size of a given disparity. The second is “statistical” significance. This type of significance addresses whether a given disparity could have arisen due to random chance alone. Both types of significance need to be considered in disparity analyses. For example, suppose DBE participation in a given category (not subject to race-conscious contracting requirements) was 13.9% and DBE availability in that same category was 14.0%. The disparity ratio would be 0.139 ÷ 0.140 = 0.99. While it is possible that this disparity might be statistically significant (if, e.g., a large number of contracts and subcontracts was involved), it is difficult to imagine such a small disparity being a source of concern to anyone. In contrast, sometimes sub- stantively significant disparities may not be statistically signif- icant. This often occurs when statistics are based on a relatively small sample of contracts. The Equal Employment Opportunity Commission has adopted a standard for substantive significance in the employ- ment discrimination setting. This so-called “80% rule” has been endorsed as a relevant way to gauge legally meaningful racial or gender disparities rather than simply testing for statis- tical significance alone. According to Meier, Sacks, and Zabell (1986, 32): Properly understood, the 80% rule has the potential to rational- ize much of the case law to date. . . . As a practical matter, even when statistically significant differences have been noted, the courts have been reluctant to find adverse impact when the dif- ferences lack what is variously described as “practical,” “substan- tive,” or “constitutional” significance. And conversely, substantial disparities have been found insufficient to establish a prima facie case when the sample sizes are so small as to make statistical sig- nificance unlikely. The 80% rule appears to be a reasonable articulation of a statistical criterion to determine whether statis- tically significant differences are substantial enough to warrant legal liability. The 80% rule states that for statistical disparities to be taken as legally dispositive in the discrimination context, they should be (a) statistically significant and (b) “substantively” significant. Substantive significance is taken to mean, for example, a DBE utilization measure that is less than or equal to 80% of the corresponding DBE availability measure. The distinction between these two types of “significance” has sometimes been a source of confusion to courts (Gastwirth 1988, 248). Among the studies we reviewed, four indicated whether dis- parity ratios were at or below the 80% threshold but did not perform any statistical significance testing on the disparity ratios. As Meier, Sacks, and Zabell (1986) note, a proper use of the 80% rule requires disparity ratios to be at or below 80% and also be statistically significant. 49 169CDOT (2001), GDOT (2005), OH (2001), NJ (2005). The OH (2001) study included some limited analysis of private sector disparities but still treated “overutilization” of M/WBEs by state agencies subject to affirmative action requirements as evidence of the absence of discrimination (“American Indian [firms at Ohio DOT] are overutilized at rates that are statistically significant. . . . Accordingly, race- and gender-conscious remedies are . . . [not] recommended . . . for American Indians.”). 170Concrete Works III, 321 F.3d at 984–85; see also Western States, 407 F.3d at 992; Northern Contracting II, 2004 U.S. Dist. Lexis at *37–38; Hershell Gill Consulting Engineers, Inc. v. Miami-Dade County, Florida, 333 F.Supp.2d 1305, 1318 (S.D. Fla. 2004) (“[The court] will keep the potential effect of the MWBE programs in mind when analyzing the evidence presented by the County.”). 171See also fns. 395 and 201 and the accompanying discussion in the text.

Three of the studies we reviewed employed the 80% rule and tested for statistical significance. Three more employed statis- tical significance testing but did not employ the 80% rule. However, in all six of these studies, the significance test used is only correct under the assumption that contracts and subcon- tracts are all the same size. If each contract and subcontract were the same size, the correct t-statistic would be: where u is the ratio of DBE contract and subcontract dollars to total contract and subcontract dollars, and a is the ratio of DBE firms to all firms. However, contract and subcontract values vary greatly in amount. Consequently, the correct t-statistic is:172 where ci is the dollar award or payment amount for contract or subcontract i.173 This statistic is employed to do disparity testing in only three of the studies we reviewed. Not all data are normally distributed, but many probabil- ity distributions are approximately normal and samples from many other distributions are approximately normal. The t- distribution is one such distribution. It is often employed in practice to model data based on the assumption that the sam- ple they are drawn from is approximately normal. In any particular application, the absolute value of the t- statistic resulting from equation (2) can be compared to a table of critical values for Student’s t distribution to deter- mine whether the result is statistically significant.174 In dis- crimination cases, the courts have usually required p-values of 5% or less to establish statistical significance in a two-sided case. The analogous p-value for the one-sided case is half of the two-sided p-value, or 2.5%. A two-sided p-value of 5% corresponds approximately to two standard deviations. The relevance of “two standard deviations” in statistics and the law is that it corresponds to a 95% confidence interval around a normal distribution. In the simplest terms, in normally distributed data, approximately 95% of the values lie within two standard deviations (a 95% confidence interval) above or below the mean, and 5% lie outside this range. Therefore, if we compare two numbers ( )2 1 2 2 t u a a a c c i i = − −( ) ( ) ∑ ∑   ( )1 1 t u a a a = − −( ) drawn from a normal probability distribution and their dif- ference is greater than or equal to two standard deviations, we can infer that the observed difference in the numbers is due to more than just random chance. If we were to draw these two numbers repeatedly, we would observe that this inference was correct, on average, only 19 out of every 20 draws. In other words, it would be correct 95% of the time and incor- rect 5% of the time. One other type of significance testing was observed in three of the studies we reviewed—Monte Carlo simulation studies. Starting from the project database of contracts and subcontracts, all with differing dollar sizes, these studies simulate the award process by programming a computer to randomly assign contract awards to the several types of DBEs as well as to non-DBEs, based on their estimated availability percentage. For example, if black-owned firms had esti- mated availability of 5%, then the computer would ran- domly pick 5% of the contracts and subcontracts and assign them to black-owned firms. The total value of the randomly assigned awards would then be totaled and compared to availability to assess whether there was a disparity between utilization and availability. The simulation exercise is then repeated a large number of times. If utilization fell below availability in 95% or more of the runs, that disparity is sta- tistically significant. Monte Carlo studies are typically used in cases where sam- ple sizes are small and the underlying distribution is not known. In large samples, however, sample mean statistics tend to converge toward a normal distribution, and the t-statistic provided in equation (2) can be used. In the three studies we reviewed, the Monte Carlo simulations were performed only on the full overall sample of contracts and subcontracts. In one study we reviewed, simulations were performed once for all federally funded construction and engineering contracts statewide over the entire study period and once for all state- funded construction and engineering contracts statewide over the entire study period. Both of these samples were quite large and the t-test from equation (2) could have been easily imple- mented instead. No explanation was provided of the choice to use Monte Carlo simulations. However, these three studies also presented dozens of sep- arate tables of disparity statistics disaggregated along several dimensions, for example, agency, funding source, major pro- curement category, contractor type, time period, and geo- graphic region. Several of these tables therefore had only a small number of contracts and subcontracts included. Dis- parity statistics were presented in each of these tables, but no significance testing was done. These tables would have been excellent candidates for Monte Carlo simulations that, as we have said, can be helpful for assessing the significance of dis- parity ratios in cases with small samples. No explanation was provided of the choice not to use Monte Carlo simulations. 50 172TX (1994, 88–90). 173In implementing this test statistic, prime contractor award or payment amounts must be a net of subcontract award or payment amounts. 174A table of critical values for the t-distribution can be found in any college-level statistics textbook.

Economy-Wide Disparity Analyses for the Relevant Markets Recommended Approach. Statistical analyses that assess how minorities and women fare in several key aspects of busi- ness enterprise activity should be conducted to determine whether a state DOT is passively participating in an industry sector tainted by discrimination. In Concrete Works III, the court discussed the importance of evidence of economy-wide (or “marketplace”) discrimination: In Adarand VII, we specifically concluded that evidence of mar- ketplace discrimination can be used to support a compelling interest in remedying past or present discrimination through the use of affirmative action legislation. . . . We clearly stated that evi- dence explaining “the Denver government’s role in contributing to the underutilization of MBEs and WBEs in the private construc- tion market in the Denver MSA” was relevant to Denver’s burden of producing strong evidence. . . . The City can demonstrate that it is a “passive participant in a system of racial exclusion practiced by elements of the local con- struction industry” by compiling evidence of marketplace dis- crimination and then linking its spending practices to the private discrimination. Therefore, evidence of marketplace discrimina- tion is not only relevant but, in this case, it is essential to the City’s claim that it is an indirect participant in private discrimination. Consequently, we again reject [plaintiff’s] argument and con- clude that the district court’s determination that the marketplace data was irrelevant was a legal error that significantly affected the court’s analysis of Denver’s evidence.175 A disparity studies should not ignore such evidence.176 Evidence of economy-wide discrimination in disparity and availability studies has taken several forms, including the following: • Regression analyses comparing business formation rates between minorities, women, and similarly situated non- minority males in the relevant markets. These have been implemented using the Census Bureau’s 5% Public Use Microdata Samples (PUMS) from the decennial census and/or the Current Population Surveys (CPS), produced jointly by the Census Bureau and the Bureau of Labor Statistics. • Regression analyses comparing the earnings of minority and female business owners to those of similarly situated nonminority male business owners in the relevant markets. These have also been implemented using the PUMS and/or the CPS. • Regression analyses comparing denial rates on commercial loans between minority, female, and similarly situated non- minority male business owners. These have been imple- mented using data from the Survey of Small Business Finances produced by the Federal Research Board and the Small Busi- ness Administration. • Disparity ratios comparing market share of revenues to market share of business population between minority, female, and nonminority businesses using data from the Census Bureau’s SBO. • Disparity ratios comparing minority and female utilization to availability using data on private sector construction projects from Reed Construction Data and/or F. W. Dodge and/or building permit databases. Regression analyses of business formation rates have also been used to quantify Step 2 adjustments, which must be con- sidered under 49 C.F.R. § 26.45(d). This is accomplished by first estimating a business formation regression model for non- minority males and then applying that model to the minority and female observations to estimate the business formation rate that would be expected if minorities and women faced the same market structure as nonminority males. Anecdotal Analyses Recommended Approach. Anecdotal evidence has been collected in a variety of formats including mail surveys, indi- vidual interviews, group interviews or focus groups, and pub- lic hearings. All of these approaches can and have produced qualitative evidence of barriers to full and fair participation by DBEs in the public contracting and subcontracting process. High-quality studies often employ multiple approaches to gathering this type of evidence, e.g., mail surveys and focus groups or personal interviews. Mail surveys are particularly important to establish a broad base of coverage that is capable of being quantified. Several studies used mail or telephone surveys, but most failed to test for nonresponse bias. Since response rates on voluntary sur- veys tend to be low (typically 5%–115%), it is important to test whether nonrespondents differ from respondents in ways that would alter the conclusions drawn from the survey. Failure to test for nonresponse bias may likewise undermine the persua- siveness of the results. Studies should gather evidence from non-DBEs as well as DBEs. It is critical to explore the extent to which barriers reported by anecdotal sources are the result of discrimination rather than the usual challenges facing all businesses related to developing markets, finding suppliers, managing cash flow, etc. This is also the state DOT’s opportunity to explore the operations of the DBE Program. This should include ques- tions regarding the use of race-conscious goals from the point of view of both the DBEs and non-DBEs. These include 51 175Id. at 976. 176See Adarand VII, 228 F.3d at 1167–68.

52 Table 15. Types of anecdotal evidence collected in state DOT disparity and availability studies. Study StudyType Indi- vidual Inter- views Focus Groups Public Hear- ings Mail Surveys Tele- phone Surveys Nonre- sponse Testing IDOT (2004) A Mn/DOT (2005) A MoDOT (2004) A NDOR (2000) A WSDOT (2006) A x CDOT (ongoing) D x x x x HDOT (ongoing) D x x MDT (ongoing) D x x x NCDOT (ongoing) D n/a n/a n/a n/a n/a n/a NY (ongoing) D x x ADOT (2009) D D x x x Alaska DOT&PF (2008) D x x x Caltrans (2007) x x ITD (2007) D x NDOT (2007) D x ODOT (2007) D x x TDOT (2007) D n/a n/a n/a n/a n/a n/a MD (2006) D x x GDOT (2005) D x x x NJ (2005) D x NCDOT (2004) D x x x VDOT (2004) D CDOT (2001) D OH (2001) D x FDOT (1999) D n/a n/a n/a n/a n/a n/a NMDOT (1995) D x SCDOT (1995) D x x x LA (1991) D x Note: (1) x indicates this type of anecdotal evidence was collected as part of the study; n/a indicates that we do not know whether this type of anecdotal evidence was collected as part of the study or not, because the study was unavailable to us; a blank indicates that this type of anecdotal evidence was not collected as part of the study; (2) anecdotal evidence is sometimes collected directly by the state DOT rather than the consultant and presented in its annual DBE goals submissions to the FHWA.

how goals are set; evaluating bidders’ DBE submissions; and monitoring compliance with DBE contractual commitments. Evidence gathering should also include the effectiveness of race-neutral measures such as unbundling contracts and setting aside contracts for bidding by small firms; bonding and financing support programs; certification outreach; and other supportive services. Special emphasis should be placed on the experiences of DBEs that desire to obtain prime contracts as a measure of continuing barriers to full participation in the marketplace. Careful consideration of race-neutral measures is necessary to provide support for the state DOT’s projection of the amount of the DBE goal it can meet solely through race- neutral measures. Studies should also have a wide enough variety of intervie- wees, survey participants, etc., to ensure representation of all racial and ethnic minorities, white women and white men, and all major procurement categories. All disparity studies provided some anecdotal evidence; one availability study was supplemented with anecdotal evi- dence through interviews with business owners.177 The evi- dence collected ranged from individual interviews, to group interviews, to public hearings, to large scale quantitative sur- veys. As Table 15 shows, the most commonly used technique was individual interviews. While the interviews generally elicited useful information, some studies interviewed only DBEs, or only a very small num- ber of non-DBEs. This is a serious deficiency, as it is important to tease out the effects of discrimination from the general bar- riers faced by all small and new firms. This lack of balance may undermine the results, since the race-conscious elements must be narrowly tailored to only the categories of identified victims of discrimination and the impact of race-conscious programs on third-parties such as non-DBE subcontractors must be considered. 53 177This anecdotal evidence was gathered after the completion of the study in light of Western States.

Next: Chapter 4 - Study Resource Issues »
Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

TRB’s National Cooperative Highway Research Program (NCHRP) Report 644: Guidelines for Conducting a Disparity and Availability Study for the Federal DBE Program explores guidelines for state departments of transportation (DOTs) on how to conduct effective and legally defensible disparity and availability studies to meet the requirements of the Disadvantaged Business Enterprise (DBE) program for federally funded projects. The report includes guidance designed to assist DOTs in determining when and if a disparity or availability study is recommended, a model scope of work that may be used in a request for proposals, and detailed recommendations on how to design and implement disparity and availability studies.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!