Cover Image

Not for Sale

View/Hide Left Panel
Click for next page ( 49

The National Academies of Sciences, Engineering, and Medicine
500 Fifth St. N.W. | Washington, D.C. 20001

Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 48
48 of Table 14. As with the raw data, both the median and mean mended that utilization also be calculated for prime contracts standardized DBE goals--overall and race-conscious--are and subcontracts combined in order to provide a fuller picture larger among those state DOTs that used disparity or availabil- of DBE participation relative to all contract and subcontract ity studies to help in the goal-setting process. spending.167 It is also worth noting that for both FFY 2006 and FFY 2007, Some studies did not have complete subcontractor data. no state DOT that relied upon a disparity or availability study The usefulness of their utilization (and disparity) statistics is to assist in its DBE goal setting had an entirely race-neutral correspondingly reduced. DBE goal.163 Few studies provided any utilization statistics by NAICS sector or subsector; omitting such detailed industry analy- sis is a practice that has been viewed negatively by some State DOT Utilization Analyses courts.168 Recommended Approach. A high-quality utilization and disparity analysis requires a well-constructed project State DOT Disparity Analyses database detailing several years of past contract and subcon- tract activity. It is important that the database capture sub- Recommended Approach. A disparity analysis of state contracting activity for both DBEs and non-DBEs.164 DOT spending is simply a comparison of DBE utilization to Utilization and disparity statistics are most informative when DBE availability. Therefore, the preceding discussions of mar- they are disaggregated along the following four key dimen- ket definition, availability measurement, and utilization sta- sions, including: tistics are all relevant to the ability to produce a useful disparity analysis. The primary analysis should be conducted on proj- Race, ethnicity, and gender, as well as more aggregated ects or business activities that were generally not subject to groupings--minorities, white women, and all DBEs; race-conscious contracting requirements. This means state Contracts with DBE goals versus contracts without DBE DOT contracts without DBE goals, or the utilization of DBEs goals;165 in the surrounding economy or, preferably, both. Proper test- Major procurement categories (i.e., construction and ing for substantive and statistical significance must be per- construction-related professional services); and formed in order to identify whether disparities are large and NAICS sectors (two-digit) and subsectors (three-digit), whether the observed disparities could have arisen due to ran- and industry groups (four-digit). dom chance alone. Disaggregating by highway districts can also be useful, Data Impacted by Race-Conscious depending on the state DOT's needs. Contracting Requirements All the disparity studies reviewed provided utilization statis- tics by race, ethnicity, and gender as well as for the aggregated As already explained, performing a disparity analysis on groupings of MBE, WBE, and DBE.166 Most studies provided contracting and subcontracting dollars that were subject to comparisons of federally assisted contracts to non-federally race-conscious affirmative action requirements, as is often assisted contracts or otherwise attempted to provide compar- the case for state DOTs, is of limited value in strict scrutiny isons of DBE utilization on projects with goals to projects analysis. Much more informative is disparity analysis on con- without goals. All of the studies examined utilization by major tracting and subcontracting dollars that were not subject to procurement categories. affirmative action requirements, at the state DOT, economy- Most studies conducted separate utilization analyses for wide, or both. prime contracts versus subcontracts. In such cases, it is recom- Some studies performed disparity analyses only on state DOT contracts and subcontracts that were subject to race- 163See line 9 in Table 14, above. This changed in FFY 2008 with the introduction of a completely race-neutral goal at NDOT subsequent to completion of its dis- parity study. 167See 49 C.F.R. 26.45(a) (goals are set on DOT-assisted contracts, not just the 164See Step A--Create a Database of Representative, Recent, and Complete State subcontract portion of DOT-assisted contracts); Northern Contracting, Inc. v. DOT Projects, for an extended discussion of how to build such a database. Appen- Illinois Department of Transportation, 2005 U.S. Dist. LEXIS 19868, *73 (Sept. 8, dix A also provides an discussion regarding the collection of subcontract data. 2005) (Northern Contracting II) ("At no point do the Regulations limit the appli- 165See Chapter Three, Economy-Wide Disparity Analysis for the Relevant Mar- cation of DBE goals to the subcontracting portion of contracts. To the contrary, kets, supra, for discussion of different methods for making this comparison. the Regulations expressly provide that the goals requirements are imposed on 166The one availability study that included a utilization analysis (NDOR, 2000) prime contractors."). did not disaggregate by race, ethnicity, and gender. 168See, e.g., fn. 191.

OCR for page 48
49 conscious contracting requirements.169 These studies included type of significance addresses the size of a given disparity. no comparison of DBE participation on projects with goals to The second is "statistical" significance. This type of significance DBE participation on projects without goals and also lacked addresses whether a given disparity could have arisen due to any economy-wide analysis of business disparities. When random chance alone. Both types of significance need to be studies such as these find "overutilization" in certain cate- considered in disparity analyses. gories, policy makers may be misled to conclude that there is For example, suppose DBE participation in a given category an absence of discrimination. (not subject to race-conscious contracting requirements) was Several courts have recognized that this is incorrect. The 13.9% and DBE availability in that same category was 14.0%. plaintiffs in Concrete Works III argued that "overutilization" The disparity ratio would be 0.139 0.140 = 0.99. While it is on projects subject to race-conscious contracting require- possible that this disparity might be statistically significant ments indicated an absence of discrimination. The court (if, e.g., a large number of contracts and subcontracts was rejected this argument, concluding that the more pertinent involved), it is difficult to imagine such a small disparity being inquiry should focus on M/WBE participation on projects a source of concern to anyone. In contrast, sometimes sub- without goals.170 stantively significant disparities may not be statistically signif- Disparity analyses should strive to compare DBE partici- icant. This often occurs when statistics are based on a relatively pation and DBE availability on projects that were not subject small sample of contracts. to race-conscious contracting requirements, either in the The Equal Employment Opportunity Commission has public sector, economy-wide, or both. This has been accom- adopted a standard for substantive significance in the employ- plished in several ways, described below: ment discrimination setting. This so-called "80% rule" has Comparing disparity ratios on contracts subject to goals to been endorsed as a relevant way to gauge legally meaningful those on contracts not subject to goals. In states where racial or gender disparities rather than simply testing for statis- there is no M/WBE program for state-funded projects, this tical significance alone. According to Meier, Sacks, and Zabell can be accomplished by comparing disparities on federally (1986, 32): assisted contracts with those on state-funded contracts. Selecting certain contracts to be "control" or "zero goals" Properly understood, the 80% rule has the potential to rational- contracts and comparing disparity ratios on these con- ize much of the case law to date. . . . As a practical matter, even when statistically significant differences have been noted, the tracts to those on contracts with goals. courts have been reluctant to find adverse impact when the dif- Comparing disparity ratios on state DOT projects subject ferences lack what is variously described as "practical," "substan- to goals to disparity ratios from other public entities in the tive," or "constitutional" significance. And conversely, substantial relevant market that do not use goals. disparities have been found insufficient to establish a prima facie Comparing disparity ratios on state DOT projects subject case when the sample sizes are so small as to make statistical sig- to goals to disparity ratios from private entities in the rele- nificance unlikely. The 80% rule appears to be a reasonable articulation of a statistical criterion to determine whether statis- vant market. tically significant differences are substantial enough to warrant legal liability. Importance of Significance Testing Another important aspect to disparity analysis is significance The 80% rule states that for statistical disparities to be testing. There are two dimensions to assessing significance. The taken as legally dispositive in the discrimination context, they first is "constitutional" or "substantive" significance.171 This should be (a) statistically significant and (b) "substantively" significant. Substantive significance is taken to mean, for example, a DBE utilization measure that is less than or equal 169 CDOT (2001), GDOT (2005), OH (2001), NJ (2005). The OH (2001) study to 80% of the corresponding DBE availability measure. The included some limited analysis of private sector disparities but still treated "overutilization" of M/WBEs by state agencies subject to affirmative action distinction between these two types of "significance" has requirements as evidence of the absence of discrimination ("American Indian sometimes been a source of confusion to courts (Gastwirth [firms at Ohio DOT] are overutilized at rates that are statistically significant. . . . 1988, 248). Accordingly, race- and gender-conscious remedies are . . . [not] recommended . . . Among the studies we reviewed, four indicated whether dis- for American Indians."). 170 Concrete Works III, 321 F.3d at 98485; see also Western States, 407 F.3d at 992; parity ratios were at or below the 80% threshold but did not Northern Contracting II, 2004 U.S. Dist. Lexis at *3738; Hershell Gill Consulting perform any statistical significance testing on the disparity Engineers, Inc. v. Miami-Dade County, Florida, 333 F.Supp.2d 1305, 1318 (S.D. ratios. As Meier, Sacks, and Zabell (1986) note, a proper use Fla. 2004) ("[The court] will keep the potential effect of the MWBE programs in mind when analyzing the evidence presented by the County."). of the 80% rule requires disparity ratios to be at or below 80% 171 See also fns. 395 and 201 and the accompanying discussion in the text. and also be statistically significant.

OCR for page 48
50 Three of the studies we reviewed employed the 80% rule and drawn from a normal probability distribution and their dif- tested for statistical significance. Three more employed statis- ference is greater than or equal to two standard deviations, we tical significance testing but did not employ the 80% rule. can infer that the observed difference in the numbers is due However, in all six of these studies, the significance test used is to more than just random chance. If we were to draw these only correct under the assumption that contracts and subcon- two numbers repeatedly, we would observe that this inference tracts are all the same size. If each contract and subcontract was correct, on average, only 19 out of every 20 draws. In were the same size, the correct t-statistic would be: other words, it would be correct 95% of the time and incor- u-a rect 5% of the time. (1) t= One other type of significance testing was observed in a (1 - a ) three of the studies we reviewed--Monte Carlo simulation studies. Starting from the project database of contracts and where u is the ratio of DBE contract and subcontract dollars subcontracts, all with differing dollar sizes, these studies to total contract and subcontract dollars, and a is the ratio of simulate the award process by programming a computer to DBE firms to all firms. randomly assign contract awards to the several types of DBEs However, contract and subcontract values vary greatly in as well as to non-DBEs, based on their estimated availability amount. Consequently, the correct t-statistic is:172 percentage. For example, if black-owned firms had esti- u-a mated availability of 5%, then the computer would ran- (2) t= a (1 - a ) ci2 domly pick 5% of the contracts and subcontracts and assign ( c i )2 them to black-owned firms. The total value of the randomly assigned awards would then be totaled and compared to where ci is the dollar award or payment amount for contract availability to assess whether there was a disparity between or subcontract i.173 This statistic is employed to do disparity utilization and availability. The simulation exercise is then testing in only three of the studies we reviewed. repeated a large number of times. If utilization fell below Not all data are normally distributed, but many probabil- availability in 95% or more of the runs, that disparity is sta- ity distributions are approximately normal and samples from tistically significant. many other distributions are approximately normal. The t- Monte Carlo studies are typically used in cases where sam- distribution is one such distribution. It is often employed in ple sizes are small and the underlying distribution is not practice to model data based on the assumption that the sam- known. In large samples, however, sample mean statistics tend ple they are drawn from is approximately normal. to converge toward a normal distribution, and the t-statistic In any particular application, the absolute value of the t- provided in equation (2) can be used. In the three studies we statistic resulting from equation (2) can be compared to a reviewed, the Monte Carlo simulations were performed only table of critical values for Student's t distribution to deter- on the full overall sample of contracts and subcontracts. In mine whether the result is statistically significant.174 In dis- one study we reviewed, simulations were performed once for crimination cases, the courts have usually required p-values all federally funded construction and engineering contracts of 5% or less to establish statistical significance in a two-sided statewide over the entire study period and once for all state- case. The analogous p-value for the one-sided case is half of funded construction and engineering contracts statewide over the two-sided p-value, or 2.5%. the entire study period. Both of these samples were quite large A two-sided p-value of 5% corresponds approximately to and the t-test from equation (2) could have been easily imple- two standard deviations. The relevance of "two standard mented instead. No explanation was provided of the choice to deviations" in statistics and the law is that it corresponds to a use Monte Carlo simulations. 95% confidence interval around a normal distribution. In the However, these three studies also presented dozens of sep- simplest terms, in normally distributed data, approximately arate tables of disparity statistics disaggregated along several 95% of the values lie within two standard deviations (a 95% dimensions, for example, agency, funding source, major pro- confidence interval) above or below the mean, and 5% lie curement category, contractor type, time period, and geo- outside this range. Therefore, if we compare two numbers graphic region. Several of these tables therefore had only a small number of contracts and subcontracts included. Dis- parity statistics were presented in each of these tables, but no significance testing was done. These tables would have been 172TX (1994, 8890). excellent candidates for Monte Carlo simulations that, as we 173In implementing this test statistic, prime contractor award or payment have said, can be helpful for assessing the significance of dis- amounts must be a net of subcontract award or payment amounts. 174A table of critical values for the t-distribution can be found in any college-level parity ratios in cases with small samples. No explanation was statistics textbook. provided of the choice not to use Monte Carlo simulations.