Analyzing Information on Women-Owned Small Businesses in Federal Contracting (2005)

Disparities are often taken to indicate the presence of discrimination—for example, that observed differences in earnings between women and men must be due in part to discrimination against women by employers. However, measured disparities may be due to any number of factors, and they need not imply discrimination. This chapter briefly reviews methodological issues involved in explaining statistically measured disparities that are found in the use of women-owned small businesses in federal contracting. Research on factors that result in barriers for women-owned small businesses, including possible discriminatory practices or behaviors in the contracting process or discrimination in other domains, such as bank lending, seems useful to include in the longer term research agenda that we recommend that the SBA develop in this area (see Chapter 6).

Determining the factors that explain observed disparities is a difficult task. Determining the extent to which discriminatory practices or behaviors are among the causal factors is a particular challenge. A recent panel of the Committee on National Statistics conducted an extensive review of data and methods for measuring racial discrimination, which are also relevant for measuring gender discrimination. Much of the discussion below of concepts and estimation methods draws on that panel report (National Research Council, 2004).

The National Research Council (2004:Ch.3) used a social science definition of racial discrimination that can be translated into a similar definition of discrimination against women (or women-owned small businesses).¹ This social science definition has two components: (1) differential treatment on the basis of gender that disadvantages women and (2) treatment on the basis of inadequately justified factors other than gender that disadvantages women (differential effect). The first component, which constitutes intentional discrimination, is frequently unlawful under either the U.S. Constitution or specific legislation under the “disparate treatment” legal standard. The second component includes instances in which treatment based on inadequately justified factors other than gender results in adverse consequences for women, such as a promotion practice that generates differential effects.² A process with adverse consequences for women may or may not be considered discrimination under the law under the “disparate impact” legal standard, depending on whether there is a sufficiently compelling reason for its use and whether there are alternative processes that would not produce gender disparities.³ In the areas in which this type of discrimination is unlawful, the reason is to curtail the use of unintentional practices that can harm women, as well as to sanction intentional discrimination that might not be identified because of the difficulty in establishing intent in the legal setting.⁴

The social science definition of discrimination is broader than the legal standards of disparate treatment and disparate impact because it includes processes and behaviors that warrant attention by policy makers, even though they would not meet a legal standard of proof. For example, subtle—but not necessarily illegal—forms of discrimination in the contracting process could adversely affect the probabilities of women-owned small businesses obtaining federal contracts on which they bid. Also, overt or subtle

discrimination could occur outside the contracting process (for example, lending practices that gave women-owned businesses less access to venture capital), which could discourage women from bidding on federal contracts. The result could be underrepresentation of women-owned small businesses, even if the contracting process were totally neutral among bidders.

Because discriminatory behavior can rarely be directly observed, researchers face the challenge of determining when discrimination against women-owned small businesses has actually occurred and whether it explains some portion of an adverse outcome for these businesses. To measure discrimination, researchers must answer the counterfactual question: What would have happened to a woman-owned small business if the business had not been woman-owned? Answering this question is fundamental to being able to conclude that there is a causal relationship between the gender of the business owner and discrimination, which, in turn, is necessary to conclude that gender-based discriminatory behaviors or processes contributed to an observed differential outcome for women-owned small businesses compared with other businesses.

By definition, literally answering the counterfactual question is impossible, because one cannot clone the contracting situation, substitute an owner identical in all respects except for gender, and rerun the decision process. It is scarcely even possible to conduct a carefully controlled laboratory or field experiment in which the only variable that is manipulated by the researcher is the gender of the business owner.

Field experiments, called audit studies, have been successfully used to examine race-based discrimination in housing markets by real estate firms. In these studies, prospective renters or buyers that have similar characteristics except for their race or ethnicity visit real estate offices to see how many referrals they are given and in what locations (see, e.g., National Research Council, 2002; Turner et al., 2002, 2003; Yinger, 1986, 1993, 1995). Similar kinds of audit studies have been conducted of decisions by employers to interview job seekers who are matched in terms of job readiness but differ in terms of race (see, e.g., Turner, Fix, and Struyk, 1991).

It appears to be more difficult, however, to conduct such experiments for federal contracting. It would be fraudulent for independent researchers to enlist firms to misrepresent their ownership status in bidding, using random assignment by the experimenter. If the federal government were to decide to commission such an experiment, it would still be difficult to carry out given all of the steps that bidders must go through in the registration and bidding process and the detailed information about their experience and credentials that they must provide.

Thus, in practical terms, the question becomes what are acceptable statistical methods to move from observing a statistical disparity to concluding the existence of discrimination. Yet as one moves from meticulously designed and executed laboratory experiments through the variety of studies based on observational data, increasingly strong assumptions are needed to support the claim that X “causes” Y.⁵

There are many aspects of potential discrimination in the federal contracting process and in the processes that produce ready, willing, and able bidders that could be studied by the SBA and others with regard to the experience of women-owned small businesses. Because federal contracting is the domain of most direct interest for the federal government, we primarily discuss the uses and limitations of statistical estimation methods for studying one or more aspects of the contracting process. That process includes not only the decision to accept or reject a bid, but also such decisions as whether to bundle agency requirements into fewer, larger procurements or more and smaller procurements, whether to add more work to existing contracts or let new contracts, what criteria to use and what weights to give to different criteria in making a contract award, and allocations of time and resources to reach out to various types of businesses with various methods to encourage them to register and become capable of federal contracting work. Decisions in each of these areas may affect the opportunities for women-owned small businesses in federal contracting. For example, the use of one or more award criteria that are not good predictors of successful performance could have an adverse impact on businesses that are less qualified on these criteria. The methods we discuss for analyses of one or more aspects of the contracting process include multivariate regression models (both statistical decomposition models and theory-based models), matching and propensity score methods, and natural experiments. Such methods could also be used for studies that peer further back into the causal chain by which pools of ready, willing, and able bidders are developed—for example, studies of sources of venture and working capital for new and continuing businesses, various forms of technical assistance and mentoring, and the like.

It is quite common for researchers to employ statistical regression models when addressing questions of discrimination, such as racial or gender discrimination in hiring and promotions. Two types of models have been used to decompose racial or gender differences in outcomes. They are (1) regression models with gender-specific intercepts, which assume that the effects of other variables (e.g., education) are the same for both men and women, and (2) gender-specific regression models that relax this assumption by allowing for interaction between gender and other variables. All such models pose problems for interpretation and are basically descriptive rather than causal.

A standard way to explore the difference in an outcome between groups is to decompose the difference into “explained” and “unexplained” components. A primary concern of the SBA is the success in the federal contracting process of women-owned small businesses that are registered with the Central Contractor Registration (CCR) versus other registered businesses. The simplest formulation to compare outcomes for these two groups would be a regression model in which the dependent variable, Y, is the outcome of interest for each business, such as the amount of contract dollars won as reported in the Federal Procurement Data System (FPDS). The right side of the equation would include an intercept term, B; an indicator variable, W (1 for women-owned small businesses and 0 for others); a set of other variables, X₁ … X_j, that are believed to relate to the outcome; and an error term, u. Similar to the regression equations developed by the Department of Commerce (but for a different purpose), the X variables might include the age of the firm since founding and a measure of firm size (number of employees or revenues), although there are likely to be many more explanatory variables than these two. Alternatively, separate models could be developed for each group, or, equivalently, a model with interactions, to allow for the possibility that the relationships of each group to one or more of the X variables might differ.

In such a model (see National Research Council, 2004:121-125, for precise mathematical formulations), the coefficients on the X variables are used to estimate the contribution of differences in these variables to measured disparities. Thus, if registered women-owned small businesses, on average, obtain a smaller share of federal contract dollars and if they are, on average, newer and smaller than other businesses, then the age and size variables will probably explain some of the disparity.

The part of the gap that is “unexplained” by the X variables is sometimes referred to as the “share due to discrimination.”⁶ This is misleading terminology, however, because if any important control variables are omitted from the X set, then one or more of the equation coefficients, including the intercept, may be affected. More properly, the unexplained part of the gap represents not only the effects of discrimination in the contracting process, but also any unobserved differences between women-owned small businesses and other businesses in factors that would be expected to determine Y in the absence of discrimination. The unobserved variables may result in the equation underestimating or overestimating the effects of discrimination depending on how they correlate with variables in X or whether they mostly favor or mostly do not favor women-owned small businesses vis-à-vis other businesses. The inclusion in X of variables that are themselves an outcome in a particular domain (e.g., occupation or position within a firm in a study of earnings differences, or status as an economically disadvantaged firm in a study of contracting outcomes by gender of ownership) may also result in the equation misestimating the effect of discrimination.

In addition, it is misleading not to recognize that discrimination may affect the “explained” component of the equation (the X variables) and not just the unexplained component. For example, discrimination in lending practices may possibly result in women-owned businesses remaining smaller than others and therefore less likely to win contracts, regardless of whether discriminatory barriers exist in the contracting process itself.

Finally, it is important to keep in mind that the quality of the input data affects the validity and interpretability of regression model results (see “Data Quality” section below). Poorly measured variables in a multiple regression equation will result in biased and inconsistent regression coefficient estimates. Moreover, when the model and the input data are problematic, caution must be used in interpreting the results with standard tests for statistical significance.

Statistical decomposition of the factors affecting an outcome of interest, such as disparities between women-owned and other small businesses in dollars of federal contract awards, is a useful descriptive tool, providing such decomposition is carefully performed. More powerful, although very

difficult to implement, are statistical models that are informed by a theory of the discriminatory processes that may be at work and that address two important sources of bias—omitted variables bias and sample selection bias.

For statistical modeling (see National Research Council, 2004:Ch.7), the researcher needs to develop a theory of how discriminatory processes may operate in the domain of interest and formalize assumptions and conditions under which counterfactual logic can be applied. More specifically, for developing a set of X variables, the researcher would need to have a good understanding of the process that would determine Y in both the absence and presence of discrimination in order to be able to make causal assertions. The National Research Council report (2004:130-145) works through a detailed example of a theory-based regression model of discrimination in the labor market based on determining how a rational firm would make hiring decisions to improve productivity. The example also discusses methods to address omitted variables and sample selection bias.

For a theory-based model of success in federal contracting, measured as dollars awarded, one would need a detailed understanding of the factors that contracting officials take into account in making accept or reject decisions. Interviews with contracting officials on important determinants of contracting success together with analysis of the types of firms that tend to win competitive contracts with specified features could suggest the characteristics of registered vendors for which data are needed as input to the model. Such characteristics might include not only standard measures of firm age and size, but also measures of facilities, equipment, geographic location, previous bids, previous successful bids and add-ons to existing contracts, previous business experience of key personnel, whether the firm qualifies as economically disadvantaged, and the like. To be able to reliably infer discrimination from the results of the analysis, it is very important that the equation include all of the relevant variables that are known to the contracting officials in charge of decisions and that it not include variables that may be known to the researcher but not to the contracting officials.⁷ More generally, to develop an appropriate statistical model with a good fit to the data, the researcher should be prepared to conduct considerable exploratory work with test data sets to assess alternative forms of the equation and the input variables.

Because the target group of interest is registered women-owned small businesses, some restrictions might be placed on the outcome variable in a

theory-based model, such as defining it as dollars awarded in contracts below a threshold size. Alternatively, the comparison group might be defined as other registered small businesses and not all other registered businesses regardless of size. Moreover, instead of registered businesses, it might be preferable, first, to define the target group as bidders on procurements. If, controlling for other factors, women-owned small businesses are successful in competing for contracts on which they bid and could be reasonably expected to win (e.g., the contracts are not too big), then analytical interest might shift to prior steps in the contracting process, such as decisions on outreach to businesses by type to encourage them to bid or decisions on how to bundle agency requirements into contracts.

Because contracting practices may vary by agency and among agencies or regions within a large cabinet department (see Chapter 3 on contracting practices in the Defense Department), an appropriate analysis would need to look at contracting agencies separately or include variables for them in an equation. In-depth case studies of contracting practices for specific agencies and offices that vary in the percentage of contracts with specified characteristics that are awarded to women-owned small businesses could help generate hypotheses about causal processes and what kinds of X variables to include in models. To recognize the possible effects of supply-side variation, it would be also necessary for the case studies to take account of differences among industries or product lines in the availability of women-owned small businesses and all others.

Matching methods provide an alternative to multivariate linear regression as a way to control for variables that are likely to matter for an outcome. In this case, matching consists of comparing outcomes of two paired firms (or, more generally, two paired groups of firms) that are comparable on relevant observed attributes except for ownership. Matching of observational data attempts to mimic the experimental setting in the same way as the paired testing that is used in audit studies of housing markets. To the extent that (1) the observed factors capture the relevant variables affecting the outcome and (2) the comparability is close, differences in the outcome variable on the basis of ownership in a matching study can be attributed to discrimination. (However, the same caveats expressed above about making causal inferences from fitting multiple regression models to observational data apply to the use of matching methods with observational data.) Matching has been the subject of considerable research, and relatively sophisticated matching methods, such as propensity score matching, have been developed.

The objective in matching for the federal contracting process would be to construct matched sets or strata using relevant nonownership variables that are available for firms. Analogous to overfitting in specifying a multiple regression, the analyst doing the matching must make the trade-off between matching on too few variables, with the result of poor comparability within matched sets, and matching on too many variables, with the result of poor statistical power and problems with interpretation. A common way to manage this trade-off is to combine matching on a small number of variables that are known to have large effects together with matching on propensity scores estimated from a larger set of additional variables thought to be relevant.

The propensity score is a device for constructing matched sets when there are too many covariates so that it becomes increasingly difficult to find matched pairs with similar values of all of the covariates. The propensity score would be estimated by fitting a logistic regression to women-owned small businesses versus other businesses (or other small businesses) using the covariates as the explanatory variables. Firms with similar propensity scores are grouped into the same strata to create matched sets.

In comparison to multiple regression, matching methods reduce the risk of imposing an inappropriate functional form on the relationship between the outcome variable and the observed covariates. A drawback, however, is that a matched analysis does not use the entire pool of observations; rather, in the contracting case, each woman-owned small business would typically be matched to one nonwoman-owned small business and the unmatched businesses would be discarded. When the number of women-owned small businesses to be matched is small, the size of this sample drives the accuracy of the estimated difference. In this situation, the incremental loss of precision from discarding the nonmatched members of the other business group is low.

Choosing between matching and regression methods often involves weighing the trade-off between reduced sample size from matching and the functional-form assumptions needed for regression, such as linearity of the relationship between the explanatory variables and the dependent variable. Rosenbaum (2002) provides an excellent review of these methods and a discussion of the advantages and disadvantages of matching versus multiple regression in various situations. However, these methods do not help with the key problems of omitted variables bias or sample selection bias because matching is performed on the basis of observed variables only. More accurate and complete data collection, which may involve case studies and surveys, is critical to reducing omitted variables bias by permitting more complete matching and model specification.

Another approach to addressing the problem of omitted variables is to exploit so-called natural experiments to observe the natural variations that occur both before and after a specified time period during which an intervention is introduced. Instead of random assignment, as in a controlled laboratory or field experiment, the researcher defines treatment and comparison groups and uses naturally occurring events for comparisons.

Social scientists have used a “differences-in-differences” approach (i.e., the gender difference in some outcome of interest both before and after an intervention) to test the effects of changes occurring at some specified time period that affect some actors but not others (see, e.g., Card and Krueger, 1994; Tyler et al., 1998; National Research Council, 2004:148-154). In the language of causal modeling, the policy change is a formal manipulation, which is applied to some actors but not others. (In some studies, the policy change affects all actors, and the comparison is done before and after the change.) The pre-policy-change data are used to estimate the counterfactual condition of what would have happened had the policy change not occurred. Such designs are also sometimes called quasi-experiments (see Campbell and Stanley, 1963; Meyer, 1995; Shadish, Cook, and Campbell, 2002).

In the context of federal contracting, should the SBA decide to authorize preferential set-aside contracting programs for women-owned small businesses for selected industries, then researchers could analyze this set of interventions as a natural experiment. Comparisons could be made of the contracting success of women-owned small businesses in the affected industries before and after the program introduction. Comparisons could also be made of the contracting success of women-owned small businesses after the program introduction in the affected industries and other industries. To take full advantage for research of such a policy change (if it indeed is put into effect), the SBA should plan for needed additional data collection in the CCR, the FPDS, and other pertinent data systems as soon as possible after a decision to authorize any new set-aside programs is reached.

Natural experiments have a number of limitations for the study of discrimination:

• The change under study may be endogenous—that is, a reaction to particular circumstances that warranted a policy change or intervention. To the extent that discrimination against women-owned small businesses in contracting exists for industries designated as “underrepresented” but not other industries, then the estimated effect of comparing these industries with other industries before and after the policy change would tend to

overstate the amount of discrimination against women-owned small businesses overall prior to the change.

• The effects of policy interventions may spill over into the control group used in the study. For example, the effects of set-aside programs for women-owned small businesses for some industries may encourage contracting agencies to become more favorable to women-owned small businesses generally. This phenomenon would reduce estimates of the effect of the policy change based on a differences-in-differences design.

• Differences in trends in other factors that affect outcomes cannot always be addressed adequately even in a differences-in-differences design, particularly when the policy intervention takes place over a period of time, as is likely to be the case with a new set-aside program for women-owned small businesses.

• One can hardly ever be sure that the change in policy under study has eliminated a role for discrimination in the decision under study. In most cases, the best one can hope for is that a comparison of groups affected by the change in policy will identify the reduction in discrimination induced by the policy rather than the level of discrimination that existed prior to the change.

• In some cases, changes in policy that lead to positive effects in one dimension may induce negative effects in another. For example, the introduction of a new set-aside contracting program for women-owned small businesses might possibly lead to fewer efforts at outreach.

• Natural variation in the data may be insufficient to identify the effects of interest or may be correlated with other, unmeasured factors that may bias the results. (See Holzer and Ludwig, 2003, on the use of natural experiments to study discrimination; see Meyer, 1995, and Shadish, Cook, and Campbell, 2002, for a general discussion of the strengths and weaknesses of these designs.)

Careful assessment of the quality of input data would be critical for appropriate use of the statistical analysis methods discussed above, as would consideration of needed sample sizes. If sample size is inadequate in disaggregated samples, it may be useful to pool data across several years, agencies, and industries, provided such pooling does not obscure important differences on these dimensions.

With regard to data quality, an important consideration is accurate classification of businesses by type. A recent study commissioned by the SBA Office of Advocacy (Eagle Eye Publishers, Inc., 2004a) estimated that $2 billion of a total of $54 billion of federal contracting funds in 2002 that were classified as awards to small businesses in fact went to businesses for

which the parent company was a large business or to nonprofit organizations or government agencies. A sample-based audit of the CCR and the FPDS could be a way to establish the reliability of the classification of women-owned small businesses versus other businesses in these two key databases on federal procurement. A match commissioned from the Census Bureau of CCR and FPDS records with the Survey of Women-Owned Business Enterprises could be another way of verifying the quality of the CCR and FPDS data.

Other data quality issues would depend on the nature of the analysis and the data sources used. For example, a study of financing barriers for women-owned businesses would require an assessment of sampling and nonsampling errors in such data sources as the Federal Reserve Board’s Survey of Small Business Finances.

We support the conclusion of the Panel on Methods for Assessing Discrimination (National Research Council, 2004:159) that “the use of statistical models, such as multiple regressions, to draw valid inferences about discriminatory behavior requires appropriate data and methods, coupled with a sufficient understanding of the process being studied to justify the necessary assumptions.” It is a challenging undertaking to analyze the possible role of discriminatory practices and behaviors at any point in the federal contracting process, let alone the chain from new business formation to registering and bidding to supply federal requirements.

We believe that in-depth research on disparities and possible discrimination in the contracting process could usefully inform policy making, but such research should be viewed as a long-term investment on the part of the SBA and other interested agencies. It requires development of a staged, prioritized research agenda; collection and evaluation of needed data from small-scale case studies, surveys, and administrative records; and sophisticated, careful analysis using best practices and state-of-the-art research methods.