Defining the Population of Interest
A significant portion of the workshop discussion focused on identifying a probabilistic econometric model for identifying the population of interest consistent with HUD's objectives. Participants suggested that before assessing whether the current model is appropriate for the parameters of interest, some important questions need to be addressed:
What is the model for the outcomes?
What is actually being sampled?
Given the sample, what is the role of sampling weights? How does the analysis move from paired events to a universe of populations?
What is the population to which one can generalize from such events?
How does that universe relate to the housing market?
During the workshop, the sponsors defined the target population as the housing market in the test sites. Participants noted that the target population differs from the population suggested by the sampling frame, and posed the question, What is the correct definition for the population given the sampling frame? The researchers responded that the actual population is the housing stock served by advertisements appearing in a site's major metropolitan newspaper on Sunday.
Participants suggested an alternative sampling frame that would resemble the target population more closely. This sample would include all known newspapers with housing advertisements circulating in the test site. With this method, researchers would lose a priori knowledge of the selection probability for a given advertisement. However, this limited knowledge may not be necessary since the true probabilities can be computed and weighted appropriately. This alternative sampling frame would yield unequal probabilities of selection, but participants did not view this as a major limitation.
Another alternative, raised by Tom Louis of the RAND Corporation, would be to abandon the sample survey goal in favor of an alternative unbiased selection procedure. The objective would be to select neighborhoods or communities in which advertising is prevalent without the goal of selecting a random sample. This method would ignore the sampling procedure and would not be concerned with sample survey weights or statistical comparisons within samples. Louis also noted that the reliance on advertisements produces selection bias since part of the market is missed (e.g., availability that is “advertised” through word of mouth). The researchers responded that pilot tests in the current study design include nonadvertisement sampling that begins to address this issue by identifying areas that are either protected (e.g., gated communities) or neglected (e.g., small communities that do not typically advertise available housing units in newspapers). Field reconnaissance in these areas provides researchers with available housing stock from which a secondary sample is drawn.
Participants discussed several ways to make the sampling frame more realistic. For example, one could look at the income and asset distribution of the minority population in individual metropolitan areas and sample housing units on the basis of that distribution. Urban Institute researchers pointed out that this sampling frame is quite different from one in which the sample of units is based on the distribution of where minorities currently live—an alternative approach offered during the discussion of auditing in underserved communities (see Chapter 6). The researchers expressed concern about this latter modification because it would institutionalize outcomes that may be the result of discrimination.
Participants were asked to discuss ways of making the sample more representative of the target populations, but in a neutral manner with respect to potential variants existing in the housing market. There was substantial agreement that this would be extremely difficult to do unless HUD were able to clarify the population to which it wants to generalize. The
alternatives to the sampling frame discussed by participants included some that represent an attempt to overcome the limitations of the paired-testing methodology and allow alternative forms of generalization. Participants agreed that the current methodology poses many difficulties when applied to underserved communities, in which the paradigm does not appear to work well given differences in housing market structures. For example, it is highly unlikely a white individual would seek housing in some small, predominantly minority community.
POTENTIAL BIAS IN TEST SITE SELECTION
Workshop participants noted that many larger metropolitan areas have newspapers targeted to particular ethnic communities. The housing search patterns for people in some communities, especially minority groups, may not include advertisements in the major Sunday newspaper. In such cases, the initial sampling frame would not encompass advertisements more likely to be read by the subpopulation of interest, a limitation that would have implications for the credibility of the point of entry into the housing market that the researchers were trying to establish. Furthermore, omitting certain newspapers from the sampling frame could compromise the ability to draw inferences about the population the auditors represent.
The researchers noted that the sampling frame will be modified during the pilot phase of the study to include alternative advertisement sources, such as small neighborhood papers. For five specific sites, two types of enhancements to the usual newspaper advertisement sample are being used.
The first type of enhancement involves exploring the overall distribution of newspaper advertisements independently for rentals and sales, with the audit researchers looking at the relative distribution of those advertisements across neighborhoods within an audit site. Researchers will assess the multisource enhancements used in the pilot studies and develop a sampling frame for selecting advertisements in Phase II of the study. In Phase II, researchers will draw from a multiframe sampling source, using multiple neighborhood or community newspapers. However, sampling issues arise from multiple newspaper advertisement sources. In particular, the sponsors asked the workshop participants to discuss the implications for the sampling frame of the potential overlap in housing units—units that are sampled more than once because they are advertised in several papers.
Some participants questioned why multiple newspaper sources were not included in the Phase I study. The researchers responded that they
wished to preserve known probabilities of selection for the advertisements. Devising a method to calculate the sampling probabilities would have been daunting given multiple newspapers and the fact that on a given day, advertisements must be drawn from 22 sites. In addition, the researchers wanted to mirror the 1989 sampling procedure. They acknowledged the limitations of this method.
Researchers noted further that the Phase I sampling frame was chosen to be comparable with the 1989 HDS. The newspaper methodology used for the 1989 and 2000 HDS to select advertised housing units does not reach the entire housing market. In fact, many minorities often do not seek housing opportunities in areas that do not receive a large amount of newspaper coverage. In addition, there are issues within minority neighborhoods and other types of areas regarding obtaining access to the housing stock that is available for both rental and sales. In some instances, it is unlikely a majority individual would seek housing in certain minority neighborhoods. In other instances, housing units may not be advertised in mainstream or community neighborhoods. Participants also discussed issues involved in testing in smaller metropolitan areas, some of which are joint rural-type counties.
The second enhancement of the sample involves obtaining census estimates for the proportion of available rental and sales housing for each test site. During the initial visit, auditors will ask the housing agent for the addresses of similar units. The resulting information will yield an auxiliary set of addresses to be used in identifying underrepresented areas that never reach the major metropolitan newspapers. Census housing market data will be matched with the auxiliary addresses. The researchers will then sample from these underrepresented areas—areas with fewer advertisements than available housing units—which are likely to be in minority communities.
DRAWING INFERENCES TO THE POPULATION OF INTEREST
In assigning tester characteristics, researchers guarantee that auditors are qualified for the housing units. Workshop participants suggested that by allowing for individuals who are on the margin financially, analysts may miss the discrimination of interest. Specifically, some participants believe that pairs of clearly qualified and clearly unqualified home seekers of differing races would receive similar treatment (either acceptance or rejection), but that on the margin, housing agents would make more subjective judg
ments and exercise greater discretion by offering additional assistance or compensation to the nonminority tester. By representing only eminently qualified testers, therefore, the audit results may understate the real degree of housing discrimination in the market. The sponsors responded that they believe discrimination may occur for both marginal and overqualified individuals and that both forms of discrimination are important to measure. The strategy of the study, however, is to identify clear and convincing disparate treatment when less discretion on the part of housing agents is allowed.
Participants discussed whether the population from which the auditors are drawn is a fair representation of the people who are seeking housing, in other words, whether the audit findings can be used to generalize about the discrimination a typical couple would face in the housing market. Participants suggested that unobservable characteristics of the testers may represent differences from the population of home seekers and that these characteristics may not be eliminated through training or protocols. However, as Stephen Ross noted, audit pairs conduct more than one audit. Since many of the pairs are fixed in terms of the people who make them up, their experiences in different settings allow researchers to test for whether unobservable tester characteristics matter. This issue is discussed further in Chapter 5.
Ross noted that the researchers have a difficult problem in dealing with very small sample sizes. HUD wants not only a national measure of housing discrimination and changes in the national incidence of discrimination, but also individual metropolitan measures. The audit methodology will produce only 70 samples. Accordingly, the research team will explore different options for obtaining exact estimates, such as permutation-type estimators or estimators that are corrected for small sample sizes. Such methods were not discussed during the workshop, but Ross (2000) notes that because permutation tests rely on frequency data, weights cannot be used.
WEIGHTING ADVERTISED UNITS
Participants noted it may be impractical to sample every newspaper in a large metropolitan area, but, as noted earlier, researchers are exploring the use of multiple papers and other advertisement resources in smaller areas, where a multiframe sampling source is more practical. The Urban Institute can use data gathered on the relationship between the sample and the popu
lation for these smaller areas to provide information about potential differences in the incidence of discrimination across advertisement sources.
Some participants were unclear on the appropriate weight that should be applied to the results, but noted that it should reflect the universe of available housing stock. Weighting audit results by housing unit size may not be particularly relevant, but weights should incorporate the various sources of information on housing availability, including major metropolitan newspapers, community newspapers, Internet-related databases, the Realtors® database (or MultiList), and other sources. Participants discussed the implications of accessing a fairly comprehensive database for available sales housing, such as the MultiList available to real estate agents. Perhaps such a list would serve as a base against which researchers could sample units and create approximate weights. Although participants did not know of the feasibility of this option, one stated it is more appropriate to obtain approximate weights for the right population than precise equal probability weights for the wrong population. Each specification has associated tradeoffs that should be considered before results are reported.