5
Sampling for Nonresponse Follow-Up: Achieving Adequate Precision at Acceptable Cost

The 1990 census was conducted as a mail-out/mail-back operation with field follow-up to any address that did not return the census form by mail. Although the majority of households returned the form by mail, the most expensive part of the census was the follow-up for nonresponse: it accounted for about 20 percent of the $2.6 billion cost (over 10 years) (Bureau of the Census, 1992; U.S. General Accounting Office, 1992). In addition, completion of nonresponse follow-up took much longer than scheduled, delaying and potentially reducing the quality of the Post-Enumeration Survey (PES), designed to measure the differential undercount. For the 2000 census, the Census Bureau plans to use sampling to reduce costs, to improve management of field operations, and to speed completion of nonresponse follow-up.

Chapter 2 of this report discusses this panel's support for the concept of using sampling for nonresponse follow-up. In this chapter we address the major design decisions that are required to implement it successfully. Although decisions about the nonresponse follow-up design need to take into account estimation plans, we defer most discussion of estimation to our final report. We defer discussion of integrated coverage measurement to Chapter 6, except to the extent that details of that process are needed here.

ALTERNATIVE DESIGNS FOR NONRESPONSE FOLLOW-UP

The Census Bureau will provide more ways to be counted in the 2000 census than ever before. Data will be collected about nongroup housing units by several modes:5 (1) initial census forms returned by mail; (2) return of replacement forms mailed to units that do not respond within 2 weeks of the initial mail-out or mailing of other forms (e.g., in a foreign language) requested by residents; (3) forms completed by telephone; (4) "Be Counted" forms with valid addresses; and (5) forms returned by the postmaster as undeliverable because the unit is vacant. Although modes (2) through (5) might all be considered forms of nonresponse follow-up, we refer to housing units enumerated by all five modes as initial respondents. For this discussion, all of those responses are excluded from the main nonresponse follow-up universe. (We discuss handling of postmaster returns below.) For units in the nonresponse follow-up universe, data collection modes will include: in-person enumeration during nonresponse follow-up; vacancies identified by field enumerators; and close-out procedures (e.g., interviews of neighbors) when

5  

Plans for service-based enumeration aimed at counting people without a usual residence are discussed in Chapter 4 and are not discussed further in this chapter.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 30
Preparing for the 2000 Census: Interim Report II 5 Sampling for Nonresponse Follow-Up: Achieving Adequate Precision at Acceptable Cost The 1990 census was conducted as a mail-out/mail-back operation with field follow-up to any address that did not return the census form by mail. Although the majority of households returned the form by mail, the most expensive part of the census was the follow-up for nonresponse: it accounted for about 20 percent of the $2.6 billion cost (over 10 years) (Bureau of the Census, 1992; U.S. General Accounting Office, 1992). In addition, completion of nonresponse follow-up took much longer than scheduled, delaying and potentially reducing the quality of the Post-Enumeration Survey (PES), designed to measure the differential undercount. For the 2000 census, the Census Bureau plans to use sampling to reduce costs, to improve management of field operations, and to speed completion of nonresponse follow-up. Chapter 2 of this report discusses this panel's support for the concept of using sampling for nonresponse follow-up. In this chapter we address the major design decisions that are required to implement it successfully. Although decisions about the nonresponse follow-up design need to take into account estimation plans, we defer most discussion of estimation to our final report. We defer discussion of integrated coverage measurement to Chapter 6, except to the extent that details of that process are needed here. ALTERNATIVE DESIGNS FOR NONRESPONSE FOLLOW-UP The Census Bureau will provide more ways to be counted in the 2000 census than ever before. Data will be collected about nongroup housing units by several modes:5 (1) initial census forms returned by mail; (2) return of replacement forms mailed to units that do not respond within 2 weeks of the initial mail-out or mailing of other forms (e.g., in a foreign language) requested by residents; (3) forms completed by telephone; (4) "Be Counted" forms with valid addresses; and (5) forms returned by the postmaster as undeliverable because the unit is vacant. Although modes (2) through (5) might all be considered forms of nonresponse follow-up, we refer to housing units enumerated by all five modes as initial respondents. For this discussion, all of those responses are excluded from the main nonresponse follow-up universe. (We discuss handling of postmaster returns below.) For units in the nonresponse follow-up universe, data collection modes will include: in-person enumeration during nonresponse follow-up; vacancies identified by field enumerators; and close-out procedures (e.g., interviews of neighbors) when 5   Plans for service-based enumeration aimed at counting people without a usual residence are discussed in Chapter 4 and are not discussed further in this chapter.

OCR for page 30
Preparing for the 2000 Census: Interim Report II attempts to contact residents are unsuccessful. In February 1996 the Census Bureau officially introduced its plan to use sampling for nonresponse follow-up in the 2000 census. That plan calls for a two-stage design. During stage 1, the Census Bureau would conduct follow-up by enumerators in the field to raise the response rate to at least 90 percent of housing units on the MAF in each census tract.6 Once the 90 percent mark has been reached, stage 2 follow-up would be conducted on a 1-in-10 random sample of the remaining housing units. Characteristics for the remaining nonsampled housing units would be estimated on the basis of the random sample. Below, we describe more fully this design and two alternatives that the Census Bureau has considered. During stage 1 of nonresponse follow-up, the Census Bureau would begin personal visits to nonresponding housing units in the vast majority of tracts, much as in 1990. Field enumerators would visit housing units in the nonresponse follow-up universe, asking residents the questions on the census form or helping them to fill out the form. Residents who were initially mailed the long form would be asked to fill out the long form. Units where no one was contacted on the initial visit would be scheduled for revisits. In contrast to 1990, however, this stage of nonresponse follow-up would cease in a tract once the response rate (initial response plus this work) had reached 90 percent: that is, when a completed census form or confirmation that the dwelling was vacant had been obtained for 90 percent of the addresses to which a census form had been originally mailed. In a tract with a 90 percent or higher initial response rate--which would be rare— no stage 1 nonresponse follow-up would be conducted. Because the goal of stage 1 nonresponse follow-up is to reach 90 percent as quickly and inexpensively as possible, those responses would not necessarily represent the whole nonresponse follow-up universe. For example, the overall response for a tract could reach 90 percent before any follow-up had occurred in certain blocks. Also, stage 1 responses might tend to exclude housing units with difficult access or where residents were seldom at home. To allow unbiased estimation for the final 10 percent of housing units in each tract, the Census Bureau's announced plans for stage 2 of nonresponse follow-up include a 1-in-10 random sample of the remaining nonresponding housing units. Consequently, there would be direct enumerations for at least 91 percent of housing units in each tract at the end of stage 2 nonresponse follow-up. In contrast to stage 1, in stage 2 data would be collected for each sampled unit. Enumerators would make repeated visits to a unit-ideally at different times of the day and different days of the week--until a resident 6   The Census Bureau originally planned to use counties as the geographic unit for controlling completion of stage 1 nonresponse follow-up. That choice drew wide criticism from the Census Bureau's advisory groups and other interested parties: the concern was that many counties are so heterogeneous that a countywide response rate of 90 percent could be achieved while large areas of the county trailed far behind (White and Rust, 1996). In September 1996 the Census Bureau announced a revised plan to reach 90 percent in each tract. Because the country's 65,000 census tracts, which average 4,000 people and 1,500 housing units, are much smaller than the average county, the revised plan will greatly smooth out response rates at the completion of stage 1 nonresponse follow-up.

OCR for page 30
Preparing for the 2000 Census: Interim Report II responds or they determine that the unit is vacant. For units for which these visits do not yield a response before the final days of nonresponse follow-up, enumerators would conduct closeout procedures--interviewing neighbors to procure proxy information or estimating the number of residents from what they know about the neighborhood. We refer to this plan as the 90-percent truncation design. In this design, as well as others under consideration, characteristics of the nonsampled housing units would be estimated on the basis of the random sample. The Census Bureau is also considering two alternative plans for nonresponse follow-up (Killion, 1996b). Under the first alternative, time truncation, the stage 1 of nonresponse follow-up would be truncated at a given time. Instead of ending when the response rate reaches 90 percent in a census tract, this stage would last a specified period of time, such as 3 weeks, to allow making at least one visit to every housing unit in the nonresponse follow-up universe. The Census Bureau would then draw a large enough random sample of the remaining units to reach 90 percent response in each tract after completion of stage 2. In tracts that had already reached 90 percent or higher in stage 1, a 1-in-10 sample would be drawn. Under a second alternative, referred to as direct sampling to 90 percent, sampling for nonresponse follow-up would begin immediately after the mail-back period, with no stage 1. Nonresponse follow-up sampling rates would be set to achieve a final response rate of 90 percent in each census tract. For example, a 1-in-2 sampling rate for nonrespondents would be used in tracts with initial response rates of 80 percent, while a 5-in-6 sampling rate would be used in tracts with a 40 percent response rate. For all tracts with a initial response rate of 90 percent or higher, a 1-in-10 sampling rate would be used. Figure 1 shows a general schematic that covers all three design alternatives. "A" represents the percentage of housing units that respond by mail or are otherwise excluded from the nonresponse follow-up universe. The initial response rate (A) does not depend on the nonresponse follow-up design and will vary greatly from tract to tract. The Census Bureau has predicted an average initial response (excluding postmaster returns) of 67 percent across all tracts in the United States (Killion, 1996b), but with great variation across tracts. The percentage of not-sampled housing units (D) would be approximately the same (9 or 10 percent) in each tract for each of the three designs under consideration. FIGURE 1. Stages of enumeration. NRFU, nonresponse follow-up. The designs differ mainly in the allocation of nonresponse follow-up between B and C, where B denotes the percentage of households for which data are collected in stage 1, and C denotes the percentage for which data are collected in stage 2. In the sampling

OCR for page 30
Preparing for the 2000 Census: Interim Report II design in the 90 percent truncation design, almost all the effort is in the stage 1 part (B = 90 - A) and sets C at 1 percent of the population; the time truncation design is a compromise between B and C, with the relative effort varying among tracts. Under each of the design alternatives, the current assumption is that the Census Bureau will use characteristics and counts from the sampled housing units to estimate corresponding characteristics and counts for the nonsampled units. For example, under 90 percent truncation design, with a 1-in-10 sampling rate, data from each sampled household will be imputed to nine nonsampled housing units. This estimation method produces unbiased estimates of the results that would be achieved using complete nonresponse follow-up. Estimation for any of the designs will essentially result in weighting each sampled housing unit by the inverse of the sampling rate for the tract, with this sample weighting implemented by means of imputation. ASSESSMENT OF THE BASIC DESIGN ALTERNATIVES In this section we assess the above three nonresponse follow-up designs, and others, three using three main criteria: operational feasibility, cost, and size and equity of sampling variation. In addition, we consider how public reactions might differ for design variations. The choice of sampling unit (housing unit or block), a major design component, is discussed in the next section. Operational and Costs Issues Nonresponse follow-up is a massive logistical operation with few, if any, peace-time rivals. In order to complete nonresponse follow-up within a 6-week schedule, the Census Bureau estimates a need for 550 temporary offices with 260,000 to 300,000 enumerators (Bureau of the Census, 1996a, 1997). Although nonresponse follow-up was also scheduled for 6 weeks in 1990, only 72 percent of the workload was completed on time, and some district offices did not complete nonresponse follow-up until the 14th week (U.S. Government Accounting Office, 1992). Timely completion of nonresponse follow-up is critical in 2000 so that integrated coverage measurement can begin in time for production of reapportionment counts by December 31, 2000. In addition, as we discuss in Chapter 2, there is reason to expect that reducing the time to completion would improve the accuracy of enumerations during the latter stage of nonresponse follow-up; Ericksen et al. (1991) found that the rate of erroneous enumeration increased with time to completion in 1990. Despite the massive number of enumerators, nonresponse follow-up will need to be conducted in waves. Enumerators will receive their assignments in sets of about 40 units at the beginning of the nonresponse follow-up period; when an assignment is completed, or nearly so, the enumerator receives a new set. It is anticipated that the average completion time per set will be about 1-1/2 weeks, although times will vary by neighborhood characteristics, enumerator productivity, and enumerator work hours (part

OCR for page 30
Preparing for the 2000 Census: Interim Report II time for most enumerators). Conducting nonresponse follow-up in waves allows for flexible assignment of workload without requiring enumerators to report daily. Timely communication between supervisors and their staffs is critical to the successful completion of the field work on time. There is no way to schedule the enumeration workload accurately. Despite the short length of the nonresponse follow-up operation, it will require a very large number of replacement enumerators in many areas: during the 1995 census test, the average enumerator tenure was just 2 weeks, including training time. Consequently, some areas are bound to fall behind schedule as enumerators cut back on their scheduled hours, complete a temporary employment period, or resign unexpectedly. If there is timely feedback on completion rates, there may be the opportunity to catch up by transferring staff from nearby areas that are ahead of schedule. While the logistical challenges will be great for any of the three nonresponse follow-up designs being considered, the two-stage designs face additional problems. Compared with direct sampling, the truncation designs place greater importance on timely communication for two reasons. First, reaching 90 percent response in all tracts during stage 1 may depend on not "overshooting" in some tracts. This requires up-to-date data on tract-level completion rates and the ability to modify enumerators' assignments quickly. Second, the use of a two-stage design increases the importance of not falling behind schedule in any area (in either stage 1 or stage 2) because there is little time to catch up. In initial discussions of the truncation designs, Census Bureau field managers anticipated that the transition from stage 1 to stage 2 of nonresponse follow-up would lead to a 1-week hiatus in enumeration operations, resulting in substantial staff loss, more than occurs during operations. The lost field time and staffing problems could make it impossible to complete the workload within the required schedule. On the basis of this anticipated break in operations, the Census Bureau estimates that the 90 percent truncation design would cost about $300 million more than direct sampling (total census costs over 10 years of $4.0 and $3.7 billion, respectively). Due to the need to visit every nonresponding housing unit, the Census Bureau estimates that the time truncation design would cost $700 million more than direct sampling ($4.4 billion total cost). The panel remains somewhat skeptical of the cost disadvantages attributed to the truncation designs, especially the time truncation design. In theory, those designs might be less costly than anticipated by the Census Bureau because they cover more of the easier-to-reach households in stage 1 and take as little as a 1-in-10 sample of the hardest-to-reach units. In contrast, direct sampling draws equally from easy- and hard-to-enumerate units in the nonresponse universe. Also, it might be possible to stagger the field operations to avoid down time of more than a couple days, if any, for most enumerators. As noted above, nonresponse follow-up enumeration needs to occur in waves. Consequently, stage 1 in some tracts could be completed while stage 1 in other nearby tracts continues or, perhaps, before it even begins. While stage 1 enumeration continues in "late" tracts, the Census Bureau could select stage 2 samples and distribute assignments for tracts (or parts of tracts) where stage 1 has been completed, eliminating the need for a break in field operations.

OCR for page 30
Preparing for the 2000 Census: Interim Report II Careful assignment of initial workloads would help to facilitate this process. Despite our skepticism on certain points, the panel still recognizes the substantial advantages of a onestage procedure, such as direct sampling, in terms of training, scheduling, and monitoring. Sampling Variability Associated with Alternative Designs Table 1 compares coefficients of variation (CV, see Chapter 2) of estimated population counts under three different nonresponse follow-up designs for a range of initial response rates. We use a constant tract size (1,500 housing units) and assume that the ratio of the standard deviation of household size to mean household size is a constant 0.67 during each stage of nonresponse follow-up.7 The table accounts only for variation due to nonresponse follow-up sampling.8 Under these assumptions, the 90 percent truncation design (with 1-in-10 sampling of the final 10 percent) produces a constant coefficient of variation of 1.64 percent for all initial response rates up to 90 percent (first section of Table 1). For a tract with a population of 4,000, that translates into a standard error of 66 people, implying that there is a two-thirds probability that the estimated population falls between 3,934 and 4,066. In contrast, for the great majority of tracts that have initial response rates of 83 percent or less, direct sampling to 90 percent would cut the coefficient of variation by half or more. However, this benefit of direct sampling to 90 percent in each tract would not be realized in tracts with high response rates. In fact, that design would have the somewhat perverse property of producing the least accurate results in areas with initial response rates of close to 90 percent (the coefficient of variation would gradually fall for tracts that achieved rates above 90 percent). The final section of Table 1 presents a direct sampling design with sampling rates set to equalize the coefficient of variation among tracts of the same size, but with different initial response rates. The overall workload under this modified direct sampling design was chosen to match or fall slightly below that for direct sampling to 90 percent. The result is a uniform reduction in the coefficient of variation by a factor of 2.1 (a 4.5-fold reduction in the variance), compared with the 90 percent truncation plan. We do not include the time-truncation design in Table 1 because the results depend critically 7   The ratio of 0.67 is based on using the national distribution of household size (Bureau of the Census, 1990) and assuming that 10 percent of housing units are vacant at each stage of nonresponse follow-up. The correct value might be higher, to the extent that vacant units are more likely to fall into the nonresponse follow-up sample, or lower, to the extent that within-tract variation in household size is less than the overall variation. 8   We use the following formula: CV2 = 100 (D + D2/C) (0.67)2 / n , where C and D are measured in percents (Thibaudeau and Navarro, 1995), and n = the number of HUs in a tract. If estimation for nonsampled units is done by imputation (weights restricted to integers), this formula slightly underestimates the coefficient of variation when the sampling rate, C/(C+D), is not the inverse of an integer.

OCR for page 30
Preparing for the 2000 Census: Interim Report II TABLE 1 Coefficient of Variation (CV) for Population Counts and Percentage of Housing Units Reached for an Average-Sized Census Tract Under Three Alternative Designs, by Initial Response Rate   90% Truncation, 1-in-10 Sample of Last 10 Direct Sampling, to 90% Direct Sampling, Equal CVs Initial Response Rate (%) Percent Reached CV (in %) Percent Reached CV (in %) Percent Reached CV (in %) 40 91 1.64 90 .60 85.1 .77 50 91 1.64 90 .61 85.8 .77 60 91 1.64 90 .63 86.7 .77 70 91 1.64 90 .67 88.1 .77 75 91 1.64 90 .71 88.9 .77 80 91 1.64 90 .77 90.0 .77 85 91 1.64 90 .95 91.5 .77 90 91 1.64 91 1.64 93.4 .77 NOTES: The table assumes: a constant tract size of n = 1,500 housing units; a constant ratio of 0.67 for the standard deviation of household size divided by the mean household size; and sample data that are weighted by the inverse of the sampling rate.

OCR for page 30
Preparing for the 2000 Census: Interim Report II on response rates in stage 1, which are unknown. Results would tend to fall between the two extremes for 90 percent truncation and direct sampling to 90 percent. Because the computations for Table 1 rest on assumptions that cannot be verified until the 2000 census, the absolute levels of the coefficients of variation should be interpreted cautiously. However, we believe that relative comparisons among the designs are fairly accurate. Similar tables could be constructed for tracts of other sizes or for smaller geographic units, such as block groups or even individual blocks (assuming that 90 percent was reached uniformly within tracts under the truncation design). For smaller areas, the coefficients of variation would grow in inverse proportion to the square root of the number of housing units. Consequently, the relative advantages of the direct sampling designs would not change from those shown in Table 1. For larger geographic areas, such as congressional districts and states, sampling variability due to nonresponse follow-up sampling would shrink to 0.1 percent or less under direct sampling. At this level of geography, variability due to nonresponse followup would be negligible compared with sampling variability due to integrated coverage measurement--about 0.5 percent for most congressional districts (Killion, 1996b). Consequently, the choice among these three nonresponse follow-up designs will not significantly affect the precision of the population counts for those larger geographic units. The inefficiency of the 90 percent truncation design stems from the low sampling rate of 1 in 10, which means that each sampled unit gets counted (weighted) ten times. In contrast, no units gets counted more than about three times in the direct sampling design aimed at equalizing the coefficients of variation. We note that there are alternative estimation methods (e.g., ones that incorporate data from the initial or stage 1 respondents) for which the relative disadvantage of two-stage designs would not be so large. However, those methods would introduce further complexity or reliance on assumptions. The design alternatives also need to be evaluated in terms of how well they achieve equitable accuracy across areas. Of course, the first criterion is reduction of the differential undercount, which is why we believe that integrated coverage measurement is essential. However, we do not expect that sampling for nonresponse follow-up will have a significant impact on this goal (positive or negative), and we do not see any evidence that the design options differ in this regard. In contrast, the options do differ substantially in how well they equalize precision across tracts. We believe that the appropriate goal is that tracts of equal size have equal sampling variability (Thibaudeau and Navarro, 1995). Because the accuracy of the data is key, we believe that this measure of equity should prevail over others (such as equal final-response rate which penalizes areas with high initial response under direct sampling to 90 percent, or equal effort, which penalizes areas with low initial response). An argument could be made for a plan under which tracts with the highest initial response rates are sampled enough to produce the lowest sampling variability, in order to motivate greater mail response. However, until research efforts provide evidence that such a plan would improve the overall response, we are forced to assume that such a policy would not draw any less resistance than one that penalizes areas with high initial response. An exception to the equity goal might apply to very small political jurisdictions,

OCR for page 30
Preparing for the 2000 Census: Interim Report II many of which are smaller than the tract size specified in Table 1. Because of the specific uses of the population counts for those jurisdictions, the Census Bureau might well choose to use higher sampling rates than for tracts in larger jurisdictions. Such a policy would be analogous to the higher rate of 1990 long-form sampling in jurisdictions with less than 2,500 population. The additional cost of this would be small compared with the overall nonresponse follow-up budget because the total population of these areas is fairly small. Other Criteria Under any nonresponse follow-up design, some housing units in the nonresponse sample will require closeout procedures after multiple visits fail to yield a response from a resident. It is difficult to know how much systematic or random error the closeouts add to the final estimates, but it is clear that closeouts are less desirable than direct responses. Although 90 percent truncation would probably require fewer closeouts than direct sampling, those cases would receive more weight under the former design and thus actually contribute as much (or more) of the final imputed data on which counts will be based. For example, consider a tract containing 50 housing units for which it was impossible to get a response under any design (the census day residents have moved or they just do not answer the door): one would expect to get about 5 of them in the 90 percent truncation sample and perhaps 20-30 with direct sampling, and they would receive approximately the same total weight in the final estimate. For the reasons noted above, direct sampling would reduce the sampling variability associated with those units. There is also some reason to expect that the weighted number of closeouts might be higher under the two-stage designs. The likelihood of getting a response from a sampled unit increases with the number of knocks on the door and the time span between the first and last knock. The shorter sampling period under the truncation designs may increase the number of closeouts because of census day residents who move early in the nonresponse follow-up period. Also, if the two-stage designs are less efficient operationally, they might produce fewer visits to each sampled unit before the need to use closeout procedures. However, the larger concern is that some tracts will fall well short of the 90 percent goal at the scheduled end of stage 1 nonresponse follow-up. If the response rate in a tract is still only 60 or 70 percent, the Census Bureau will need to choose among arbitrarily closing out a large number of units, delaying the beginning of stage 2, or modifying the sampling plan design. Direct sampling also offers the advantage of improved control over where data are collected relative to that for truncation designs. Under the 90 percent truncation plan, response rates for different parts of a tract could differ greatly at the end of stage 1 even if the initial response rate was constant. Because enumerators will work on different days, the tractwide goal of 90-percent truncation before sampling could easily be reached by achieving 95 percent for the part of the tract with mainly single-family residences, but only 70 percent in a large apartment building. Although tract-level counts and characteristics would achieve the desired accuracy, results for apartment renters in the tract would have greater-than-average sampling error. Similarly, without adequate

OCR for page 30
Preparing for the 2000 Census: Interim Report II controls, some enumerators might choose to improve their ''yield" by systematically avoiding units that were assigned the more time-consuming long form. In those areas, the accuracy of long-form data would suffer disproportionately. Under direct sampling, neither of these problems can occur because there is strict control of the units that provide nonresponse follow-up data. Another advantage of direct sampling is that it shortens the time period between census day and the beginning of the sampling operation, which reduces the number of errors due to mobility and some respondent memory problems. However, the Census Bureau anticipates public perception problems with any direct sampling design. Because public cooperation is vital to the success of the census, it is important to maintain public confidence that the Census Bureau is trying to count everyone and that the final counts can be believed. In focus groups conducted prior to the 1995 census test, residents of the test sites expressed particular concern about the use of sampling that begins before an effort to reach everyone but seemed more willing to accept sampling after response had already reached 90 percent (Research/Strategy/Management and Beldon & Russonello, 1995). The Census Bureau also anticipates opposition to any plan that fails to reach at least 90 percent of households in every tract (such as the third design in Table 1). The panel believes, however, that such focus group information may be largely anecdotal and that accurately predicting public perceptions or responses to specific sampling schemes is not yet possible (see below). Although public perception appears to favor the 90 percent truncation design, we caution against putting too much faith in that assessment. The focus groups described in Research/Strategy/Management and Beldon & Russonello (1995) covered eight questions, only two of which focused on sampling designs. Consequently, the negative reaction to direct sampling may be based on a superficial presentation of the options, and some people may distrust any use of sampling. However, we believe that 90 percent truncation would fare no better than direct sampling in public reactions if its drawbacks were made clear. Conclusions About Design Alternatives To place the various alternatives on equal footing, we have considered only designs with roughly comparable workloads. We see no reason that our conclusions in this section would not apply equally well for sets of alternatives that involved either a heavier or lighter workload. We believe that the best design involves direct sampling with sampling rates designed to equalize variances for tracts of equal size. Direct sampling is the most cost-effective use of resources during nonresponse follow-up. Relative to direct sampling, either truncation design being considered causes additional logistical challenges, is likely to increase costs, and would produce more variable estimates. Truncation might also produce greater inequity of precision among tracts. Although the discussion in this chapter concentrates on sampling at the census tract level, in Chapter 2 we point out the appropriateness of evaluating sampling error at geographic levels where counts have important legal, political, and financial implications.

OCR for page 30
Preparing for the 2000 Census: Interim Report II We note that the relative accuracy of alternative sampling schemes will hold at any geographic level. Recommendation: The Census Bureau should endorse direct sampling for nonresponse follow-up, using a design aimed at equalizing coefficients of variation for tracts of the same size. In comparing alternative designs, the evaluation of sampling error should take place at the geographic levels at which counts have important legal, political, and financial implications. The Census Bureau should begin immediately to educate the public and census users on the general merits of sampling for nonresponse follow-up and the specific merits of direct sampling. We recognize that a variety of concerns may mitigate against our recommended nonresponse follow-up design. For example, should negative public reaction to final response levels below 90 percent necessitate setting a floor such as 90 percent for all census tracts, modifying the direct sampling design by setting a minimum sampling rate of at least 1 in 4 or 1 in 3 would avoid the worst inefficiency and ensure roughly equal accuracy across the range of initial response rates. A similar modification could ameliorate the worst statistical aspect of time truncation. Finally, we note that 85 percent truncation followed by 1-in-3 sampling would produce a constant coefficient of variation of 0.95, 42 percent below that for 90 percent truncation. The key to all of these options is to avoid a very low sampling rate in any tract. Recommendation: If the Census Bureau needs to select a design alternative other than direct sampling with equal coefficients of variation, it is imperative that they avoid using a very low sampling rate, such as 1 in 10, in any tracts. OTHER DESIGN ISSUES Determining the Appropriate Amount of Nonresponse Follow-Up Based on negative reactions to previous alternatives that involved final response levels below 90 percent (A+B+C in Figure 1, above), the Census Bureau is considering only options with final response levels of at least 90 percent. It is not clear to the panel that that is the right target level. For a particular nonresponse follow-up design (e.g., direct sampling to equalize coefficient of variation) with an average final response level in the neighborhood of 90 percent, the exact level involves a policy tradeoff of costs and sampling variability: How much nonresponse follow-up is the public willing to pay for? How much sampling error is the public willing to accept? Although we cannot answer those two questions, we can offer a perspective for considering the latter question. No census is without error. Even if complete nonresponse follow-up is conducted, small-area counts and data on population characteristics will include errors due to misinterpretation of residency rules, incorrect information collected from proxy respondents, and other problems. A critical problem in specifying the amount of nonresponse follow-up to conduct

OCR for page 30
Preparing for the 2000 Census: Interim Report II is that there is no way to accurately forecast the size of the follow-up population. In 1990 the mail response rate was 7 percentage points lower than it had been in 1980, ballooning the nonresponse follow-up universe by about 37 percent (Edmonston and Schultze, 1995). This unexpected increase in workload was a primary factor in the cost and schedule overruns of the 1990 census. While numerous innovations hold promise to improve the mail response rate in 2000 (not necessarily relative to the rate achieved in 1990, but relative to what would be achieved in 2000 without the innovations), it is not possible to predict what that rate will be. Plans that set a specific target for the final response rate risk repeating the problems of 1990 if the initial response rate is too far below what was forecast. With the reduced time in 2000 to complete nonresponse follow-up, this approach risks a large increase in the number of closeouts in some places and uneven accuracy in the final results. Thus, a better approach would be a flexible policy that sets the total nonresponse follow-up workload within realistic time limits and allows the final overall response rate to follow the overall initial response rate. Another risk to the successful completion of nonresponse follow-up is insufficient funding. If a certain workload is mandated but funding is not adequate to hire qualified staff for the time needed for multiple follow-ups to find residents, then the quality of the census results will suffer significantly. While nonresponse follow-up sampling offers the prospect for improving the quality of that operation at reduced costs, an attempt to extract too much cost savings for a given reduction in workload would be counterproductive. Recommendation: Whatever plan for nonresponse follow-up is adopted, it is imperative that it be executed with an adequate budget. A modest nonresponse follow-up design (i.e., one with a relatively small workload) that is executed as planned is far superior to a more ambitious design that runs short of time or resources. Unit of Sampling In the 1995 census test the Census Bureau evaluated two distinct methods for selection of the samples immediately after completion of the initial response period: a housing-unit sample, in which nonresponding units were sampled randomly from each block (possibly none in some blocks), and a block sample, in which a 100 percent nonresponse follow-up effort was conducted in a random sample of blocks (with no nonresponse follow-up in nonsampled blocks). Direct sampling was used in the test, but the same issues would arise for truncation designs and for direct sampling (with various sampling rates). In the absence of integrated coverage measurement, the most efficient choice would probably be a housing-unit sample. Consideration of a block sample arises from the need for complete nonresponse follow-up in integrated coverage measurement blocks (and, perhaps, in surrounding blocks). A housing-unit sample would require use of different procedures in the blocks that are in and not in integrated coverage measurement. Three factors need to be considered in choosing between the two types of samples:

OCR for page 30
Preparing for the 2000 Census: Interim Report II The potential cost advantages of a block design: Enumerators might be more productive in carrying out a block design due to reduced travel time, less time spent reorienting themselves to addresses, and the opportunity to find multiple respondents at the same time. The design effect of a block sample. Systematic differences among blocks within the same sampling stratum in terms of counts and other characteristics for nonresponse follow-up housing units (positive intrablock correlation) would imply larger sampling variances for a block sample than for a housing-unit sample of the same size. The ratio of the variance for a block sample to that for a housing-unit sample is known as the design effect (Kish, 1965). The potential for differential coverage in integrated coverage measurement blocks in comparison with other blocks with a unit-sample design: integrated coverage measurement estimation assumes that previous census operations occurred uniformly in both blocks alike. If instead, coverage in integrated coverage measurement blocks had differed systematically from that in non-integrated coverage measurement blocks, that fact would bias estimates from integrated coverage measurement. If systematic differences occurred and varied by geographic area or population group, that interaction would affect the integrated coverage measurement correction of the differential undercount. Use of a housing-unit sample in blocks not in integrated coverage measurement would introduce nonuniform procedures, increasing the opportunity for systematic differences between the two types of blocks. The 1995 census test provided empirical evidence on each of these factors. Based on that evidence, the Census Bureau has decided to use a housing-unit sample in blocks without integrated coverage measurement. There was no evidence of a cost difference under 90 percent truncation and very little difference under direct sampling (Treat, 1996a). Given the proximity of adjacent blocks, that finding seems quite understandable. The Census Bureau used 1990 data for the 1995 census test sites to estimate the relative size of sampling error that would result from block sampling versus unit sampling for nonresponse follow-up. For the two urban sites, the ratios were approximately 3 to 1 (Vacca, Mulry, and Killion, 1996), substantially larger than any of the ratios among designs considered in Table 1. These ratios are comparable to those found by Schindler (1993), using 1990 census data for Connecticut. These results provide strong reason to prefer a unit design for nonresponse follow-up, unless the integrated coverage measurement blocks (a block sample) would experience systematically different coverage during nonresponse follow-up. Using 1995 census test data, the Census Bureau compared results for the block and housing-unit samples for five nonresponse follow-up characteristics: average household size, vacancy rate, whole household substitution rate, last resort rate, and distribution of household size. No statistically significant main effects occurred at the level of alpha = 0.2 (two-sided tests). However, the limited size of the 1995 census test does not rule out the possibility that substantial systematic coverage differentials exist for one or more of these outcomes. The conclusion that coverage differentials do not exist requires additional evidence, for example, about the nature of the field operations. Indeed, Census Bureau personnel believe that the enumerators in the 1995 census

OCR for page 30
Preparing for the 2000 Census: Interim Report II test did not know what type of block they were in; they simply received lists of addresses to visit and went to those addresses. However, the stakes will be much higher in 2000. Under 90 percent truncation, just 1 percent of housing units in blocks without integrated coverage measurement would be visited during the stage 2 nonresponse follow-up, so that a block where an enumerator visits several units may stand out as unusual. One step to reduce identification of integrated coverage measurement blocks (Census Bureau field personnel may have better suggestions) might be to divide nonresponse follow-up sampling among multiple enumerators in integrated coverage measurement blocks. We note that the likelihood that integrated coverage measurement blocks would be compromised is greatly reduced with direct sampling. In the procedures for the 2000 census, there will be statistical power to detect even relatively minor coverage-differential main effects or interactions that the 1995 test could not detect. Current plans call for approximately 25,000 integrated coverage measurement blocks and about 3 million housing-unit sample blocks (those without integrated coverage measurement)--compared with 544 block-sample blocks and 1,485 unit-sample blocks in 1995. Consequently, the standard errors for estimated differentials will be about one-eighth as large as they were in the 1995 census test. Recommendation: We support the decision to use the housing unit as the sampling unit during nonresponse follow-up. However, the Census Bureau should consider procedures to disguise the integrated coverage measurement blocks from enumerators, residents, and others. Follow-Up of Units with Postmaster Returns During the 1995 census test, 6 percent of prenotice letters and 7 percent of initial questionnaires across the two urban sites were returned by postmasters for being undeliverable as addressed. About two-thirds of the postmaster returns identified the unit as vacant, rather than nonvacant (e.g., nonexistent or bad address). In 1990, all postmaster returns of the census form (there was no prenotice letter) were visited to verify that the unit was either vacant or nonexistent. The official plans for 2000 call for follow-up visits to a 1-in-10 sample of postmaster returns identified as vacant and estimation for the other 90 percent, in an operation separate from the main nonresponse follow-up. Analysis of data from the 1995 census test (Green and Vazquez, 1996) provides valuable information on whether to use postmaster returns from the prenotice letter or the initial questionnaire to identify likely vacant or nonexistent units, when to follow up returns, and how large a sample to follow up. Of the initial questionnaires returned from units identified as vacant, integrated coverage measurement confirmed that 66 percent of those in Oakland and 59 percent in Paterson were indeed vacant on census day; however, 28 and 35 percent, respectively, were identified as occupied. Rates for the units from returns of the prenotice letter were slightly worse and those for nonvacant units were substantially worse. Given these findings, it makes sense to handle vacant units from postmaster returns separately from the main nonresponse follow-up to avoid the cost of mailing replacement questionnaires

OCR for page 30
Preparing for the 2000 Census: Interim Report II and to reduce the time between census day and the follow-up operation. It is not as clear to us how to handle other types of postmaster returns. Is 10 percent the best proportion of returns to follow up? Optimal design theory says that, given fixed resources, the ratio of sampling rates for two strata should be proportional to the ratio of the within-strata standard deviations divided by the square root of the ratio of the per-unit sampling costs in the two strata (Cochran, 1977). We propose an average sampling rate that exceeds 50 percent for the main nonresponse follow-up population. A 10 percent rate for the postmaster returns would be too low unless those cases are more costly to follow up (the reverse seems likely) or variation among the returns is very small compared with that in the nonresponse follow-up population. The latter might be true if very few of the returns are actually occupied. However, at an occupancy rate of about 30 percent, the variance of household counts for vacant units identified by postmaster returns would be roughly the same as that for the main nonresponse follow-up sample. Recommendation: The Census Bureau should base the sampling rate for postmaster identified vacant units on optimal allocation of resources across all follow-up activities. Role of Administrative Records Prior to December 1996, Census Bureau plans called for administrative records (or combinations of records) to substitute for field enumeration for selected housing units. These records were to be taken at face value--that is, there would be no field validation of those records. Administrative records deemed inadequate for substitution would not be used at all in nonresponse follow-up. The Census Bureau had not developed or tested criteria for specifying which administrative records might be used in such a way. Consequently, the Bureau set an apparently conservative goal of substituting for just 5 percent of nonrespondents (about 1.7 percent of all housing units). In December 1996 the Bureau announced that it was dropping plans to use administrative records data in this way. The decision resulted from the fact that the necessary research into such use of administrative records was not sufficiently advanced to permit such use by 2000. The panel questions whether the proposed use of administrative records would ever prove effective, as the plan seems to err in both directions. Given the problems with even the best administrative records, none should be accepted as truth without some form of validation during the census. At the same time, the plan ignores the potential for substantial information in records that are not judged adequate for substitution (even the lack of an address for certain records carries information). A more promising approach would seem to be to use administrative records in conjunction with field validation for a sample of housing units. Administrative records could then be used as inputs for estimating the attributes of housing units with no field follow-up (Zanutto, 1996). We understand that use of administrative records in this way has not been precluded for the 2000 census; however, difficulties in matching households

OCR for page 30
Preparing for the 2000 Census: Interim Report II from administrative records and the census (see Chapter 7) make it unlikely that such an approach can be used effectively in 2000.