2
Fundamentals of Coverage Measurement

The decennial census is used for a wide variety of purposes by federal, state, and local governments, by businesses, and by academe. However, the Constitutional goal of the census is to allocate the population to the states and local areas to support apportionment and congressional redistricting. This use of the census counts makes determination of the correct location of enumerations especially important and also focuses attention on racial differentials. Further, due to racial segregation, differential net undercoverage is likely to impact geographic differential undercoverage. Clearly, the broad goal of measuring the quality of the coverage of the census is to assess the extent of census coverage error by domain and by demographic group.

Coverage measurement is a collection of techniques that measure the differences between census enumerations and the corresponding true counts for groups or areas. Coverage measurement is the quantitative aspect of coverage evaluation, which also encompasses more qualitative techniques, such as ethnographic observation. The differences between census counts and the corresponding true counts at the level of the individual (or the household) are referred to collectively as census coverage errors, and in this chapter we categorize types of census coverage error and indicate methods that can be used for their summarization. We then detail the three primary (potential) uses of census coverage measurement that rely on summarizations. Finally, we provide a brief overview of the methods that are currently used in the U.S. census for coverage measurement.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 15
2 Fundamentals of Coverage Measurement The decennial census is used for a wide variety of purposes by fed­ eral, state, and local governments, by businesses, and by academe. How­ ever, the Constitutional goal of the census is to allocate the population to the states and local areas to support apportionment and congressional redistricting. This use of the census counts makes determination of the correct location of enumerations especially important and also focuses attention on racial differentials. Further, due to racial segregation, dif­ ferential net undercoverage is likely to impact geographic differential undercoverage. Clearly, the broad goal of measuring the quality of the coverage of the census is to assess the extent of census coverage error by domain and by demographic group. Coverage measurement is a collection of techniques that measure the differences between census enumerations and the corresponding true counts for groups or areas. Coverage measurement is the quantitative aspect of coverage evaluation, which also encompasses more qualitative techniques, such as ethnographic observation. The differences between census counts and the corresponding true counts at the level of the indi­ vidual (or the household) are referred to collectively as census coverage errors, and in this chapter we categorize types of census coverage error and indicate methods that can be used for their summarization. We then detail the three primary (potential) uses of census coverage measure­ ment that rely on summarizations. Finally, we provide a brief overview of the methods that are currently used in the U.S. census for coverage measurement. 1

OCR for page 15
16 COVERAGE MEASUREMENT IN THE 2010 CENSUS TYPES OF CENSuS ERRORS There are two obvious ways in which the census count for an indi­ vidual can be in error: A person could be included in the census as an enu­ meration when he or she should have been omitted—an overcount—or the person could be omitted from the census when she or he should have been included—an undercount. In addition, since the primary applica­ tions of census counts are for apportionment of the states and redrawing of congressional districts, it is important that each individual be counted in their appropriate location. When a person is counted in other than the correct location, the effect of this error depends on both the distance between the recorded location and the true location and on the intended application of the counts (see below). Given that, we decided in this report to separately categorize under­ counts from overcounts, which are always errors regardless of the location of the enumeration, and those from enumeration errors that result from counts in the wrong location. This approach is not due to any sense that the latter errors are less important, but that they have different causes and therefore different solutions, and second that they are of different types as a result of the various degrees of displacement. This classification of census coverage error differs from the classi­ fication that has been typical up now. In that classification scheme, an overcount was any erroneously included enumeration, which included enumerations that were in the wrong location, regardless of whether the error was a few blocks or hundreds of miles. Similarly, an undercount was any erroneously omitted enumeration, which included enumerations that were in the census but were attributed to another (incorrect) location. As a result, in the previous scheme, an enumeration in the wrong location was represented as two errors: an overcount for the location that was recorded and an undercount at the correct location. The approach adopted here for classifying coverage error is consistent with a framework developed by Mulry and kostanich (2006), which is described in Chapter 5. We now provide more detail on the nature and causes of these various types of census coverage error. undercounts Omissions result from a missed address on the decennial census’ Master Address File (MAF), a missed housing unit in a multiunit resi­ dence in which other residences were enumerated, a missed individual in a household with other enumerated people, or people missed due to having no usual residence.

OCR for page 15
1 FUNDAMENTALS OF COVERAGE MEASUREMENT Overcounts Overcounts result from including enumerations that should not have been included in the census and from counting people more than once. Enumerations that should not have been included in the census are for people who were not residents of the United States on Census Day, and includes those born after census day and those who died prior to census day; people in the United States temporarily; and enumerations of ficti­ tious people. As explained above, we restrict the term “erroneous enu­ merations” to those enumerations that should not have been included in the census anywhere at all, thereby excluding duplicates and those counted in the wrong location. Duplicates Duplicates can result from: (1) repeat enumerations of a subset of the individuals from a household, sometimes as a result of the multiple opportunities for being enumerated in the census; (2) an address being represented in more than one way on the MAF, resulting in the dupli­ cation of all residents; and (3) the inclusion of a person at two distinct residences, possibly both of which are part­time residences or because of a move shortly before or shortly after Census Day. Counts in the Wrong Location The two fundamental types of census coverage error, overcounts and omissions (undercounts), reduce the accuracy of the total count for the people in any geographic area that contains or should contain the indi­ vidual counted in error. In addition, as mentioned above, there can also be errors in the geographic location of an individual or an entire household, which can also impact the accuracy of census counts. Counting a person in the wrong location can result from a mis­ understanding of the census residence rules and the resulting reporting of someone in the wrong residence. This can result from having an enumera­ tor assign a person to the wrong choice from among several part­time residences or from the Census Bureau’s placing an address in the wrong census geographic location (called a geocoding error). Placing a person in the wrong geographic area will lower the count for the correct geographic area and raise it for the incorrectly designated area. Therefore, whether there is an effect on census accuracy depends on the distance between the correct and incorrect locations and on the summary tabulation in ques­ tion: the more detailed the tabulation is with respect to geography, or the greater the displacement, the greater the chance that geographic errors will affect the quality of the associated counts. Placing a person in the

OCR for page 15
1 COVERAGE MEASUREMENT IN THE 2010 CENSUS wrong location can therefore result in zero additional errors or two addi­ tional errors. (One additional error is also possible, by placing a duplicate enumeration in the wrong location.) Demographic Errors A similar outcome will occur when a person’s demographic charac­ teristics are recorded in error. This happens when a person is assigned to the wrong demographic group through a reporting error or through use of imputation of an individual’s demographic characteristics when those characteristics are not provided by the respondent. Again, placing a per­ son in the wrong demographic group will lower the count for the correct demographic group and raise it for the incorrectly designated demo­ graphic group. Whether this error has an effect on the decennial census counts depends on the aggregate of interest: as above, the more detailed the tabulation demographically, the greater the chance that demographic errors will affect the quality of the associated counts. Imputations In addition to census coverage errors that result from the data col­ lected in the census, there are also enumeration errors that result from the methods, typically imputation, that are used to address census non­ response. As mentioned in National Research Council (2004a), in addition to item imputation (which is used to address missing characteristics for so­called data­defined enumerations), there are five different degrees of “missingness” for the residents of a housing unit that can result in five different types of whole person or whole household imputation (count imputation). Imputation methods used to address whole household non­ response will often result in counts for a housing unit that do not agree with the true number of residents of that housing unit, and these dif­ ferences contribute to coverage error. However, we assert that the dis­ crepancies that result from the application of an imputation technique are not errors of either omission or overcoverage and therefore should not contribute to assessments of the magnitudes of the components of census coverage error. Whole household imputation is simply a means for producing counts that are as accurate as possible when aggregated for various domains of interest.1 Thus, the effectiveness of an imputation algorithm should be assessed by its aggregate performance (e.g., bias, variance, mean­square error for domains of interest) and should not be 1 We use the term “domain” to refer to any demographic or geographic aggregate of interest.

OCR for page 15
1 FUNDAMENTALS OF COVERAGE MEASUREMENT considered as correct or incorrect at the level of the household. To sum up then, there are four basic types or components of census coverage error: omissions, duplicates, erroneous enumerations, and enumerations in the wrong location. COvERAgE ERROR METRICS FOR AggREgATES Since census coverage errors can be positive (overcounts) or nega­ tive (undercounts), they can partially cancel each other out when census counts are aggregated over a domain. Specifically, the difference between the census count and the true count for a domain is equal to the number of overcounts minus the number of undercounts, plus the net from enu­ merations in the wrong location for the residents of the housing units in that domain. The net coverage error or the net undercount, defined as the difference between the census count and the true count for a domain, is therefore a useful assessment of the effect of census coverage error on an aggregate of interest. Net coverage error has two benefits: (1) it directly assesses the utility of census counts for aggregates of interest, and (2) it can be compared with previously published estimates of net coverage error for historical com­ parisons of census quality. Percent net undercount expresses net coverage error as a percentage of the true count and therefore facilitates comparison of the net coverage error between domains. Differential net undercount, the difference between the percentage net undercount for a specific domain and the percentage net undercount for another domain (or for the nation), is therefore a useful measure of the degree to which one domain is (net) undercounted relative to another. To be precise, let Ci be the census count for the ith domain, and let C+ be the census national total. Similarly, let Ti and T+ be, respectively, the true count for the ith domain and the true national total. Then the differential net undercount is Ci − Ti C+ − T+ Ci C+ − =− . Ti T+ Ti T+ Many uses of census data (e.g., apportionment and fund allocation) depend on census counts as proportional shares of the population, rather than as population counts, and in those situations a measure of the quality of the counts for a domain of interest is Ci Ti −. C+ T+ For comparison of the quality of two sets of estimated counts used as counts, a common yardstick is the sum of squared net errors over domains. When comparing the quality of two sets of estimated counts

OCR for page 15
20 COVERAGE MEASUREMENT IN THE 2010 CENSUS used as population shares, the error of the shares is again commonly sum­ marized, with errors as the difference between population shares and true shares, by adding squared errors over domains, but now weighted by the population size (Ti ), since otherwise one is equating a given error in popula­ tion shares for a small and a large domain. Specifically, the following loss function would be reasonable to use: 2  Ci Ti  ∑  C − T  Ti .   i + + Although it is clearly very useful, net census error, or net undercount, is an inappropriate summary assessment of census coverage error when the objective is census improvement because a substantial number of overcounts and undercounts may cancel each other for a given domain, which may obscure problems with census processes. Also, while these errors may balance each other for a given domain for a given census, they may not balance to the same extent either in more detailed aggregates or in subsequent censuses. To address this possible imbalance, some have argued for tabulating census gross error, which is the sum of the number of errors, overcounts, undercounts, and errors in the wrong location, relative to domains of interest. However, there are two problems with gross error as a summary measure of the quality of the census enumeration process. First, as noted above, enumerations in the wrong location will only matter when the degree of displacement and the tabulation in question are such that the displacement places someone in the wrong tabulation cell. Therefore, enu­ merations in the wrong location should not be interpreted as equivalent to overcounts or omissions. Furthermore, census coverage errors, which we classify as erroneous enumerations, duplicates, omissions, and counts in the wrong place, all have somewhat different causes. Given the cur­ rent objective of supporting a feedback loop for census improvement, it is important to separate out the summaries of these various components so that their magnitudes can be assessed individually, rather than trying to aggregate them into a single error measure. Second, for counts in the wrong location, rather than a percentage error measure—which is an appropriate summary measure for omissions, erroneous enumerations, and duplications—a more useful summary assessment would provide a representation of the frequency of enumerations in the wrong location as a function of some representation of the degree of the displacement (so that location error rates would diminish as the displacement increases). This approach would facilitate the assessment of the degree to which errors from enumerations in the wrong place effect various applications of the counts.

OCR for page 15
21 FUNDAMENTALS OF COVERAGE MEASUREMENT The term components of (census coverage) error communicates this idea of separating out the enumeration errors into these categories of duplica­ tions, erroneous enumerations, omissions, and geographic errors so that their individual causes can be better analyzed. For sake of completeness, we again mention that there are also errors in counts that are attributable to errors in a person’s demographic charac­ teristics, and there can, of course, also be errors in a person’s other char­ acteristics, for example, whether the residents own or rent their housing units. These are ignored in this discussion, though errors in characteristics used to model net coverage error can negatively affect its estimation, and it is therefore important to reduce the frequency of such errors. Whether one uses net coverage error or rates of components of census coverage error to represent the quality of the census counts for a domain clearly depends on the analysis that one has in mind. To support as much flexibility in summarization and analysis as possible, information on cen­ sus coverage error needs to be retained at as basic a level as possible, in addition to the summary tabulations that the Census Bureau provides. This retention would have two advantages. First, it would permit a more precise assessment of the effect of census errors on any specific applica­ tion of the counts. For instance, one could assess the impact of omissions (ignoring the extent to which they are offset by overcoverage errors) on a specific domain of interest that is not provided in the standard Census Bureau tabulations from the coverage measurement program. Second, and more importantly, retention of information on census coverage error at the level of the individual allows for the examination of (causal) asso­ ciations using statistical models that relate whether a coverage error was or was not made as a function of the census enumeration processes used and individual and housing unit characteristics. Such an analysis could also include correlates of whole­household omissions, correlates of omis­ sion errors that only affected some residents of a household, correlates of whole­household duplications, correlates of partial­household duplica­ tions, or correlates of the coverage error of counting individuals in the wrong place (for various degrees of misplacement). In sum, there are various components of census error that have vari­ ous applications, and there is therefore a need for access to those errors at an individual level and to link those errors to potential causal factors to support various descriptive and analytic needs. By “descriptive” we mean summary assessments of the quality of census counts for domains; by “analytic” we mean the development of statistical models that attempt to discriminate between individuals and households that are and are not counted in error.

OCR for page 15
22 COVERAGE MEASUREMENT IN THE 2010 CENSUS PuRPOSES Coverage measurement has historically served multiple purposes. Since its earliest inception in the 1950 census, it has had the goal of evaluating the accuracy of census counts for geographic and demographic domains, with a focus on assessing net error for domains. The primary goal was to inform users as to the quality of the census counts for various applications. In addition, but to a much lesser extent, coverage measure­ ment has also been used to provide information relevant to developing a better understanding of census process inadequacies, leading to improve­ ments in design for the subsequent census. The estimation of net error has also raised the possibility of provid­ ing alternative counts for use in formal applications, known as census adjustment. We know of only one formal use of adjusted census counts to date, namely, the use of adjusted counts to modify population controls used for the Current Population Survey (CPS), the National Health Inter­ view Survey, the National Crime Victimization Survey, and the Survey of Income and Program Participation during the 1990s, which in turn affected the estimate of the number of people unemployed during the 1990–2000 intercensal period by the Bureau of Labor Statistics. However, the primary focus of coverage measurement in both 1990 and in 2000 was to produce adjusted census counts for official purposes, assuming that it could be demonstrated that the adjusted counts would be preferred to the unadjusted census counts for apportionment and redistricting. The stated Census Bureau plan that the primary purpose of the cover­ age measurement program in 2010 would be to measure the components of census coverage error in order to initiate a feedback loop for census process improvement is a substantial innovation. An interesting ques­ tion is the extent to which a coverage measurement program can be used for this purpose, and a major charge to this panel was to determine the extent to which this new focus of coverage measurement should affect the design of the coverage measurement program and the resulting output and analyses. Evaluation of the Accuracy of the Census Counts Census counts serve a variety of important purposes for the nation, including apportionment, legislative redistricting, fund allocation, gov­ ernmental planning, and support of many private uses, such as business planning. Users of census data need to know how accurate the counts are in order to determine how well they can support these various applica­ tions. The needed information includes an understanding of the extent to which the accuracy of census counts differ by location or by demographic

OCR for page 15
2 FUNDAMENTALS OF COVERAGE MEASUREMENT group and the extent to which accuracy has improved from one census to the next. The total population count of the United States is probably the most visible output of a census, so one obvious measure of coverage accu­ racy for the census is the error in the count for the entire United States over all demographic groups. However, essentially all applications of the census—e.g., redistricting and local planning—use population counts at various levels of geographic and demographic detail. Consequently, it is important to assess the rates of net undercoverage by various geographic or demographic domains. Historically, a key issue has been, and remains, the differential net undercount of blacks, Hispanics, and Native Americans, which has resulted in the repeated underrepresentation of areas in which those groups make up a large fraction of the residents. In particular, the dif­ ferential net undercount of these groups has led to their receiving less than their share of federal funds and political representation (see, e.g., Ericksen et al., 1991, for more details). Given this, it is as important as ever for the Census Bureau, in evaluating possible alternative designs for the decennial census, to not only assess the likely impacts on the frequency of components of census coverage error, but also to assess the impacts on differential net coverage error for historically undercounted minority groups. Census Adjustment The 1999 Supreme Court decision (Department of Commerce v. United States House of Representatives, 525 U.S. 316) precluded the use of adjust­ ment based on a sample survey for congressional apportionment. In addition, the Census Bureau concluded that time constraints currently preclude the computation and evaluation of adjusted counts (based on a postenumeration survey) by April 1 the year after a census year, therefore preventing the use of adjusted counts for purposes of redistricting (see National Research Council, 2004a:267). Furthermore, the current approach to adjustment has a number of complications that continue to present a challenge to the production of high­quality estimated counts, including the quality of the data for movers (often missing or collected by proxy), matching errors, the treatment of missing data for nonmovers, the estimation of the number missed by both the census and the postenumeration survey, and the heterogeneity remaining after the use of poststratification of the match rate and the cor­ rect enumeration rate (resulting in correlation bias). This last objection will be reduced, but not eliminated, with the likely shift to the use of logis­ tic regression instead of poststratification in 2010 (discussed below).

OCR for page 15
2 COVERAGE MEASUREMENT IN THE 2010 CENSUS In addition, the use of adjustment is complicated since for some important applications one needs adjusted counts at low levels of demo­ graphic and geographic aggregation, and a sample survey, by design, is intended to make estimates at more aggregate levels. A decision whether to use adjusted counts for any purpose must therefore rest on an assess­ ment of the relative accuracy of the adjusted counts compared with the census counts at the needed level of geographic or demographic aggrega­ tion. One key issue that depends on the application is whether to base this assessment on population shares or population magnitudes. The Census Bureau’s decision not to adjust the redistricting data for the 2010 census, due for release by April 1, 2011, was based on the difficulty of making this assessment within the required time frame. Census Process Improvement Although it is important to assess census coverage, it would also be extremely helpful to use that assessment to improve the quality of sub­ sequent censuses. Consequently, an important use of coverage measure­ ment is to help to identify important sources of census coverage errors and possibly to suggest alternative processes to reduce the frequency of those errors in the future. Although drawing a link between census coverage errors and deficient census processes is a challenging task, the Census Bureau thinks that substantial progress can be made in this direc­ tion. Therefore, the 2010 coverage measurement program has the goal of identifying the sources of frequent coverage error in the census counts. This information can then be used to allocate resources toward develop­ ing alternative census designs and processes that will provide counts with higher quality in 2020. It is conceivable that use of such a feedback loop could also provide substantial savings in census costs, in addition to improvement in census quality because the tradeoff between the effect on accuracy and on census process costs might now be better understood. The panel fully supports this modification of the objectives of coverage measurement in 2010. To see the value of this shift in the objective of coverage measurement, consider, for example, the findings from demographic analysis for the 2000 census, which showed that there was a substantial undercount of young children relative to older children. Specifically, Table 2­1 shows the net undercount rates—(DSE – C)/DSE, where DSE indicates the adjusted count, and C indicates the corresponding census count)—by demographic group in 2000 on the basis of the revised demographic analysis estimates (March 2001) (see National Research Council, 2004b:Chapter 6): One hypothesis is that the undercoverage for children aged 10 and under was at least in part due to the imputation of age for those left off the census

OCR for page 15
2 FUNDAMENTALS OF COVERAGE MEASUREMENT TABLE 2-1 Net Undercount in 2000 Age Group Demographic Group 0–10 10–17 Black male 3.26 –1.88 Black female 3.60 –1.20 Nonblack male 2.18 –2.01 Nonblack female 2.59 –1.55 NOTE: The undercount is as measured by demographic analysis. SOURCE: Data from National Research Council (2004b:Chapter 6). form in households exceeding six members; this hypothesis is examined in keller (2006). The 2000 census forms only collected characteristics data for up to six household members. For households that reported more than six members, characteristics data for the additional members either were collected by phone interview (for households that provided a telephone number) or were imputed on the basis of the characteristics of other house­ hold members and the responses for other households. The hypothesis is that these imputations systematically underrepresented young children since they were underrepresented in the pool of “donor” households. 2 Although demographic analysis can measure the net undercoverage of these groups, it cannot currently shed further light on the validity of this hypothesis. Data from a postenumeration survey might be useful in this regard, because characteristics data are collected for most of the respon­ dents of the postenumeration survey, and those data would likely allow an assessment of the extent to which imputations in large households dis­ torted the age distribution. Potential alternatives that could be considered for the 2010 census include changes to the collection of data for members of large households and improved imputation techniques. The panel is optimistic that the use of coverage measurement can strongly support the improvement of census methods, but the operation of this feedback loop will not be straightforward. Coverage measurement results will sometimes provide strong indications of the likely source of some errors; for other errors, the source will often remain unclear. An example of the former is for people aged 18–21, who have a duplication rate that is extremely high: One might surmise that it is at least partly 2 We note that even if it were determined that increasing this limit from six to seven would reduce the rate of omission of young children in large households, other considerations involving the rate of nonresponse and the quality of the collected information would have to be evaluated before making such a change.

OCR for page 15
 COVERAGE MEASUREMENT IN THE 2010 CENSUS which were similar to those used in the 1970 census. However, the rise in the population of undocumented aliens reduced the quality of the estimates from demographic analysis in comparison with those in 1970. Initial demographic analysis­based estimates of the net national under­ count were very close to 0 percent. However, there is evidence that the initial demographic analysis estimates failed to account for a substan­ tial increase in the undocumented immigrant population. Subsequent analysis suggests that the undercoverage of the census was as much as 1.5 percent given reasonable assumptions about the size of the uncounted undocumented population.15 1990 Census Initial plans for the 1990 coverage measurement program were for a PES of 300,000 housing units, which the Census Bureau argued was needed to support net undercoverage estimates (and potentially adjust­ ment) at the level of geographic aggregation consistent with such uses as reapportionment and redistricting. However, this design was rejected by the Secretary of Commerce and was replaced by a PES of 150,000 house­ holds, which was only to be used for purposes of coverage measurement and not for census adjustment. That decision precipitated a lawsuit that maintained the size of the PES at 150,000 households (ultimately, 165,000) but reopened the possibility of using the PES to adjust the 1990 census: the decision on adjustment, to be made by the Secretary of Commerce, would benefit from the deliberations of the members of a Special Secre­ tarial Advisory Panel. The 1990 PES was the first postenumeration survey with a survey instrument specifically designed for coverage measurement. The final design included more than 5,000 block clusters that were independently listed. (The design included people living in many types of group quarters residences.) The design also included a considerable amount of over­ sampling of blocks that contained a large fraction of historically hard­to­ enumerate people. The P­sample residents were interviewed and matched to the E­sample, the census enumerations in the PES blocks, first using computer matching software, with difficult cases then subject to cleri­ cal review. Unmatched cases were followed up in the field. Given the development of a specific survey instrument, and due to better efforts to collect data, the percentage of nonrespondents and cases with unresolved match status was substantially lower than in 1980. Dual­systems estimates were constructed by separately computing net undercoverage estimates for 1,392 poststrata. Due to their high variance, these estimates were 15 Robinson and West (2000) give the estimate of 1.2 percent.

OCR for page 15
 FUNDAMENTALS OF COVERAGE MEASUREMENT smoothed using empirical Bayes regression methods. Finally, the esti­ mates were carried down to the level of census blocks using synthetic estimation. The problem of estimating both the number of undocumented aliens resident in the United States on census day and the percentage of those that were enumerated in the census posed more of a challenge to demographic analysis in 1990 than in 1980, given the larger size of the undocumented population in 1990. A “residual” process was developed to address this problem. The basic idea is as follows. An estimate of the number of legal foreign­born residents was developed using reported data on legal immi­ gration. This figure was subtracted, at the national level, from the esti­ mated number of foreign­born residents, either from the census long­form sample or from the CPS (and now, the American Community Survey), to arrive at an initial estimate of the number of illegal immigrants. This esti­ mate was then inflated to account for undercoverage of this population (for details, see Robinson, 2001). Clearly, this required a few assumptions that were unlikely to hold, even approximately. However, there was, and still is, no preferred alternative methodology. The Census Bureau released preliminary PES results in April 1991, estimating a national net undercount of 2.1 percent, with a difference of 3.1 percent between the rate of undercoverage of blacks and nonblacks. An internal Census Bureau group voted seven to two in favor of adjust­ ing the 1990 census to remedy differential undercoverage. The Special Secretarial Advisory Panel split equally on the decision on adjustment. Ultimately, the Secretary of Commerce decided not to adjust the 1990 census. 2000 CENSuS16 Background The initial planning for the 2000 census coverage measurement pro­ gram was based on two assumptions: (1) that coverage improvement programs were unlikely to greatly reduce black–nonblack differential net undercoverage, and therefore the 2000 census was likely to continue an historical pattern of substantial differential net undercoverage of minori­ ties, and (2) the time necessary to compute PES­based adjusted counts and validate them would make it extremely difficult to deliver adjusted counts in time for apportionment or redistricting of the U.S. House of Representatives unless substantial changes were made to the design of the census. Consequently, the Census Bureau initially decided to use 16 Much of the material in this section is taken from National Research Council (2004b).

OCR for page 15
6 COVERAGE MEASUREMENT IN THE 2010 CENSUS for its coverage measurement program in 2000 a strategy referred to as integrated coverage measurement. The idea of integrated coverage mea­ surement was to limit the more extreme efforts at coverage improvement that delayed the PES data collection and to rely on the PES to address some of the coverage problems in the primary enumeration. The result­ ing PES counts, which would be the official census counts in a so­called “one­number census,” would be of higher quality than the unadjusted census counts. It was anticipated that the execution of the PES would greatly benefit from the experience gained in carrying out the 1980 and 1990 PESs, and from the earlier (and therefore higher quality) data col­ lection made possible by the elimination of late­stage coverage improve­ ment programs. The integrated coverage measurement design called for a 700,000 household survey and accompanying matching operation, which would provide reliable direct estimates for states. (Direct estimates refers to each state’s estimate of net undercoverage, which are based on the sample collected from households in that state alone.) The plan to use integrated coverage measurement was jettisoned after the Supreme Court decision in January 1999 (Department of Commerce v. United States House of Representatives, 525 U.S. 316), which prohibited the use of sampling methods to produce counts for purposes of apportion­ ment. This decision required the Census Bureau to greatly modify the design of the 2000 census as well as the associated coverage measurement program. With respect to the census itself, the Census Bureau was not allowed to use sampling for nonresponse follow­up as planned, since that would result in sample­based census counts. This change, in turn, affected PES plans for the 2000 census because sampling for nonresponse follow­up ruled out using a particular ver­ sion of PES, referred to as PES­B. In a PES­B, one attempts to enumerate in­movers in the P­sample blocks, assuming that their size and charac­ teristics are roughly equivalent to those of the out­mover population. To determine whether those individuals were enumerated in the census, one determines the address at which they resided on census day, and then checks the census records at those locations to see if there is a match. However, with a census using sampling for nonresponse follow­up, many of those locations would not be included in the sample follow­up, or they would not have a census enumeration status, greatly complicating the use of PES­B. For that reason, PES­C was created, which again uses the size of the in­mover population to estimate the size of the out­mover population, as in PES­B, but instead uses the match status of the out­mover popula­ tion. Because of the difficulty locating and contacting the out­movers, match status is often based on proxy information of dubious quality. For that reason, it is generally believed that PES­C is inferior to PES­B. How­ ever, when the design for the 2000 Census was revised as a result of the

OCR for page 15
 FUNDAMENTALS OF COVERAGE MEASUREMENT Supreme Court decision of 1999, the Census Bureau did not have time to move back to a PES­B strategy. The Accuracy and Coverage Measurement Program The modified coverage measurement program, referred to as Accuracy and Coverage Evaluation Program (A.C.E.), was scaled back to a 300,000­ household PES. Given the early work already carried out in selecting the integrated coverage measurement sample, the reduced A.C.E. sample was selected by subsampling from that sample. The smaller sample size for the A.C.E. could be justified since there was no longer a need to produce direct state estimates to support apportionment and so estimates from A.C.E. could borrow information across state boundaries. In addition, it is important to note that adjusted counts were not needed now until April 1, 2001, in support of redistricting, which provided additional time for validating the A.C.E. estimates. As noted above, the 2000 A.C.E. sample was twice as large as the PES in 1990, with a P­sample of 300,000 households in 11,000 block clusters. Given the larger sample, there was less need to oversample, and the resulting variances of net coverage estimates were very likely substantially reduced in comparison with those for 1990. A.C.E. also used computer­assisted telephone and personal interviewing to facili­ tate the collection of P­sample data. A.C.E.’s dual­systems estimates used 448 poststrata (which were later collapsed to 416). This number, and the larger sample size, reduced the need, relative to 1990, for empiri­ cal Bayes smoothing. In light of the possibility of adjustment (before the Supreme Court decision), the idea was that if A.C.E. could be demonstrated to provide valid estimates of net coverage error for poststrata and if the estimated net error differed appreciably by poststrata, then adjusted population counts from A.C.E. should be used for redistricting and for other official purposes. In practice, however, the Census Bureau’s evaluations of A.C.E. discovered several problems. One problem was that A.C.E. estimated an overall net undercount of 1.2 percent while the initial demographic analysis estimated that the census had a 0.7 percent net overcount. Later revision of the demographic analysis estimates resulted in the estimate of a 0.3 percent net undercount, but this was still inconsistent with the estimate from A.C.E. Other problems with A.C.E. concerned balancing error, the uncertain effects of a substantial number of late additions to the census, the level of error from synthetic estimation, the relative lack of duplicates identi­ fied by A.C.E., and the validity of whole­household imputations. These problems collectively led to a recommendation by the Census Bureau,

OCR for page 15
 COVERAGE MEASUREMENT IN THE 2010 CENSUS seconded by the Secretary of Commerce on March 6, 2001, not to use A.C.E. counts for redistricting. Post-2000 Research A.C.E. Revision I After the 2000 census, the Census Bureau carried out research to investigate the sources and magnitudes of error in the 2000 census, A.C.E., and demographic analysis. As part of this effort, the Census Bureau carried out two studies to collect additional information rel­ evant to specific concerns regarding A.C.E. The purpose of the work was to determine the extent of person duplication, and, on a sample basis, to identify the correct residence for E­sample enumerations and to determine the correct match status for P­sample cases. The studies were the Evaluation Follow­Up Study, which involved reinterviewing a subsample of 70,000 people in E­sample housing units in 20 percent of the A.C.E. block clusters to determine correct residences on Census Day (with additional clerical review of 17,500 people who were unresolved) and the Person Duplication Studies, which involved nationwide com­ puter matching of E­sample records to census enumerations using name and date of birth. This nationwide search permitted the first determina­ tion of the extent of remote duplication in the census, that is, cases in which the duplicated individuals did not both reside in the PES block cluster. The Census Bureau also examined the implementation of the targeted extended search (searches outside of the relevant P­sample block cluster for a match for situations in which there was a likely error in identifying the correct block) for matches to P­sample cases; estima­ tion of the match rate and the correct enumeration rate for people who moved during the data collection for A.C.E.; and the effects of census imputations. As a result of these very detailed investigations, the Census Bureau judged that A.C.E. counts substantially underestimated the rate of census erroneous enumerations and hence tended to overestimate the true population size.17 The Census Bureau subsequently released revised A.C.E. estimated counts on October 17, 2001, which were referred to as A.C.E. Revision I counts. Given concern stemming from the finding that there were considerably more errors of duplication than were originally estimated by the A.C.E., the Census Bureau again recommended that 17 Fenstermaker and Mule (2002) estimated that there were a total of 5.8 million dupli­ cates in the 2000 census. The problem could have been even larger, but the Census Bureau mounted an ad hoc operation early in 2000 to identify duplicate MAF addresses and associ­ ated returns, which removed 3.6 million people from the 2000 census.

OCR for page 15
 FUNDAMENTALS OF COVERAGE MEASUREMENT these adjusted counts not be used, this time for the allocation of federal funds or other official purposes. A.C.E. Revision II Between October 2001 and March 2003, the Census Bureau undertook a further review of all the data collected in the census and the A.C.E. In addition, based on a comparison of the sex ratios from demographic analysis to those from the A.C.E., the Census Bureau decided to revise (increase) the A.C.E. estimates for males so that the resulting sex ratios were consistent with the sex ratios for demographic analysis for blacks 18 years old and older and for all other males more than 30 years old. This revision was based on the Census Bureau’s belief that the A.C.E. counts had been reduced by correlation bias. The adjustment that was imple­ mented is implicitly based on the assumption that the correlation bias for the parallel female groups was close to zero. The estimates based on these revisions are referred to as A.C.E. Revision II. Evaluations of the quality of these final A.C.E. estimates resulted in the announcement, on March 12, 2003, by the Census Bureau that the A.C.E. Revision II counts would not be used as the base for producing intercensal population estimates. The Panel to Review the 2000 Census (National Research Council, 2004b) generally agreed with the decisions made at each stage of this three­stage process, namely: not to use the A.C.E. counts—either the origi­ nal, Revision I, or Revision II—for purposes of redistricting, fund alloca­ tion or other official purposes or for intercensal estimation. However, the panel was not in complete agreement with the supporting arguments of the Census Bureau.18 As a by­product of this intensive effort to understand whether adjusted counts were preferable to unadjusted counts for various purposes, the Census Bureau produced comprehensive documentation and evaluation of the A.C.E. processes. A considerable amount of material is available at the following locations. Evaluations supporting the March 2001 decision can be found at http://www.census.gov/dmd/www/EscapRep.html; evalu­ ations supporting the October 2001 decision can be found at http://www. census.gov/dmd/www/EscapRep2.html; and evaluations supporting the March 2003 decision can be found out at http://www.census.gov/dmd/ www/ace2.html. Collectively, these reports document the A.C.E. proce­ dures in detail, examining what was learned about the quality of A.C.E. and A.C.E. Revisions I and II through the additional information collected. 18 Forthe panel’s arguments, a more detailed description of A.C.E. and the various evalu­ ation studies, and the material on which this abbreviated history is based, see National Research Council (2004b:Chapters 5–6).

OCR for page 15
0 COVERAGE MEASUREMENT IN THE 2010 CENSUS Limitations of A.C.E. The A.C.E. in the 2000 census was planned from the outset as a method for adjusting census counts for net coverage error so it did not focus on estimating the number or frequency of the various components of census coverage error. As an important example, the limited geographic search for matches in the A.C.E. (for estimation of net coverage error) relied on the balancing of some erroneous enumerations with omissions, in which the erroneous enumerations were at times valid E­sample enumerations but in the wrong location. People can be counted in the wrong location as a result of a geocoding error (placing an address in the wrong census geography) or enumerating a person at a second home. Because such erroneous enumerations and omissions were expected to balance each other on average, they were expected to have little effect on the measure­ ment of net coverage error. Therefore, the A.C.E. did not allocate the additional resources that would have been required to distinguish these situations from enumerations in the wrong place. Similarly, the A.C.E. did not always distinguish between an erroneous enumeration and counting a duplicate enumeration at the wrong location. In addition, the A.C.E. effectively treated all cases with insufficient information for matching as imputations although it is clear that a sub­ stantial fraction of them are correct. The Census Bureau has done research that demonstrates that for a substantial subset of the cases match status can be reliably assessed. (See Chapter 4 for the discussion on missing data methods for details on this research.) We now mention several other limitations of the A.C.E. in 2000 for measuring census component coverage error, along with any plans to address the limitation in the designs for the census and census coverage measurement in 2010. However, we must stress that our treatment here of the theoretical underpinnings of A.C.E. is incomplete. For a complete description of this, see Hogan (2003). First, inadequate information was collected in the census on census day residence. In 2000, comprehensive information was not collected from a household in the census interview regarding other residences that residents of a household often used or on other people who occa­ sionally stayed at the household in question. This limited the Census Bureau’s ability to correctly assess residency status for many people. The Census Bureau intends to include more probes to assess residence status in the 2010 census questionnaire, and the coverage follow­up interview will also collect additional information on residence status for those housing units that are likely to have incorrectly represented the number of residents on the census form. In addition, more probes about residence will also be included on the 2010 census coverage measure­ ment questionnaires.

OCR for page 15
1 FUNDAMENTALS OF COVERAGE MEASUREMENT In 2010, the duplicate search will be done nationwide, and not only for the PES population, to help determine census day residence. In con­ junction with search, as part of the coverage follow­up interview, the Census Bureau plans on incorporating a real­time field verification of duplicate enumerations in 2010. (For details on issues in determining correct residence, see U.S. Census Bureau, 2003.) Second, nonresponse in the E­ and P­samples complicated matching of the P­sample to the E­sample and of the E­sample to the census (to identify duplicates). It also complicated estimation because it interfered with assigning a person to the correct poststratum (for details, see Mulry, 2002). Third, the use of the methodology for individuals who moved between Census Day and the day of the postenumeration interview (known as PES­C; see above) resulted in a large percentage of proxy enumerations, which in turn resulted in matching errors. The Census Bureau is planning on returning in 2010 to the use of PES­B (similar to the 1990 methodol­ ogy), which relies completely on information from in­movers. Fourth, the A.C.E. Revision II estimates refined undercoverage esti­ mates for black men over 18 and for “all other men” over 30 using sex ratios from demographic analysis (ratios of the number of women to the number of men for a demographic group) to correct for correlation bias (for details, see Bell, 2001; Shores, 2002). This method assumes that the net coverage error for women for the relevant demographic group is ignor­ ably small. However, for nonblack Hispanics, refinement (other than that used for all nonblack males) using sex ratios would require a long his­ torical series of Hispanic births and deaths and, more importantly, highly accurate data on the magnitude and sex composition of immigration (both legal and undocumented). Yet the historical birth and death data for Hispanics are available only since the 1980s, and the available measures of immigration are too imprecise for this application. Consequently, this use of demographic analysis to refine A.C.E. estimates was not directly applicable to nonblack Hispanic males in 2000.19 In addition, there is a great deal of uncertainty about the degree to which various assumptions need to obtain to support the use of this methodology for either blacks or nonblack Hispanics. (The decision could also depend on the yardstick in question, i.e., whether counts or shares are the quantities of interest.) Fifth, poststratification is used to reduce correlation bias since it parti­ tions the U.S. population into more homogeneous groups regarding their enumeration propensities. The number of factors that could be included in 19 Forexample, it is noteworthy that about 55 percent of working­age (18–64) Hispanics are foreign born in comparison with less than 5 percent of whites and slightly more than 5 percent of blacks.

OCR for page 15
2 COVERAGE MEASUREMENT IN THE 2010 CENSUS the poststratification used in the A.C.E. was limited because the approach (essentially) fully cross­classified many of the defining factors, with the result that each additional factor greatly reduced the sample size per post­ stratum; for details of the 2000 poststrata, see U.S. Census Bureau (2003). The 2010 plan is to use logistic regression modeling to reflect the influence of many factors on coverage rates. Sixth, small­area variations in census coverage error that are not cor­ rected by application of the poststratum adjustment factors to produce estimates for subnational domains (referred to as synthetic estimation) were not reflected in the variance estimates of adjusted census counts. The Census Bureau is examining the use of random effects in their adjustment models to account for the residual variation in small­area coverage rates beyond that which is modeled through synthetic estimation. Finally, the sample design made the 2000 A.C.E. less informative than it might have been in measuring components of census coverage error by not providing a greater sample targeted on people and housing units that are difficult to enumerate. As noted above, this approach was under­ standable given the focus of the A.C.E. on producing adjusted census counts. However, given the new priority of measuring the components of census coverage error, a number of design and data collection decisions in the general framework of PES data collection, especially sample design, remain open to modification. The Role of Demographic Analysis From 1950 through 1980, demographic analysis served as the primary source of estimates of national net coverage error in the decennial census. However, demographic analysis is fundamentally limited: It can only provide estimates of net coverage error for some national demographic groups. Demographic analysis cannot yet provide estimates for any geo­ graphic region below the national level, and it does not provide estimates of net undercoverage for the Hispanic population. Because of these limita­ tions, DSE, based on a postenumeration survey, displaced demographic analysis as the primary coverage measurement instrument, starting with the 1990 census. However, its limitations do not mean that demographic analysis is no longer useful. As was evident in 2000, demographic analysis still performs two useful functions. First, it provides estimates of population size that can be used in a variety of ways to assess the quality of dual­systems esti­ mates, including comparison of dual­systems estimates to demographic analysis estimates that are less affected by the quality of the data on external migration and sex and through age ratios. Demographic analysis played this important role in the 2000 census. Second, sex ratios and other

OCR for page 15
 FUNDAMENTALS OF COVERAGE MEASUREMENT TABLE 2-3 Net Undercoverage for U.S. Censuses, 1940–2000 1990a 2000b Group 1940 1950 1960 1970 1980 U.S. Total Demographic analysis 5.4 4.1 3.1 2.7 1.2 1.8 0.1 PES — — 1.9 — 0.8–1.4 1.6 –.5 Black Demographic analysis 8.4 7.5 6.6 6.5 4.5 5.7 2.8 PES — — — 5.2–6.7 4.6 1.8 Nonblack Demographic analysis 5.0 3.8 2.7 2.2 0.8 1.3 –0.3 PES — — — — — — — aRevised estimates. bRevised demographic analysis and final A.C.E. estimates. SOURCE: Data from Anderson (2000) and National Research Council (2004b). information from demographic analysis can be useful in improving the estimates from DSE. For example, the modification of the count for adult black men, mentioned above, remains a possibility for 2010. In addition, looking to the future, ongoing research is attempting to address the two primary deficiencies of demographic analysis—the lack of subnational estimates and the lack of estimates for the Hispanic population. To complete this history of coverage measurement, Table 2­3 shows the national estimates of net coverage error and the estimates disaggregated by black and nonblack for the decennial censuses from 1940 to 2000.

OCR for page 15