The 2000 decennial census provides data for many important purposes. Population counts and other census data are used to reapportion the U.S. House of Representatives; draw new boundaries for legislative districts; allocate billions of dollars in federal and state funds; support public and private sector planning, decision making, and research; undergird estimates from other government statistical programs; and serve as a valuable reference for the media and the general public. Census information is the product of a massive, complex, and costly set of operations.
Such an effort deserves a thorough assessment to understand how and why procedures worked and to understand the quality and limitations of the data for their intended uses. Accordingly, at the request of the U.S. Census Bureau, the Committee on National Statistics established the Panel to Review the 2000 Census to provide an independent review of statistical methods and other procedures in the census that may affect the completeness and quality of the data.
The 2000 census process included an Accuracy and Coverage Evaluation (A.C.E.) Program, in which the results of an independent sample survey of household residents were matched to census enumerations in a sample of small areas nationwide. The A.C.E. provides a basis for judging the net undercount in the census enumeration, as well as for statistical adjustment of census counts to reflect net undercount using the method of dual-systems estimation. In March 2001, the Census Bureau concluded that it could not resolve uncertainties in the census, the A.C.E. results, and separate population estimates from demographic analysis by a legally mandated deadline of April 1, 2001. Therefore, the Bureau recommended that census counts for congressional redistricting should not be statistically adjusted; this recommendation was subsequently approved by the U.S. Department of Commerce. In mid-October 2001, the Census Bureau is expected to decide whether to recommend that statistical adjustment be used on census data for allocation of state and federal funds and other purposes.
SCOPE OF REPORT
This interim report assesses the operations of the census and the A.C.E. on the basis of information available to the panel from the Census Bureau as of August 2001. The Bureau is expected to release the results of additional evaluations to accompany its second decision in mid-October, and it has planned a longer range evaluation program as well. We commend the Census Bureau for its openness in providing documentation and evaluation data, as they are completed, to our panel and other advisory and oversight groups.
The panel has considered statistical data on census operations, the A.C.E. Program, imputation for missing information on census questionnaires, late additions to the census counts, mail return rates, and demographic analysis. The information on these topics does not support definitive assessments, particularly of which census operations and design features had the greatest effects on the completeness and quality of the census information. We offer this report to provide preliminary assessments and suggestions for next steps.
This interim report makes no judgment on the appropriateness of adjusting or not adjusting the census for net population undercount. The panel in its November 2000 letter report concluded that the Census Bureau’s plans for evaluations that could be completed within the time available appeared to be sufficient for making a reasonably confident decision about adjustment in March 2001. However, the panel noted that whether the evaluations would permit such a decision would not become clear until the results were known. The evaluations were carried out as planned, and the Bureau determined that the results were inconclusive about important aspects of the A.C.E. and insufficient to resolve differences among the census estimate, A.C.E., and demographic analysis. The panel concludes that the Census Bureau followed its specified process and, thus, that its recommendation to release the counts from the census enumeration for redistricting was justifiable.
CENSUS AND A.C.E. OPERATIONS
We conclude that the 2000 census was well executed in many respects, particularly given the difficulties of last-minute changes in the overall design and other problems encountered prior to 2000. Ample funding enabled the Bureau staff to carry out the census on schedule.
We examined six major innovations in the census, four of which appeared to be successful: (1) contracting for data operations and using improved technology for capturing the data on the questionnaires; (2) use of a redesigned questionnaire and mailing strategy to encourage response; (3) paid advertising and
expanded outreach; and (4) aggressive recruitment of enumerators for followup of nonresponding households. The effectiveness of a fifth innovation—greater reliance on computers for data editing and imputation for inconsistent and missing responses—cannot be determined until information is available with which to evaluate the effects on data quality.
The sixth innovation was to develop the Master Address File (MAF) —a listing of all potential residences in the country—from new and multiple sources, including the U.S. Postal Service’s Delivery Sequence File and interaction with local and tribal governments. Although the concept behind this MAF-building process made sense, there were problems in execution that may have increased duplicate and other erroneous enumerations. An ad hoc operation had to be mounted in summer 2000 to reduce duplication in the census from the MAF. Further evaluation is required to assess the success of that operation and the quality of the overall MAF.
Mail Response Rates
The mail response rate in the 2000 census—the percentage of all addresses from which a census questionnaire was received—held steady at the 1990 level. This was an important achievement in light of the decline in the response rate over several past censuses and given the expense and time required to visit addresses lacking a completed questionnaire. The mail return rate, a more refined indicator that focuses on occupied households by excluding vacant and nonresidential addresses, was very close to the 1990 rate. There was a marked decline in the return rate for recipients of the long-form questionnaire, which could have adverse affects on the quality of the long-form data. Mail return patterns were similar between 1990 and 2000: most areas that were hard to count and had low mail return rates in 1990 had low mail return rates in 2000.
The A.C.E. is, like the census itself, a set of complex procedures and operations. The panel finds that the A.C.E. was well planned and documented and that it seems generally to have been well executed. However, until the Census Bureau completes additional studies of error in the A.C.E., the panel cannot offer a definitive assessment of it.
The net undercount of the population, as measured by the A.C.E., declined from about 4 million people (1.6 percent of the population) in 1990 to about 3.3 million people (1.2 percent of the population) in 2000. Moreover, net undercount rates were considerably reduced for such historically less-well-counted
groups as children, minorities, and people who rent their homes. The level of gross errors—both omissions and erroneous enumerations—may also have been somewhat reduced.
Scattered evidence points to significant problems that may have occurred in enumerating people residing in group quarters, such as dormitories and nursing homes (about 3% of the population). The treatment of group quarters residents may also have been a problem in the A.C.E.
Apart from the question about group quarters residents, the reductions in the overall net undercount and in the differential undercount are puzzling because estimates of key components of the dual-systems estimation formula are similar between the 1990 Post-Enumeration Survey and the 2000 A.C.E. In investigating this puzzle, the panel focused on two groups of people whose census records were not included in the A.C.E. process: people who required imputation to complete their census records and people reinstated in the census too late for A.C.E. processing.
People Requiring Imputation
The panel found a large part of the explanation for the reduction in net undercount in the group of people who had insufficient information (concerning name and other characteristics) to carry out the individual matching required in the core of the A.C.E. process. Their census records were completed by computerized imputation routines that used information from the household or a neighboring household. The 2000 census included about three times as many people requiring imputation (5.8 million) as did the 1990 census (1.9 million). This much larger number contributed to reducing the net undercount when the dual-systems estimate from the A.C.E. was compared with the census count. Moreover, the people requiring imputation in 2000 were disproportionately found among minorities, renters, and children, thus accounting in large part for the reduction in differential net undercount for these groups relative to non-Hispanic whites, owners, and older people.
Of the 5.8 million people who required imputation for some or all of their characteristics, a large number—2.3 million—were imputed in situations when the household size and characteristics of other members were known. Another large group—also about 2.3 million—were imputed into households believed to be occupied for which household size, but not other information, was available. In these cases, the imputation used information for a similar size household in the immediate neighborhood. In terms of assessing the quality of the census, a much more problematic group—amounting to 1.2 million people in 2000—are those who were imputed into the census when there was no information about the size of the household or, in some instances, whether the address was occupied. Factors contributing to this type of imputation, which could have been problems in the MAF or in follow-up operations, should be thoroughly investigated.
People Reinstated in the Census
Another group of people excluded from the A.C.E. processing were residents of housing units that were temporarily removed from the census data processing in summer 2000 as part of the ad hoc operation to reduce duplication from the MAF. This operation identified 6 million people as potential duplicates whose records were temporarily deleted from the census. After further examination, 3.6 million of them were confirmed deletions, and 2.4 million were reinstated, but too late for inclusion in the A.C.E. These 2.4 million reinstated people contributed to reducing the net undercount when the dual-systems estimate from the A.C.E. was compared with the census count. They were about equally likely to be found among historically better-counted groups as among historically worse-counted groups, so they did not affect differences in net undercount rates.
The Census Bureau’s initial population estimates obtained through demographic analysis—a technique that uses birth, death, and Medicare records and estimates of net immigration to build an estimate of the population—were lower than the estimates from the census and the A.C.E. This result suggests that both the census and the A.C.E. overcounted the population. The panel finds, however, that there are serious questions about the demographic analysis, especially with regard to the methods for estimating the components of net immigration. The panel concludes that because of the uncertainties about the accuracy of the estimates of immigrants (especially the undocumented alien component) and emigrants, compounded by the difficulties of classifying people by race, demographic analysis should not be used as a standard for evaluation at this time.
Looking to the Census Bureau’s program for further evaluation of the census, the A.C.E., and demographic analysis, we urge the Bureau to devote resources to completing planned studies on as fast a schedule as practicable. The information from the 2000 evaluations is needed not only to assess the 2000 census, but also to plan for 2010.
Important aspects of census operations that we have identified for timely evaluation (some of which will be addressed in the evaluations to be released in mid-October) include:
the completeness of coverage of the group quarters population and the effects of address list development and enumeration procedures on coverage of this population;
the quality of the MAF and the contribution (of both good and erroneous addresses) of the Local Update of Census Addresses Program and other sources;
sources of addresses of people deleted from and reinstated in the census;
reasons for larger numbers of people requiring imputation to complete their census records, such as possible problems with follow-up operations; and
evaluations of the computerized routines used for imputation.
In addition, as soon as practicable, the important demographic and socioeconomic information collected in the census long form (sent to a one-sixth sample of households) should be evaluated.
The A.C.E. operations and results were analyzed extensively by the Census Bureau prior to March 2001, but those studies left important unanswered questions, such as about the quality of the matching. The Bureau is completing additional studies that will be released in mid-October. Further research that should be conducted on the A.C.E. includes evaluation of the population strata used for estimation and sensitivity analysis of the components of the process (e.g., matching, treatment of people who moved between Census Day and the A.C.E. interview day) to help establish error bounds for the dual-systems estimates.
The Census Bureau is conducting additional evaluations of components of demographic analysis for release in mid-October. Further work is needed—particularly to improve estimates of immigration and emigration—if the results of demographic analysis are to be useful for census coverage evaluation. We urge the Bureau to increase its resources for demographic analysis. It should lead a research effort by appropriate federal agencies and outside experts to develop improved methods and sources of data for estimating legal and illegal immigrants in surveys and administrative records as input to demographic analysis and for other uses.