THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
National Academy of Sciences
National Academy of Engineering
Institute of Medicine
National Research Council
Commission on Behavioral and Social Sciences and Education
Committee on National Statistics
November 9, 2000
Dr. Kenneth Prewitt
U.S. Census Bureau
Room 2049, Building 3
Dear Dr. Prewitt:
This letter comments on the Census Bureau’s plans for deciding whether to release adjusted estimates of the population of states and substate areas from the 2000 census. That decision will be made in March 2001 so that the Bureau can meet its April 1 deadline under Public Law 94-171 for providing data to the states for redrawing congressional districts.
This is the second letter report of our Panel to Review the 2000 Census. The first report, issued in May 1999, commented on the Bureau’s plans for the design of the Accuracy and Coverage Evaluation (A.C.E.) Survey. The results of that survey will be used along with other information to evaluate the census, and if deemed beneficial, will be combined with census data, using dual-systems estimation methods, to yield adjusted estimates of the population. In this letter report we comment on the Bureau’s plans for evaluating the census data and the A.C.E. data and for deciding whether to release both the census population estimates, unadjusted for coverage errors, and the adjusted estimates.
The Census Bureau presented its plans for the evaluation and decision process that will lead up to the adjustment decision at an open panel workshop on October 2 in Washington, D.C. Bureau staff provided 16 papers that are now in draft form. The papers contain table shells that will be filled in, as data become available over the next few months, with information that is important for the evaluation and decision. The papers cover a variety of topics, including: overall census and A.C.E. quality indicators, quality of census processes, demographic analysis results, person interviewing, person matching and follow-up, missing data, variance estimates by size of geographic area, correlation bias, synthetic assumptions, and other topics. Bureau staff discussed the draft papers and responded to questions and comments from panel members and other workshop participants.
2101 Constitution Avenue, NW, Washington, DC 20418 Telephone (202) 334-3096 Fax (202) 334-3751 national-academies.org
The panel commends the Census Bureau for the openness and thoroughness with which it has informed the professional community of the kinds of evaluations that it plans to conduct of the census and A.C.E. data prior to March 2001. The papers presented at the panel workshop provide evidence of the hard work and professional competence of Census Bureau staff in specifying a series of evaluations that can inform the adjustment decision.
The panel recognizes the difficult task faced by the Census Bureau in evaluating the census and A.C.E. data by the time that it must provide congressional redistricting data to the states. Since it will not be possible for the Bureau to complete all possible analyses by March 2001, it will have to act on the basis of analyses that can be conducted before that time. In view of that constraint, the panel concludes that the set of papers presented to the workshop in draft form reflect competent, professional work to develop an informative set of evaluations for the short term. The planned analyses appear to cover all of the evaluations that can reasonably be expected to be completed within the time available. Furthermore, they appear to be sufficiently comprehensive that they will likely provide support for a reasonably confident decision on adjustment in March.
However, since the numbers themselves, which are, of course, critical to the evaluation process, are not yet available, it is not possible at this time to comment on what the adjustment decision should be nor to conclude definitively that the planned short-term evaluations will be adequate to support the decision. Such commentary will be possible only after the Bureau has completed its work and has provided the supporting data to the professional community.
In addition to the evaluations that are planned specifically to inform the adjustment decision in March 2001, the Census Bureau has a longer term evaluation agenda that includes projects to assess all major systems used in the 2000 census and many aspects of census data quality. That agenda, which will take several years to complete, also includes evaluations that are related to the A.C.E., such as a study of error in the process by which census enumerations and A.C.E. enumerations are matched in A.C.E. blocks. The panel urges the Census Bureau to identify those longer term studies that are likely to provide useful information with which to evaluate the adjustment decision, to give priority to these evaluations, and to provide detailed plans for these evaluations and a schedule that allows for completing them as soon after March 2001 as possible. Users of census data need to know when and what kinds of evaluations will be available in the longer term, just as they have been made aware of the Bureau’s plans for the evaluations to be completed by March.
The short-term evaluations that are planned to inform the adjustment decision in March will provide voluminous, complex data and analyses on a range of aspects of the census and the A.C.E. Review and assessment of this necessarily complex set of information will present a challenge for the Census Bureau’s Executive Steering Committee for A.C.E. Policy (ESCAP), which is charged to recommend an adjustment decision to the Bureau director, as well as for the professional community and stakeholders. The panel believes it would be useful for all concerned parties for the
Census Bureau to develop a summary tabular presentation of the factors affecting its decision.
One of the Census Bureau’s 16 papers, “Data and Analysis to Inform the ESCAP Recommendation,” is intended to summarize the analyses and the approach; it should usefully serve this purpose. However, that paper is itself lengthy, and we encourage the Bureau to provide a summary table in addition that focuses on key pieces of evidence in the decision-making process. The summary table would indicate for each piece of evidence such information as the nature of the empirical findings, the type(s) of analysis that were the basis for the findings, the strengths and weaknesses of the analysis, and the implications of the findings for the adjustment decision. The summary should also note relevant types of evidence that are not yet available.
We understand that no simple formula will lead from the available data to a recommendation whether or not to issue adjusted census counts by March 2001 for use in congressional redistricting. A sound recommendation will ultimately rest on professional judgment informed by the available scientific evidence. However, we believe that preparation of a summary presentation of key evidence will assist ESCAP to integrate what will necessarily be a large volume of complex information, some of which may be conflicting, in reaching an adjustment decision. Such a presentation will also assist the professional community and stakeholders to understand the basis for the decision.
In summary, the panel appreciates the openness and professionalism with which you and your staff have undertaken to provide an extensive set of data for use in determining the quality of the enumerated census and, alternatively, of any adjustments that might be made to improve the census data. In furthering your efforts, we make the following two suggestions that were discussed above:
The Census Bureau should prioritize its plans for previously planned long-range evaluation studies with a view toward completing those evaluations most directly relevant to the adjustment decision as soon as possible. While we understand that some studies, especially those involving new data collection, cannot be concluded before the statutory requirements for release of the data, we believe it would be useful for you to produce a schedule that allows for completing them soon after March 2001. The Bureau should make public its plans for these evaluations and their release.
Although the Census Bureau has developed plans for comprehensive review of large bodies of data for use in its decision on whether to release adjusted census counts, the panel believes that it would also be useful to have a summary presentation of key evidence. In particular, a summary table listing each piece of evidence and how it relates to the adjustment decision would be helpful to all parties concerned.
Panel to Review the 2000 Census
Janet L. Norwood (Chair),
Chevy Chase, Md.
Robert M. Bell,
AT&T Research Labs, Florham Park, N.J.
Norman M. Bradburn,
National Science Foundation, Washington, D.C.
Lawrence D. Brown,
Department of Statistics, University of Pennsylvania
William F. Eddy,
Department of Statistics, Carnegie Mellon University
Robert M. Hauser,
Department of Sociology, University of Wisconsin
Department of Statistics and School of Education, Stanford University
D. Bruce Petrie,
Canadian Institute of Health Information, Ottawa, Ontario
Andrew A. White, Director,
Committee on National Statistics
Constance F. Citro, Study Director
Michael L. Cohen, Study Director
Daniel Cork, Research Associate
Agnes Gaskin, Senior Project Assistant
Carrie Muntean, Research Assistant
Meyer Zitter, Consultant