Consensus Study Report
NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
This activity was supported by a contract between the National Academy of Sciences and the U.S. Bureau of the Census (1333LB21D00000003/1333LB21F00000121). Support of the work of the Committee on National Statistics is provided by a consortium of federal agencies through a grant from the National Science Foundation (No. 1560294) and several individual contracts. Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project.
International Standard Book Number-13: 978-0-309-70646-9
International Standard Book Number-10: 0-309-70646-7
Digital Object Identifier: https://doi.org/10.17226/27150
Library of Congress Control Number: 2023947244
This publication is available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
Copyright 2023 by the National Academy of Sciences. National Academies of Sciences, Engineering, and Medicine and National Academies Press and the graphical logos for each are all trademarks of the National Academy of Sciences. All rights reserved.
Printed in the United States of America.
Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2023. Assessing the 2020 Census: Final Report. Washington, DC: The National Academies Press. https://doi.org/10.17226/27150.
The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president.
The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. John L. Anderson is president.
The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.
The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The National Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.
Learn more about the National Academies of Sciences, Engineering, and Medicine at www.nationalacademies.org.
Consensus Study Reports published by the National Academies of Sciences, Engineering, and Medicine document the evidence-based consensus on the study’s statement of task by an authoring committee of experts. Reports typically include findings, conclusions, and recommendations based on information gathered by the committee and the committee’s deliberations. Each report has been subjected to a rigorous and independent peer-review process and it represents the position of the National Academies on the statement of task.
Proceedings published by the National Academies of Sciences, Engineering, and Medicine chronicle the presentations and discussions at a workshop, symposium, or other event convened by the National Academies. The statements and opinions contained in proceedings are those of the participants and are not endorsed by other participants, the planning committee, or the National Academies.
Rapid Expert Consultations published by the National Academies of Sciences, Engineering, and Medicine are authored by subject-matter experts on narrowly focused topics that can be supported by a body of evidence. The discussions contained in rapid expert consultations are considered those of the authors and do not contain policy recommendations. Rapid expert consultations are reviewed by the institution before release.
For information about other products and activities of the National Academies, please visit www.nationalacademies.org/about/whatwedo.
PANEL TO EVALUATE THE QUALITY OF THE 2020 CENSUS
TERESA A. SULLIVAN (Chair), Department of Sociology, University of Virginia
MARGO ANDERSON, Department of History, University of Wisconsin–Milwaukee (emerita)
ROBERT M. BELL,* Google and AT&T Labs (retired)
KATHRYN EDIN, Department of Sociology, Princeton University
MARC HAMEL, Statistics Canada (retired)
GEORGE T. LIGLER, Department of Multidisciplinary Engineering, Texas A&M University
THOMAS A. LOUIS,* Department of Biostatistics, Johns Hopkins University (emeritus)
LLOYD B. POTTER,* Texas Demographic Center, University of Texas at San Antonio
JOSEPH J. SALVO,* University of Virginia Biocomplexity Institute
REGINA SHIH, Department of Epidemiology, Rollins School of Public Health, Emory University
C. MATTHEW SNIPP, School of the Humanities and Sciences, Stanford University
EDWARD TELLES, Department of Sociology, University of California, Irvine
WENDY UNDERHILL, National Conference of State Legislatures, Denver
DAVID VAN RIPER,* Minnesota Population Center, University of Minnesota
Staff
DANIEL CORK, Study Director
CONSTANCE F. CITRO, Senior Scholar
MICHAEL L. COHEN, Senior Program Officer
ANTHONY MANN, Senior Program Associate
KATRINA BAUM STONE, Senior Program Officer
* Denotes member of panel’s designated data analysis subgroup.
COMMITTEE ON NATIONAL STATISTICS
KATHARINE G. ABRAHAM (Chair), University of Maryland, College Park
MICK P. COUPER, Institute for Social Research, University of Michigan
DIANA FARRELL, JPMorgan Chase Institute, Washington, D.C.
ROBERT M. GOERGE, Chapin Hall at the University of Chicago
ERICA L. GROSHEN, School of Industrial and Labor Relations, Cornell University
DANIEL E. HO, Stanford Law School and Stanford Institute for Economic Policy Research
HILARY HOYNES, Goldman School of Public Policy and Department of Economics, University of California, Berkeley
DANIEL KIFER, Department of Computer Science, Pennsylvania State University
SHARON LOHR, School of Mathematical and Statistical Sciences, Arizona State University, Emeritus
NELA RICHARDSON, ADP Research Institute
C. MATTHEW SNIPP, School of the Humanities and Sciences, Stanford University
ELIZABETH A. STUART, Department of Biostatistics, Johns Hopkins Bloomberg School of Public Health
Staff
MELISSA CHIU, Director
CONSTANCE F. CITRO, Senior Scholar
BRIAN HARRIS-KOJETIN, Senior Scholar
Reviewers
This Consensus Study Report was reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise. The purpose of this independent review is to provide candid and critical comments that will assist the National Academies of Sciences, Engineering, and Medicine in making each published report as sound as possible and to ensure that it meets the institutional standards for quality, objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process.
We thank the following individuals for their review of this report:
Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recom-
mendations of this report nor did they see the final draft before its release. The review of this report was overseen by SUSAN HANSON, Department of Geography, Clark University, and SUSAN A. MURPHY, Department of Statistics, Harvard University. They were responsible for making certain that an independent examination of this report was carried out in accordance with the standards of the National Academies and that all review comments were carefully considered. Responsibility for the final content rests entirely with the authoring committee and the National Academies.
Disclosure Review Statement
The U.S. Census Bureau’s Disclosure Review Board and Disclosure Avoidance Officers have reviewed the information products used to produce several figures and tables in this report for unauthorized disclosure of confidential information and have approved the disclosure avoidance practices applied to these releases.
By mutual agreement, Disclosure Review Board clearance was sought and obtained only for the generation of specific figures and tables published in this report, and not for the issuance of the underlying data files as standalone products or as addenda to this report.
For brevity, the SOURCE statement for each relevant figure or table in the text refers to this Disclosure Review Statement and lists the associated Disclosure Review Board approval number. These approval numbers are:
- CBDRB-FY23-0171 (Nonresponse Followup [NRFU] resolutions by phase, figures)
- CBDRB-FY23-0179 (race/ethnicity analysis)
- CBDRB-FY23-0180 (age heaping analysis)
- CBDRB-FY23-0197 (Master Address File [MAF] development, figures)
- CBDRB-FY23-0206 (correlation coefficients)
- CBDRB-FY23-0214 (NRFU resolutions by phase, tables)
- CBDRB-FY23-0215 (MAF development, tables)
- CBDRB-FY23-0221 (Group Quarters data quality)
- CBDRB-FY23-0224 (NRFU and Self-Response return rate analysis by ventiles of American Community Survey variables)
This page intentionally left blank.
Acknowledgments
The Panel to Evaluate the Quality of the 2020 Census wishes to thank the many people who have contributed to its work and to this final report.
We appreciate the support of the U.S. Census Bureau and its encouragement of independent, external review of the 2020 Census, under the acting directorship of deputy director Ron Jarmin in 2021 and then under director Robert Santos since his assumption of the office in January 2022. Karen Deaver has been an excellent collaborator as the primary information conduit between the panel and the Census Bureau, and the panel has benefited from the active engagement and interaction of Albert Fontenot and Deborah Stempowski as heads of the decennial census directorate. Kristee Camilletti provided great assistance in overseeing the contract behind this study and Kawaldeep Kaur Brar was instrumental in guiding panel members and staff through the clearance process. In the late stages of the work, RJ Marquette and Kelly Mathews were vital in ushering our requests to use the results of data analyses behind the Census Bureau information technology firewall through the Disclosure Review Board process. We have also benefited greatly from interactions with key Census Bureau staff including John Abowd, Dominic Beamer, Dierdre Dalpiaz Bishop, Maryann Chapin, Stuart Irby, Sallie Keller, Rafael Morales, Jennifer Ortman, and Victoria Velkoff.
In addition to those already named, we appreciate the contributions of Census Bureau staff who presented and discussed material with the panel during its open meetings and in regular sessions with the panel’s data analysis subgroup, including: Tamara Adams, Willette Allen, Robin Bachman, Judy Belton, Christine Flanagan Borman, Patrick Cantwell, Richard Denby, Jason Devine, Daniel Donello, Dora Durante, Monique Eleby, Christine Hartley, Michael Hawes, Kristen Hearns, Joan Hill, Cynthia Davis Hollingsworth, Eric Jensen,
Andrea Johnson, Nicholas Jones, Timothy Kennel, Sarah Konya, Julia Lopez, Barbara LoPresti, Frank McPhillips, Rachel Marks, Leanna Mayo, Thomas Mule, Roberto Ramirez, Jennifer Reichert, Rita Schuler, Kathleen Styles, Ben Taylor, Kevin Zajac, and Mary Frances Zelenak.
We reiterate our deep appreciation to the 2020 Census field staff and external census stakeholders who participated in the panel’s public meetings in 2021 and refer readers to the acknowledgments and compilation of meeting agendas in our interim report for the full credits.
This report contains many original analyses that would not have been possible without sustained work by our analytic subgroup. Its members were Robert M. Bell, Thomas A. Louis, Joseph J. Salvo, and David Van Riper with subgroup chair Lloyd B. Potter. The analytic subgroup met virtually twice a week for nearly a full year, working behind the Census Bureau’s information technology firewall and then preparing proposals to the Census Bureau to clear some of these analyses for publication. After this work continued outside the firewall, Margo Anderson and Marc Hamel joined most of the biweekly meetings. Wendy Underhill also attended many of these meetings.
The COVID-19 pandemic affected not only the careful planning of the Census Bureau, but the timeline and working conditions for this panel as well. We could not have done this work without a helpful and dedicated staff. Many members of the Committee on National Statistics (CNSTAT) and National Academies of Sciences, Engineering, and Medicine staff facilitated our work. Constance Citro, senior scholar, offered us wise counsel at every point and participated in all our sessions. Her knowledge and experience with the census proved invaluable to us. In particular, Connie’s expert drafts of core chapters on coverage measurement, race and ethnicity measurement, and age heaping were vital in sharpening our thinking and advancing our work. Michael Cohen’s assistance to the data analysis subgroup was critical to the group’s progress, and he was also instrumental to our review of census operations and administrative data. Katrina Baum Stone brought her experience with several federal statistical agencies to bear on our work on the census and helped us find the forest when there were too many trees. Anthony Mann handled panel logistics, not a simple task given the need for all meetings to be virtual or hybrid, with various members and interviewees experiencing connectivity issues and time zone constraints. Brian Harris-Kojetin, CNSTAT director through February 2023, has served as the lead contact on the Census Bureau contract and has offered us helpful information at timely moments, and CNSTAT support of this panel and its work has continued under the directorship of Melissa Chiu.
Our heartiest thank-you goes to Daniel Cork, the study director, who dealt with the minutiae of multiple subtopics, shifting data points, and often conflicting suggestions from the panelists and others. With a report of many hundreds of pages, he managed the herculean effort of tracking chapter versions, coordinating the presentation of figures, and keeping the panel timely and on
budget. We appreciated his considerable analytic talent and his skill in making complex topics more understandable. His candor, objectivity, and perseverance made this panel’s work possible.
Teresa A. Sullivan, Chair
Panel to Evaluate the Quality of the 2020 Census
This page intentionally left blank.
Contents
1.1 The Census as Essential Civic Ceremony
1.2 Nature and Limitations of This Study
1.2.1 Panel Charge and Data Analysis Subgroup Structure
1.2.3 The Enduring 2020 Census
1.2.4 Limits on Analysis and Publication
2.1 Four Key Innovation Areas, and Initial Vision for the 2020 Census
2.2 Emergence of External Pressures
2.3 Evolution of the 2020 Census Design
2.3.1 Progression of 2020 Census Operational Plans
2.3.2 Immediate Precensus Challenges and Operational Revisions
2.4.3 Changes in 2020 Census Operations Due to the COVID-19 Pandemic
2.4.4 2020 Census Data Products
3 Age Heaping as an Indicator of Data Quality
3.1 Age Heaping as a Quality Indicator for the 2020 Census
3.2 Analysis of Age Reporting in the Census
3.3 Sources of Age Heaping—Type of Response
3.4 Age Heaping by Response Mode
3.6 Data Quality in the Nonresponse Followup Operation
4 Coverage Measurement in the 2020 Census
4.1 Estimating Coverage Errors
4.1.2 Post-Enumeration Survey with Dual System Estimation
4.2.3 Race, Ethnicity, Housing Tenure, Age and Sex Groups, States
4.2.4 Post-Enumeration Survey—Components of Error
4.3 Cautions and Areas for Research
4.3.1 Uncertainty in Demographic Analysis
4.3.2 Delays in Demographic Analysis Net Undercount Estimates by Race
4.3.3 Lack of Subnational Demographic Analysis Estimates
4.3.4 Uncertainty in Post-Enumeration Survey Results
4.3.5 Comparison of Errors in the 2010 and 2020 PES
4.3.6 Limited Analysis of Components of Coverage Error
4.3.7 Limited Granularity of Coverage Estimates
4.4 Conclusions and Recommendations
5.1 Overview of Master Address File Development and the Dataset
5.2 MAF1 and MAF2, Precensus and Postcensus Pivot Points in the Master Address File
5.4 Spatial Variation in Master Address File Unit Categories
5.4.3 Unique Adds from 2020 Census Address-Building Operations
5.5 Impact of 2020 Census Address-Building Operations
5.5.1 Address List Input from User Programs
5.5.2 Field/Clerical Operations
5.5.3 Census Enumeration Operations
6.1 Self-Response Strategies and Processes
6.1.1 2010 Census Self-Response Procedures
6.1.2 2020 Census Self-Response Procedures
6.2 Analysis of Variation in Self-Response
6.2.1 Previous Work on Self-Response for Census Tracts
6.2.2 Panel’s Analysis of Self-Return Rates
6.3 Conclusions and Recommendations
7.1 Overview of 2020 Nonresponse Followup Operation and Challenges
7.2 Descriptive Analysis of Nonresponse Followup Enumeration by Modality and Phase
7.3 Cluster Analysis of Variation in Nonresponse Followup Resolutions by Phase
7.4.1 Nonresponse Followup Correlations with Social and Economic Characteristics of Census Tracts
7.4.2 Nonresponse Followup Ventile Plots
8 Use of Administrative Records for Enumeration in the 2020 Census
8.1 Administrative Records Enumerations in the 2020 Census
8.2 Models Developed for This Purpose
8.2.1 Housing Unit Status Model
8.2.2 Models to Estimate the Number of Occupants
8.4 Evaluating the Approach Used in 2020
8.5 What the Evaluation Does Not Answer
8.6.1 Administrative Records Enumeration as Part of Nonresponse Followup
8.6.4 “Real-Time” Simulation of an Administrative Records Census
8.7 Conclusions and Recommendations
9 Measurement of the Group Quarters Population
9.1 What Are Group Quarters and Why Are They Important?
9.2 Accurate Counts of Group Quarters Populations Matter for Local Governments
9.3 Number of Persons in Group Quarters by Type
9.4 Group Quarters Address List Development
9.5 Nature and Origin of Group Quarters Addresses
9.7 The Collection of Group Quarters Data in the 2020 Census
9.7.2 Imputation of Occupancy Status and Population Counts
9.7.3 Methods of Data Collection by Group Quarters Type
9.7.4 Facility Self-Enumeration
9.7.5 Item Nonresponse by Group Quarters Type
9.8 Conclusions and Recommendations
10 Measurement of Race and Ethnicity
10.1 Overview of Race and Ethnicity Measurement
10.1.1 U.S. Office of Management and Budget Standards and Census Race and Ethnicity Questions
10.1.2 Race and Ethnicity Distributions from 1970–2020—A More Diverse Nation
10.2 New Write-Ins, Data Capture, and Coding for 2020
10.2.1 Effects on Race and Ethnicity Estimates from the Census and the American Community Survey
10.5.2 Effects of Delays on Race and Ethnicity for Demographic Analysis and Population Estimates
10.5.3 Noise in the 2020 Redistricting File
10.6 Race and Ethnicity Measurement in 2020—Conclusions
10.6.1 Improvements to Collection of Race and Ethnicity Information
11 Impact of New Confidentiality-Protection Methods on 2020 Census Data Products
11.1 Census Data Products: Overview of Content and Timing of Release
11.2 Confidentiality Protection in the Decennial Census
11.3 The Road to the 2020 Disclosure Avoidance System
11.3.1 What Did the Operational Plan and Other Statements Say About Disclosure Avoidance?
11.3.3 Implications for Development of the 2020 Disclosure Avoidance System
11.4.1 Fundamental Responsibilities of Statistical Agencies
11.4.2 Census Planning, Testing, and Innovation
11.4.3 Privacy Protection in Other National Statistical Offices
11.4.4 Effects on Other Census Bureau Programs
11.4.5 Effects on Operational Quality Metrics
11.6.1 Framework for Balancing Utility and Confidentiality
11.6.2 Data Product Plan for 2030
12 Learning from 2020, Preparing for 2030
12.1 Priority Goals for 2030 Census Research and Development
12.1.1 Maximize Census Self-Response
12.1.2 Improve Quality of Nonresponse Followup Data
12.1.3 Reduce Quality and Coverage Gaps by Race, Ethnicity, and Socioeconomic Status
12.1.4 Continuous Modernization of Geographic Resources for All Living Quarters
12.2 Improving Census Research and Development
12.2.3 Systems Design to Facilitate Evaluation
12.3 Improving Census Processes
12.3.1 Continued Work on Administrative Records to Supplement Census Operations
12.3.2 Continued Development of Master Address File Curation and Updating
12.3.3 Improvements to Major Census Coverage Evaluation Methods
12.3.4 Revisit Unplanned Contingency Changes from 2020 that Merit Further Consideration
A Full List of Conclusions and Recommendations
C Additional Detail and Reference on 2020 Census Operations
D Extensions of Census Coverage Evaluations
E 2020 Census Group Quarters Definitions and Type Code List
F Additional Detail and Reference on Data Products and Disclosure Avoidance
Boxes, Figures, and Tables
BOXES
1.1 Federal Committee on Statistical Methodology (2020) Data Quality Framework
2.1 Key 2020 Census Litigation
2.2 Type of Enumeration Areas and Mail Contact Strategies in the 2020 Census
3.1 Derivation of an Age Heaping Metric
3.2 Response Categories for Age and Date of Birth, 2010 and 2020 Censuses
4.1 Methods for Estimating Census Coverage
4.2 Major Steps in Post-Enumeration Survey Data Collection and Estimation
6.1 Self-Responses Higher Quality than Enumerator Responses
6.2 Primus for Internet Response
9.1 Group Quarters in Pennsylvania Townships and Counties
9.2 Post-Census Group Quarters Review Program
9.3 Imputation of Group Quarters Characteristics
10.1 Request for Comment on Proposed Race and Ethnicity Measurement Standards
10.2 Content of the 2020 Redistricting Data File
11.1 Methods for Protecting Confidentiality, 1970–2020 Census Summary Files
11.2 Effects of the 2020 Disclosure Avoidance System on Data Quality: Overview
11.3 Legislation Protecting the Confidentiality of Decennial Census Information
11.4 Basic Steps in TopDown Algorithm for 2020 Census Disclosure Avoidance
11.7 Relevant Language—Foundations for Evidence-Based Policymaking Act of 2018
11.8 Excerpts—Year 2 Report, Advisory Committee on Data for Evidence Building
11.9 Legislation on Shared Statistical Agency–Data User Responsibility for Privacy Protection
12.1 Major Changes to U.S. Code Title 13 and Census Law Since Codification of Title 13 in 1954
FIGURES
2.1 Simplified 2020 Census type of enumeration area and mailing strategy map, by county
3.2 Age heaping in the 2010 and 2020 Censuses, using the ĥa metric
3.3 Age heaping for respondents with reported age only, 2010 and 2020 Censuses, using the ĥa metric
5.2 Percent of Master Address File units in Good Legacy Census category, by county
5.3 Percent of Master Address File units in Good Legacy Census category, by tract, Iowa
5.4 Percent of Master Address File units in Good Legacy Delivery Sequence File category, by county
5.6 Percent of Master Address File units in Bad Legacy category, by county
5.7 Percent of Master Address File units in Good Reactivations category, by county
5.9 Percent of Master Address File units in Good 2020 Adds category, by county
5.11 Percent of Master Address File units in Bad 2020 Adds category, by county
9.3 Item nonresponse rates by group quarters type, data collection via eResponse
9.6 Item nonresponse rates by group quarters type, data collection via Paper Listing
10.1 Percentage distribution of the population by race and ethnicity, 1970–2020 Censuses
10.3 Percentage distribution of response modes for race and ethnic groups, 2020 Census
TABLES
2.1 Requested and Enacted Budget for the 2020 Census, by Fiscal Year, and Date of Enactment
4.1 Components of the Dual-System Estimator of the True Population
4.6 Schedule of Person Interviewing Field Operations, 2010 and 2020 Post-Enumeration Surveys
4.7 2010 and 2020 Post-Enumeration Survey Features Relevant for Error
5.4 Addresses Added or Touched by 2020 Census Address-Building Operations, by Address Origin
7.1 National Summary of 2020 Census Nonresponse Followup Resolutions by Major Phase
9.2 Origins of Group Quarters Addresses in the 2020 Census
9.3 Transitions and Additions for Group Quarters Master Address File Units in the 2020 Census
9.4 Group Quarters by Occupancy Status and Type, 2020 Census
9.6 Allowable Enumeration Methods by Group Quarters Type and Code, 2020 Census
9.7 Persons in Group Quarters by Group Quarters Type and Method of Data Collection, 2020 Census
9.8 2020 Census Item Nonresponse Rates for Group Quarters Person-Level Items, by Type
10.2 Response Categories to Race and Ethnicity Questions, 1970–2020 Censuses
11.1 Release Dates/Scheduled Dates for Four Census Data Products, 1990–2020 Censuses
11.2 Features of the TopDown Algorithm Compared with the SafeTab-P, SafeTab-H, and PHSafe Algorithms
12.1 Response Rate (%) Via Online Forms in Recent International Censuses
C.1 Modification of Timing of 2020 Census Enumeration Operations Due to COVID-19
C.2 2020 Census Self-Response Mail Contact Strategy and Adjustments for COVID-19
C.3 2020 Census Non-ID Workload by Process and Outcome
C.4 Item Imputation Rates (Percentages), 2010 and 2020 Censuses
D.1 Hypothetical Example of How a Specific 5-Person Household Would Appear in Various Data Sources
F.1 Census Data Products, Content, Geographies, and Release Schedule, 2010–2020
