– 7 –

Nonresponse Followup

In this chapter, we explore aspects of the 2020 Census Nonresponse Followup (NRFU) related to the potential impact on aspects of census quality. We begin by examining the operational phases of NRFU and how several modes of enumeration contributed to the completion of the NRFU phase of the 2020 Census operation. We then examine geographic variation in the composition of NRFU modes of enumeration, followed by an exploration of associations between NRFU enumeration modalities and demographic, economic, and housing characteristics.

7.1 OVERVIEW OF 2020 NONRESPONSE FOLLOWUP OPERATION AND CHALLENGES

The U.S. Census Bureau identifies types of enumeration areas (TEAs) to guide enumeration strategies. Two types of TEAs are relevant for NRFU; Self-Response and Update Leave (UL). Self-Response is designed to occur in areas where the majority of housing units have mail delivered to the physical location of the housing unit. Self-Response was the primary enumeration methodology for the 2020 Census. UL is designed to occur in areas where the majority of housing units either do not have mail delivered to the physical location of the housing unit, or the mail-delivery information for the housing unit cannot be verified.

Through the 2020 NRFU operation, census takers conducted in-person interviews to enumerate households that had not yet responded online, by mail, or by phone. Sixty-seven percent of households self-responded, and 33% were

enumerated through NRFU. Due to COVID-19, the start of NRFU was shifted by approximately three months to mid-July. With this later start, the Early NRFU operation (Phase 1), designed to reach college students before the end of the spring semester, was no longer conducted. Given this change, the NRFU operation began with a soft launch on July 16 in specific areas of the country. By August 9, NRFU was being conducted nationwide, and it continued for nearly two more months until October 15 when both production operations and quality control concluded.¹

Original plans for the phased approach to NRFU set both case-completion and fixed-date criteria, by which census field supervisor areas (CFSAs) could transition between phases. On the case-completion front, early work suggested that CFSAs were reaching the 60% and 85% thresholds to switch to Phase 2 or Phase 3, respectively, earlier than expected, with “many cases” not yet having a first contact attempt. Accordingly, the threshold was reset from 60% to 85% complete to transition to Phase 2, and from 85% to 90% to transition to Phase 3. The original dates for eligibility to shift phases were made inoperable by the delay in starting NRFU and the phased restart of operations across Area Census Office jurisdictions. Ultimately, September 4, 2020, became the new date of eligibility for Phase 2 and September 11, 2020, for Phase 3 (Fontenot, 2021d).

In early September, the Census Bureau was anticipating direction from the U.S. Congress to complete the 2020 Census apportionment count by the legal deadline of December 31. The operation implemented a series of changes to reduce the NRFU timeline to meet this deadline. Shortly after implementing the changes, a federal judge issued a court order that reversed those changes. One of the changes was the number of contact attempts made for NRFU Reinterview (NRFU RI) and Self-Response Quality Assurance (SRQA) cases. The number of contact attempts was reduced from six to three; this situation existed for approximately two weeks before a court order set the criteria back to six. During that period, some NRFU RI and SRQA cases were closed after three attempts and processed as such. The majority of NRFU RI and SRQA cases did receive six attempts. Phase 2 and Phase 3 of the NRFU contact strategy were largely predicated on mid-decade research findings indicating that high-performing enumerators significantly contributed to the completion of difficult cases. However, during the 2020 NRFU operation, federal court orders blocked the Census Bureau from reducing field staff, including low-performing staff. Completion date changes and the restriction on dismissing low-performing staff prevented the new phased approach to NRFU from being conducted exactly as planned.

___________________

¹ See https://www.census.gov/programs-surveys/decennial-census/decade/2020/planning-management/operational-adjustments.html.

The 2020 Census NRFU operation was also very different from its predecessors in terms of its use of administrative records data, not only to flag potential vacant and delete-worthy cases early on (for removal from fieldwork after one confirmatory field visit) but as a means for enumerating households if information of sufficient quality existed for a nonresponding household.

As noted, during late stages of NRFU, there was a period of uncertainty about when data collection would end. The unanticipated need to adjust NRFU operations in response to COVID-19 led the Census Bureau to seek to complete data collection at the end of October 2020. Congress did not approve this, and, in early August, September 30 became the target end date for data collection. An injunction forcing the end date back to October 31 was enacted but shortly thereafter escalated to the U.S. Supreme Court, which stayed the injunction in an unsigned order, paving the way for census field operations and data collection to end on October 15. This uncertainty of ending data collection for one of the most sizable and significant logistical operations in the United States forced the Census Bureau to be nimble and adjust staffing and operations numerous times. It also raises questions about the potential impact on the quality of the enumerations during closeout.

Issues of data quality noted in Chapter 6 are also significant for NRFU data. There are numerous challenges in collecting quality census data from households. Item nonresponse is a significant challenge that often results in inaccurate values being imputed (see Chapter 3). Given that item nonresponse and associated data quality vary from one mode of enumeration to the next, we examined variation in the mode of enumeration across NRFU phases. We conducted a descriptive analysis of NRFU modalities for the nation, counties, and census tracts. We also examined the clustering of NRFU modality configurations across counties. Finally, we examined geographic variation in NRFU response mode and the association with demographic, social, and housing characteristics to assess potential data-quality differences across these characteristics.

7.2 DESCRIPTIVE ANALYSIS OF NONRESPONSE FOLLOWUP ENUMERATION BY MODALITY AND PHASE

The Census Bureau provided the panel with data on NRFU by phase for the nation, counties, and census tracts. The data included information about the number of households enumerated by NRFU enumeration modalities for each level of geography. These data were used to assess modal patterns of enumeration and associations of these patterns with demographic, economic, and housing characteristics.

Table 7.1 summarizes enumeration modes for each phase, at the national level. Of the national total NRFU enumerations, 42% were by household

Table 7.1 National Summary of 2020 Census Nonresponse Followup Resolutions by Major Phase

NOTES: NRFU, Nonresponse Followup.

SOURCE: Tabulation from Nonresponse Followup Resolutions by Phase dataset, 2020 Census operational data. See Disclosure Review Statement; CBDRB-FY23-0214.

interview, 39% were by proxy, and 17% were by administrative records, with only 2% enumerated as special (count only and special closeout). This distribution varies across phases, with most NRFU enumerations in Phase 1 being household interviews (45.5%) followed by proxy (34.5%), administrative records (19.5%), and special accounting for only 0.5%. For Phases 2 and 3, proxy is the more common mode (53.4% and 57.1%, respectively), followed by household interviews (33.8% and 27.9%), and administrative records (6.1% and 8.5%). Thus there was a smaller proportion of administrative record enumerations after Phase 1 and a more significant proportion of enumerations that were by proxy and special. Differences in the mode distribution for Phases 2 and 3 were minor, indicating that, at the end of NRFU, there were not any dramatic shifts toward lower-quality modes of enumeration. The graphic depiction of the distribution of enumeration mode by phase at the CFSA unit of analysis (Figure 7.1) is consistent with the distribution observed for national data.

To further examine differences and shifts in enumeration mode by phases, we combined Phases 2 and 3, constructed the percent of enumerations by mode, and then subtracted the percent of enumerations by mode in Phase 1 for counties (Figure 7.2). Thus, positive values for a mode indicate a shift toward that mode in the later phases, and negative values indicate a change away from that mode. We observed a shift away from household interviews and administrative records during Phases 2 and 3 and toward enumeration by proxy. This pattern is further illustrated in Figures 7.3 (county) and 7.4 (census tract), in which Phase 1 and combined Phases 2 and 3 are compared by the percentage of enumerations performed using each mode.

7.3 CLUSTER ANALYSIS OF VARIATION IN NONRESPONSE FOLLOWUP RESOLUTIONS BY PHASE

To better understand commonalities in the distributions of NRFU enumeration modalities across geography, we generated three clusters using k-means clustering for NRFU enumeration modality configurations across counties and census tracts. K-means clustering is a vector quantization method, originally from signal processing, that aims to partition n observations into k clusters, in which each observation belongs to the cluster with the nearest mean (cluster centers or cluster centroid), serving as a prototype of the cluster.

Results (Figure 7.5) suggest that Cluster 1 seems to have high administrative records takeup in Phase 1 and a significant gap in percent occupied by proxy between Phases 1 and 2–3. Cluster 2 has generally higher vacant by proxy and lower occupied by interview in both phase groups. Finally, Cluster 3 seems to have higher levels of deletes. We then mapped the three clusters by county (Figure 7.6) and census tracts (Figure 7.7). The county- and census tract--

level maps indicate that Cluster 1 tends to be in more urbanized areas whereas Clusters 2 and 3 appear to be in more rural areas.

7.4 NONRESPONSE FOLLOWUP ENUMERATION MODALITIES AND DEMOGRAPHIC, ECONOMIC, AND HOUSING CHARACTERISTICS

As with self-response, areas with a higher proportion of households enumerated with modes that result in low item nonresponse will have higher quality data than those with a higher proportion of enumerations by modes with high item nonresponse. Geographic variation in the quality of collected data is likely associated with variations in populations’ social, economic, and housing characteristics as response modality varies along these characteristics.

We examined the association between 2020 Census NRFU enumeration modalities and social, economic, and housing characteristics for census tracts in the United States. Two research questions lie at the core of this effort. First, what is the nature of the associations between the proportion of housing units enumerated by the various NRFU modalities in the 2020 Census and census tracts’ social, economic, and housing characteristics? Second, what do associations between 2020 Census census-tract NRFU modalities and social, economic, and housing characteristics indicate about variation in the quality of 2020 Census data across social, economic, and housing characteristics of populations?

For this analysis, we used the same NRFU Resolutions by Phase dataset but opted not to distinguish between the phases of the NRFU operation. To make our analysis comparable with the work in Chapter 6, we further restricted our attention to NRFU resolutions of units as “occupied” (not vacant or delete) for the principal NRFU modalities of household interview, administrative records, and proxy. We derived tract-level percentages for these three modalities (as well as population count only, special closeout, and unresolved); scaled those percentages to approximate percentage of enumerated occupied housing units by multiplying by (100−Self-Response return rate); and merged these data with the same public 2016–2020 5-year sample American Community Survey (ACS) variables as used in Chapter 6. From the resulting data, we calculated zero-order bivariate correlation coefficients between each variable in the merged dataset. The correlation coefficients between each modality and each ACS variable are graphed from high to low for each modality.

To further examine associations between NRFU response modalities and demographic, economic and housing characteristics from the ACS, we ranked census tracts for each variable. Then we grouped ranked tracts in up to 20 categories (ventiles), from highest to lowest. We dropped census tracts with missing data for an ACS variable. Means of each ACS variable were calculated

for each ventile category, as was the mean of the percent of households enumerated using each NRFU enumeration mode. Line graphs were generated from the resulting data, illustrating the variation in modality from one ventile to the next and across all ACS ventiles.

Our data-quality assessment for NRFU is focused on household interviews, administrative records, proxy interviews, and count only. As described in Chapter 3, household interviews have fewer item nonresponses for race and Hispanic origin than do other NRFU modalities. Still, administrative records have fewer item nonresponses for age. Thus, for NRFU, household interviews provide the most accurate responses, followed by administrative records, and proxy interviews appear to have the highest percentage of item nonresponse.

7.4.1 Nonresponse Followup Correlations with Social and Economic Characteristics of Census Tracts

The correlation between ACS variables and the percentage of enumeration from household interviews is somewhat similar to the pattern observed with Paper Self-Response, as noted in Chapter 6 (see Figures 7.8–7.11). Lower socioeconomic status (SES) characteristics appear to be positively associated with household interviews. This is also true for other NRFU modalities. All NRFU modalities tend to be positively associated with higher percentages of low SES characteristics and negatively associated with higher SES characteristics. Given that all NRFU modalities are more likely to have higher item nonresponse than all Self-Response modalities, populations in census tracts with a higher percent of lower SES characteristics tend to be represented by poorer-quality 2020 Census data.

7.4.2 Nonresponse Followup Ventile Plots

Figures 7.12–7.17 present analysis of the approximate percentage of enumerated occupied housing units resolved by the major NRFU modalities, in relation to ventiles of the same ACS covariates used in Chapter 6. The percentage of household interviews was higher for census tracts with larger average household sizes (see Figures 7.12–7.13). Note that the highest y-axis value of these graphs is 30%, to allow an easier review of differences between ventiles. Census tracts with smaller average household sizes were more likely to be enumerated by proxy than were tracts with larger average household sizes. Consequently, tracts with smaller household sizes will likely have poorer quality data representing them in the 2020 Census. Census tracts with smaller percentages of adults with bachelor’s degrees or higher, and with lower median household income, were more likely to be enumerated by household interviews. These tracts were also more likely to be enumerated by proxy, and slightly more likely to be enumerated by count only. Thus, census tracts with lower levels of educational

attainment and with lower household income appear to be represented in the 2020 Census with poorer-quality data. This pattern was similar for census tracts with higher unemployment rates. Tracts with high unemployment were generally more likely to be enumerated by NRFU, with a higher proportion enumerated by household interview, proxy, and count only compared with census tracts with lower unemployment rates.

Census tracts with high percentages of non-Hispanic White people were less likely to be enumerated in NRFU (Figures 7.14–7.15). Those census tracts that were over 90% non-Hispanic White were more likely to be enumerated by household interviews compared with those census tracts that were between 60–90% non-Hispanic White. This is likely because census tracts that are 90% or more non-Hispanic White are generally rural census tracts that tended to be Update Leave areas. The patterns of NRFU modalities for percent non-Hispanic Black people, Hispanic people, and people speaking a language other than English were very similar and tended to be the opposite of patterns for non-Hispanic White people. Generally, the quality of 2020 Census data representing areas with concentrations of African American people and Hispanic people, and areas with populations that do not speak English well tend to have lower incomes.

Higher concentrations of renter-occupied housing were associated with higher enumeration by NRFU (Figures 7.16–7.17). An average of more than 12% of housing units were enumerated by proxy in areas where approximately 90% of housing was renter occupied. Census tracts with more vacant housing units were also more likely to be enumerated in NRFU. Census tracts with about 45% of vacant housing units had higher percentages of households enumerated by household interviews and proxy interviews. Census tracts with higher percentages of housing units being housing complexes with 20 or more units appeared more likely to be enumerated by proxy interview compared with census tracts with smaller proportions of housing being large multifamily complexes. The percentage of households with broadband internet subscriptions was negatively associated with enumeration by NRFU. Census tracts with smaller proportions having broadband were more likely to be enumerated by household and proxy interviews.

7.5 CONCLUSIONS

Just as it is difficult to imagine successful completion of 2020 Census data collection without Internet Self-Response, so too is it difficult to imagine any successful 2020 Census data collection if the traditionally resource-intensive NRFU relied on physical transfer of assigned questionnaires/cases and daily face-to-face meetings among data-collection crews, as was the case in 2010. In late stages of 2020 Census planning, the Census Bureau partially pulled back

from full automation of workload assignments among the enumerator employee pool, instead planning for a “less is more” approach in three phases of NRFU, putting the last (and arguably most difficult) 15% of NRFU cases in specific areas in the hands of a reduced set of enumerators—those who had proven most productive in the first phase. The COVID-19 pandemic-related replanning and the shifting target date for completion of data collection substantially impacted NRFU; the Census Bureau was legally barred by court ruling from dismissing field staff (and thus from conducting the new, phased approach to NRFU exactly as planned). The 2020 Census NRFU operation was also very different from its predecessors due to its use of administrative records data, not only to flag potential vacant and delete-worthy cases early on (for removal from fieldwork after one confirmatory field visit) but as a means for enumerating households if information of sufficient quality existed for a nonresponding household.

NRFU is an inherently challenging census operation from the quality standpoint, and the Census Bureau routinely pairs NRFU with an aggressive quality-assurance operation including reinterview by different enumerators. Recall bias, or respondent difficulty in providing accurate information about a fixed (past) reference date, is always a concern, as are opportunities for miscoding and data-entry error by enumerator staff and the perennial worries about “curbstoning” (deliberate fabrication of reported information to complete cases quickly). All of these factors, and more, had particular salience in the 2020 Census with the COVID-19 pandemic-forced delays and constrained timeline; the performance of data-entry systems and safeguards on enumerator smartphones, new in application in the 2020 Census, also undoubtedly had some effect on NRFU operations. Amidst this complex set of issues, our analysis was particularly constrained by the lack of evaluation results from the Census Bureau itself, if only to provide us with a better sense of the operational paradata actually retained during the NRFU operation; so we were compelled to keep our queries at a fairly high level. The original intent of the panel’s data analysis subgroup was to try to assess the general question of whether 2020 Census data collection was unduly rushed, by examining data on the number and timing of enumerator visits to NRFU case households. However, in discussion with Census Bureau staff, we concluded that it would be too difficult and disruptive to extract those data from the broader Census Data Lake that underlies the 2020 Census information technology architecture—given that the Census Bureau’s own researchers had not yet compiled such an evaluation dataset for their own reports. Accordingly, as a compromise measure, we focused on available data on the resolution of NRFU cases, broken down by the three defined phases of NRFU for 2020.

The analyses presented illustrate significant problems with variation in the quality of 2020 Census data for areas characterized by some indicators of lower socioeconomic status. Further assessment of the impact of response modes on data quality as associated with demographic, economic, and housing composition is suggested. The results presented do not attempt to assess the magnitude of the data-quality issues. More research is needed on the quality of data generated in the imputation process. Results from such research could be employed to estimate the magnitude and impact of imputation for item nonresponse.

Not all proxy enumerations are the same. In some cases, proxy informants may be very familiar with residents of the reference housing unit and provide accurate responses regarding the number of residents, residents’ race and ethnicity, sex, and perhaps even age. Yet, at the other extreme, proxy respondents may know little about the residents of the reference housing unit. In densely populated areas with high resident turnover, complex family arrangements, or a range of other challenges and combinations of challenges to knowing neighbors, the quality of proxy responses will more likely be less

than accurate. We assume that proxy and count-only enumerations result in substantial item nonresponse and consequently in poorer-quality data.

To repeat an observation from Chapter 4, errors in the census, whether coverage or characteristics, would matter less if they were equally distributed among population groups and across geographies. Yet there is ample evidence (see Chapters 6 and 7) that self-response is not uniform across the population and therefore that NRFU is not uniform. For example, census tracts with higher percentages of renters and Hispanic households and lower median incomes based on the ACS had higher rates of NRFU enumerations, and of proxy NRFU enumerations in particular, in 2020 compared with other tracts.

These patterns strongly suggest that there are inequities among population groups and geographies in the completeness and quality of 2020 Census data.

Conducting NRFU for millions of addresses under the pressure of the census time schedule is challenging, and it is to the Census Bureau’s credit that it does so under difficult circumstances. It is possible, moreover, that the quality of the 2030 NRFU will revert to the historical mean and that 2020 was an aberration due to the COVID-19 pandemic. It is also probable, given declining response rates to surveys around the world (including item as well as household nonresponse), that it will become increasingly difficult over time to enumerate the population in NRFU. Administrative records may help, although they are not a panacea—for example, they are often missing race and ethnicity and are not available equally across geographic areas and population groups (see Chapter 8).

Other chapters discuss needed developments and testing for methods to improve self-response and the use of administrative records for enumeration, which would reduce the extent of the NRFU operation. For the 2030 Census, however, it is likely that a significant NRFU component will be needed. A prerequisite for improving the quality of the data obtained in NRFU is for the Census Bureau to conduct research at a fine-grained scale on the geographic and population distribution of NRFU and particularly of proxy NRFU responses; we formally recommend the same, in line with our other recommendations in Chapter 12. Coupled with the research program outlined in Appendix D, the results from fine-grained analysis of NRFU could position the Census Bureau to develop specialized approaches for improving NRFU in 2030 for particular areas and groups. Publishing detailed results of its NRFU analysis would also help census data users assess the relative quality of the 2020 Census for specific areas and groups, and would contribute evidence-based suggestions for improving the 2030 Census.

This page intentionally left blank.