– 6 –

Business and Private Sector Applications

Catherine Fitch (Minnesota Population Center) moderated a session on business and private sector applications of census data. Daniel Cork (Committee on National Statistics) presented the slides that Quentin Brummet (NORC at the University of Chicago) had prepared but could not deliver due to illness. Brummet’s slides covered uses of census data for sample survey operations and the possible effects from implementation of differential privacy. Ken Hodges (Claritas) discussed the impacts of differential privacy on business applications with census data, followed by Nadia Evangelou (National Association of Realtors), who discussed the specific use cases of estimating housing in floodplains and benchmarking existing home sales. Floor discussion followed their remarks.

6.1 EFFECTS OF DIFFERENTIALLY PRIVATE NOISE INJECTION ON SURVEY OPERATIONS

The presentation prepared by Quentin Brummet (NORC at the University of Chicago) made the point that many aspects of survey operations rely on census data. The primary focus of his talk was on sample design and, specifically, how accurately one could predict whether a given household would have a specific characteristic of interest for sampling. He noted that census tabulations and census data also play into aspects of conducting a survey such as prioritizing cases for contact based on what is known about vacancy and

household composition, as well as post-data collection tasks like weighting and imputation.

Brummet’s approach was to consider designing a sample of the population for a geographic area in terms of the potential impacts of the noise-injected 2010 Demonstration Data Products (DDP) data on those sample design prospects compared with the original 2010 Census data. He simulated the construction of a hypothetical sample frame that includes areas with greater than zero population for a small population group of interest and carrying out stratified sampling based on a census tract having at least 30 percent of a given population. He looked at scenarios for multistage sampling, two-stage sampling, sampling with probability proportional to population size in low density areas, and sampling with probability proportional to five times the size of the target population in high density areas. He also simulated person-level sampling within the selected census tracts.

What he looked at in particular was the effect of noise injection on coverage and the fraction of the target population for which there was a risk of not being included in the sampling frame. He also looked at the projected increase in survey costs and the marginal cost to the survey of oversampling particular groups.

With regard to costs, Brummet’s bottom line was that the projected cost increase for samples for major racial ethnic groups would be minimal when using noise-injected census data. With regard to coverage, however, the potential impacts could be substantial.

Working out his sampling scheme for a survey with a target population of 25-to-49-year-old males, Brummet projected that coverage would be substantially reduced for American Indians and Alaska Natives (AIAN). In California, coverage would worsen somewhat, not only for the AIAN population but also for African Americans. In West Virginia, coverage would worsen somewhat for African Americans, with greater reductions for Hispanics and Asian Americans and a markedly greater reduction in coverage for the AIAN population.

Brummet concluded that noise injection for 2020 could make it difficult to use household composition and vacancy data for blocks and block groups to improve survey field operations. In contrast, the effects of noise on survey costs were projected to be minimal, for example, in designing a nationally representative sample that oversampled nonwhite households. However, for smaller demographic groups, the privatized data could result in coverage issues—that is, not including the targeted population percentage in the sample.

Brummet noted that survey weighting relied on accurate population benchmarks, and any error in the census population totals would propagate to other products such as the postcensal population estimates and the American Community Survey (ACS).

He mentioned factors that would likely work in favor of noise injection having a small impact on surveys, such as having “on-spine” geography,

particularly census tracts, and the averaging of noise across geographic areas. He cautioned that the ACS was currently used for many survey purposes and that it was still unclear how noise injection in the 2020 Census would affect the ACS. He also noted that the parameters for the overall ϵ and its distribution across geography and demographic variables had not been determined for 2020.

Daniel Cork (Committee on National Statistics), following his summary of Brummet’s points, added a point of his own pertaining to on-spine versus off-spine geography. Cork’s concern was whether the introduction of tract group as another layer of geography on the spine had been done by the Census Bureau in any functional or strategic way, or whether tract groups were constructed arbitrarily. It would be preferable to construct on-spine geographies in as functional and effective a way as possible in order to optimize the noise-injection procedures for geographies of interest to data users.

6.2 CENSUS DIFFERENTIAL PRIVACY AND PRIVATE SECTOR DATA PRODUCTS

Ken Hodges (Claritas) said his company’s use case did not involve a specific question or a decision to be made. Instead, Claritas used census data to build information products for a great many businesses, which in turn, have a great many use cases. Claritas produces demographic estimates every year for small areas nationwide, starting with census data. So the company was concerned with the overall impact of differential privacy.

Hodges said that he and his colleague Sarah Burgoyne compared the 2010 DDP with the published 2010 data nationwide. They looked at selected characteristics as well as the basic count for geographic levels: block group, census tract, county, and state.

They wanted to know, not only the extent of differences between the original 2010 data and the DDP, but also how the DDP data behaved. Did the small-area data sum up to the data provided for larger areas? Did differences diminish as the population size of the geographic area increased? And did the data pass basic consistency checks of the type that Claritas typically applied to its own estimates? Hodges said to keep in mind, as the Census Bureau often has pointed out, that differences between the original 2010 data and the DDP data were not necessarily any worse than the errors introduced by swapping in the original 2010 data.

6.2.1 Totals

Hodges said the quick answer to the question of whether or not the small-area data summed exactly to the data provided for larger areas was that yes, it did. There was some question about this back in the early days of this exercise. This was important for business applications, because many of them wanted to

Table 6.1 Mean Absolute Percent Difference Between 2010 Census Published Totals and 2010 Demonstration Data Products, by Geographic Level

SOURCE: Ken Hodges workshop presentation.

use data for very small areas aggregated to custom areas relevant to a business, for example, a three-mile radius or a 20-minute drive time around a store. Their objective was to get increased accuracy through that custom aggregation.

Looking at differences in basic totals between the original 2010 and DDP data (see Table 6.1), measured by the mean absolute percent difference, Hodges said that the housing unit numbers were invariant: they showed zero difference for all geographic levels. In contrast, the household population and other totals often varied, sometimes by a lot. In some instances, small numbers generated large percentage differences, such as when a 2 became a 6. In other cases, the differences were substantial numerically such as when a 1 became a 69 or when a 4 became a 128.

With Table 6.2, Hodges spotlighted two example outliers in terms of differences between the 2010 Census tables and the 2010 DDP in persons per household or average household size: a block group in Maine that saw its population per household diminished to near 0 and another in California that jumped from 2.3 to 99.0 persons per household. He recited other such phenomena: a block group in Louisiana with four people total in the original 2010 data, 0 people in group quarters, three housing units, and two households, so that persons per household equaled two. The DDP showed 128 people in households but only three households, so average household size jumped to 42.6. Correspondingly, other outliers went in the opposite direction: one block group in California had 1,296 people in the original 2010 data, but the DDP had only 764 people. Moreover, the number of households increased substantially, producing a very sharp drop in persons per household from 2.6 to 1.3.

Hodges and Burgoyne were surprised by these kinds of differences in totals. If swapping did not change basic totals, as Hodges understood to be the case, then the differences in the DDP should be considered errors. Such differences would be built into Claritas’ estimates that were used by a great many businesses in many applications. Claritas relied on the accuracy of census totals to

Table 6.2 Example Outliers on Differences in Persons Per Household Between 2010 Census Published Data and 2010 Demonstration Data Products

NOTES: GQ, group quarters; HH, (occupied) households; HHP, persons in households; HU, housing units; PPH, persons per household.

SOURCE: Ken Hodges workshop presentation.

evaluate its own estimates once every 10 years and to judge the accuracy of all kinds of private databases for which people make all kinds of claims. So a question was whether Claritas could still check its 2020 estimates or the claims of commercial database providers if differential privacy could potentially introduce such significant differences.

6.2.2 Consistency Checks

Hodges next turned to five consistency checks: (1) households must be less than or equal to housing units; (2) family households must be less than or equal to households; (3) group quarters population must be less than or equal to total population; (4) household population must be greater than or equal to family households times two because every family household had to have at least two people; and (5) persons per household must be greater than or equal to one. The original 2010 Census data passed all five checks at all geographic levels, but what about the DDP?

The DDP data passed the first three checks but not checks 4 or 5 until the level of aggregation reached the state level. The inconsistencies were not that extensive. Only 313 block groups of 217,000 nationwide, for example, failed check 5 (persons per household greater than or equal to one). When block groups were aggregated for various applications, the failure largely evaporated.

6.2.3 Characteristics

Hodges expected to find important differences between the original 2010 Census tables and the DDP when looking at race and ethnicity, age by sex, and other characteristics, including households by type and size and presence of people age 65 and older in the household. The analysis by him and Burgoyne used an index of dissimilarity (IOD), a measure of differences between two percentage distributions. It has a value of zero if the distributions were identical, up to 100 if they had nothing in common. They found that the IOD varied widely depending on the characteristics. The index was 3.8 for race and ethnicity distributional comparisons even at the block group level, compared with 35.4 for age and sex distributional comparisons. Looking at outliers for race and ethnicity distributions, the IOD for block groups with large differences between the original 2010 data and the DDP were greatly reduced when the block groups were aggregated to census tracts.

For perspective, they examined differences in ACS data compared with published 2010 Census data, specifically estimates for households by type and size at the block group level from the ACS for the five-year period 2008–2012 with the original 2010 Census data. The differences between ACS data and 2010 decennial published data were strikingly similar to the differences between the original 2010 data and the DDP data.

6.2.4 Concluding Remarks

Hodges said that the DDP data proved very useful for understanding the impact of differential privacy. Some of the differences with the original 2010 data were large and unsettling, particularly in the basic totals. Also bothersome were the inconsistencies such as unrealistic and sometimes impossible values for persons per household. Differences for characteristics varied widely across the different characteristics. Some of the DDP data strained credibility.

In terms of priorities, Hodge’s biggest concern right now was with the basic totals. Do they have to be that different? Could they not be made to pass Claritas’ basic consistency checks? For characteristics, perhaps categories needed to be aggregated. Claritas’ users would not need all of the combinations of race and ethnicity, although other users might.

Hodges concluded by saying that Claritas understood and appreciated the challenges the Census Bureau faced with privacy and the conflicting objective of providing quality data. Claritas would want to remain as a strong advocate for the Census Bureau and for census data and looked forward to remaining engaged with the Census Bureau throughout the process of developing differential privacy protection algorithms for the 2020 Census data.

6.3 CALCULATING FLOODPLAIN WEIGHTS AND BENCHMARKING NEEDS

Nadia Evangelou (National Association of Realtors; NAR) presented two projects that the NAR undertook using 2010 Census data. The first was to estimate how many homes were located in flood zones. The second involved benchmarking NAR’s data on existing home sales.

6.3.1 Floodplain Housing Estimates

Evangelou explained that this project involved estimating the number and characteristics (such as owner- versus renter-occupied) of housing in flood zones. This information was important to NAR’s members and could also help policy makers assess needs and formulate policies for floodplain management. The first step in the project was to overlay block-level 2010 Census data with 100- and 500-year floodplain maps provided by the Federal Emergency Management Agency (FEMA). Areas in the 100-year floodplain have a 1 percent annual chance of flooding, while areas in the 500-year floodplain have a 0.2 percent annual chance of flooding. The next step was to calculate the share of each census tract’s housing units in the floodplain, using the housing unit counts in each block. For example, two of 10 blocks in a tract might fall within a floodplain, containing 120, or 20 percent, of the tract’s 600 housing units. Evangelou presented the example of Maricopa County, Arizona, which based on NAR’s calculations had an estimated 92 percent of homes intersecting a floodplain. Using 2018 ACS data, the project was able to estimate some of the characteristics of housing in the floodplain such as the percentage that is owner-occupied with a mortgage.

The project was also able to estimate the effect on the real estate market of a lapse of the National Flood Insurance Program (NFIP), which received short-term extensions for several years. With the floodplain housing weights, NAR estimated that, nationwide, about 40,000 home sale closings or 1,300 closings per day might be cancelled or delayed. The most affected states would be Florida, Texas, and California. Moving forward, NAR would want to update these floodplain weights with the 2020 decennial housing unit counts at the block level and the latest maps provided by FEMA.

6.3.2 Benchmarking Existing Home Sales

Evangelou explained that NAR’s data on existing home sales, monthly and annual, have been developed from a sample of multiple listing service sales. Over time, the universe that the sample represented changes, so NAR has regularly benchmarked or adjusted the existing home sales data series to make the sample accurately represent the number of home sales occurring throughout the country.

NAR’s benchmark against the 2000 Census estimates from the Public Use Microdata Sample resulted in a downward revision of 13 percent of the existing home sales series. In 2007–2010, the benchmark revision, which used ACS estimates, was an 11 to 16 percent downward adjustment. Checking ACS data every year since 2010 showed no need for rebenchmarking. Moving forward, NAR wanted to be able to benchmark the existing home sales figures with 2020 decennial estimates. These were just two projects for which NAR relies on census and ACS estimates.

6.4 FLOOR DISCUSSION

In the floor discussion, Helen Nissenbaum (Cornell Tech) said she was struck by Hodges’ comment that accuracy with the DDP data could be improved by aggregating categories, acknowledging that some users would probably want to see the more detailed data. Yet with more detail, people could be exposed in ways that we would not want the census data to do. Hodges offered race and Hispanic ethnicity as an example of what could be done to reduce error with fewer categories, just as would occur when aggregating across geography. Claritas’ users certainly would not need the 126 categories that result from cross-classifying the race categories alone and in combination with other races and then cross-tabulated by Hispanic ethnicity.

David Van Riper (Minnesota Population Center) asked Hodges what Claritas would do if the census data were not usable. What other data sources would Claritas use to produce its data series? Hodges replied that, for basic totals, Claritas had some resources, including its own master address file. The question was how to assess the accuracy of this resource in the absence of trustworthy census counts. Ten years ago, Hodges determined that continuing Claritas’ estimates using internal sources for the period 2000–2010 would not have been as accurate as starting from the 2000 Census data. The exceptions were in areas of very rapid population growth, for which Claritas adapted its methodology. Evangelou added that NAR produced its own housing data, but it was difficult to have accurate data at a granular level, so consequently, NAR needed census data to benchmark its series for counties.

Sierra Watt (National Congress of American Indians) asked Hodges about off-spine geographies, such as American Indian reservations, and the potential impacts of differential privacy on the 2020 Census data. Hodges said that this question was very important for Claritas because so many of its business users wanted data aggregated to areas, such as a 20-minute drive time around a store, which by definition were off-spine geographies. That capability was critical for Claritas products, which was why Hodges and Burgoyne focused on whether the small-area data summed exactly to larger-area totals in the DDP. The data

met that standard, although there were anomalies in other aspects, such as persons per household.

William O’Hare (demographic consultant) observed that Hodges had been in the information business for a long time and was very forthcoming and transparent about Claritas’ data products, but some of Hodges’ colleagues in the private sector had not met such a high standard. O’Hare asked if Hodges had any advice for people as to what kinds of questions they should ask when someone tried to sell them data that were “better than the census.” Hodges replied that the solution used to be simple, which was to ask the prospective data supplier if they were building from the decennial census numbers or the Census Bureau’s population estimates. If they were not using census data, they should be asked why, and it should be determined what were they using instead. Hodges commented that a question was whether the actual or privatized data would be the basis for the estimates program going forward, which was of concern to Claritas.

Joe Salvo (New York City Department of City Planning) asked Hodges why Claritas did not use block-level data. Hodges replied that it did, but most of its products were designed to build on block groups because the block data were so sparse. Also, the ACS data, which Claritas used heavily (as it used the census long-form sample data previously) only go down to the block group. Where Claritas did use block data was in some of its systems that were used to create those custom geographic aggregations. So to draw, say, a three-mile ring around a point, the software would take in all of some block groups but only parts of others. The block counts would be used to determine what percent of the block group was in or out of the custom area. Every decennial, Claritas used the census block data for this purpose, but in between census years, Claritas used its block master address file.

Fitch asked both presenters to comment on the trade-off between geographic detail and subject matter granularity and to make a judgment on which was more important. Evangelou replied that her association needed both granular geography and detail about housing and homeowners. Hodges said that business users of the census have always needed data for small areas. That is why companies like Claritas came into existence because census data, beginning in 1970, were made available in computer-readable form for small areas. Claritas used to generate census tract estimates, but they were too big. Claritas clients who wanted data for block groups, blocks, and even block faces have been constantly flabbergasted to learn the kinds of detailed data that were not available at the block level.

This page intentionally left blank.