THE DECENNIAL CENSUS IS A CORNERSTONE of the U.S. federal government and its statistical system. The census’s constitutionally mandated purpose is to provide population counts to reapportion the U.S. House of Representatives every 10 years; a closely related purpose is to provide block-level population counts by age, race, and ethnicity to draw the boundaries of congressional and legislative districts. In addition, census information is used to allocate hundreds of billions of dollars of federal funds to states and localities; as the basis for postcensal population estimates for states, counties, and cities; to support federal, state, and local government program planning; to provide sampling frames and population weights for household surveys; and to provide denominators for such vital statistics as birth and death rates by age, gender, race, and ethnicity.
To address these information needs, the Census Bureau is preparing to conduct the nation’s 23rd decennial census. In 2010, the Census Bureau will collect basic information for each U.S. resident, including name, gender, age, date of birth, race, ethnicity, relationship to principal respondent, number of people in the housing unit, and whether the housing unit is owned or rented. The additional socioeconomic information that was previously collected on the census long-form questionnaire for a sample of households is now collected by the continuous American Community Survey (ACS), resulting in a short-form-only 2010 census.
Census planning is itself a major effort, which since the 1950 census has involved extensive research and development over the better part of each decade. The planning culminates in a dress rehearsal for the immediately upcoming census, followed by experiments and evaluations conducted as part of that census to begin the planning for the next census.
The Census Bureau recognized that the experiments and evaluations conducted for the 2000 census did not provide an adequate basis for 2010 cen-
sus planning. The Bureau therefore requested a study from a panel convened by the National Academies Committee on National Statistics to review the 2010 Census Program of Experiments and Evaluations (CPEX) so that it would provide a strong basis for 2020 census planning. The panel was charged to:
consider priorities for evaluation and experimentation in the 2010 census. [The panel] will also consider the design and documentation of the Master Address File and operational databases to facilitate research and evaluation, the design of experiments to embed in the 2010 census, the design of evaluations of the 2010 census processes, and what can be learned from the pre-2010 testing that was conducted in 2003– 2006 to enhance the testing to be conducted in 2012–2016 to support census planning for 2020. Topic areas for research, evaluation, and testing that would come within the panel’s scope include questionnaire design, address updating, nonresponse follow-up, coverage follow-up, unduplication of housing units and residents, editing and imputation procedures, and other census operations.
Together, the panel’s interim and letter reports (included in this volume as Parts II and III) fulfill the core mandate of this charge to assess the 2010 CPEX program. In those reports, the panel generally concluded that the specific experiments chosen by the Census Bureau for the 2010 CPEX are not likely to inform changes that will substantially affect either census cost or quality in 2020, and so largely squander a valuable testing opportunity. Moreover, the panel concluded that the ability to evaluate the 2010 census (apart from or in addition to the formal CPEX evaluations) will depend on the Census Bureau’s ability to retain input and output data from each 2010 census operation, for later linkage and analysis.
RESEARCH AND DEVELOPMENT FOR A COST-EFFECTIVE 2020 CENSUS
This final report builds on the panel’s earlier work by discussing census experimentation and evaluation in a broader context. In particular, this report describes a strategy for research and development (R&D) leading to a 2020 census design that controls costs, maintains quality, and adapts to major social and technological changes. An effective research and development strategy for the 2020 census should keep a goal of containing costs while maintaining quality; it should also recognize that new technical advances may not only reduce costs, but also drive them up, so that research and effective cost modeling are necessary to determine how best to use such advances for the most cost-effective census.
Since 1970, when the current approach of a mailout-mailback enumeration, based on a master address list, with in-person follow-up of addresses for which a questionnaire is not returned, was first used, the real dollar per housing unit cost (2009 dollars) of the U.S. census has increased by more than 600 percent—it was $17 in 1970 (about the same as in 1960), $33 in 1980 (91 percent increase over 1970), $43 in 1990 (30 percent increase over 1980), and $70 in 2000 (63 percent increase over 1990). It is anticipated to be as much as $115 in 2010 (64 percent increase over 2000). In contrast, the real dollar per housing unit cost of recent Canadian censuses (in 2008 Canadian dollars) has remained constant at $39–$40 for the 1996, 2001, 2006, and 2011 censuses.
The unprecedentedly high per housing unit cost expected for 2010 has resulted in part from planned long-term investments—improvements to the Census Bureau’s TIGER geographic database, which is essential for accurate geographic coding and mapping of addresses, and the testing and implementation of the continuous ACS. High costs have also resulted from a failed initiative to automate the nonresponse follow-up operation, which will use traditional paper-and-pencil questionnaires instead of handheld computers as originally planned. Moreover, contributing to cost increases over the entire 40-year span has been the accretion of operations to address special situations, which have not been fully evaluated to determine their benefits and costs. Underlying this accretion and the failure to achieve major steps forward on a timely basis, such as the nonresponse follow-up automation initiative, is a largely incremental approach to census design and a research and development program that is often unfocused and ineffective.
The overall quality of the census enumeration using the current mailout/mailback methodology has improved between 1970 and 2000, to the point at which major continued improvement is unlikely. The net undercount of the population (measured by the demographic analysis method) decreased from 2.7 percent in 1970 to 0.1 percent in 2000. Moreover, the difference in net undercount rates between African Americans and all others decreased from 4.3 percentage points in 1970 to 3.1 percentage points in 2000. These figures mask large numbers of offsetting errors of omission of people who should have been counted and duplicate or other types of erroneous enumeration of people who should not have been counted.
Some programs put in place to address coverage errors, such as a recheck of housing units initially classified as vacant, have contributed to reducing net undercount (they have also contributed to erroneous enumerations).
Other programs, sometimes put in place with no advance testing, such as a parolee and probationer check in 1990, have contributed little to coverage improvement. Given trends that make census-taking more difficult, it is unlikely that the increased costs anticipated for the 2010 census will achieve much, if any, further increase in quality. In order to lead to successful quality improvements for 2020 any additional expenditures would, at a minimum, need to be properly focused and carefully researched and planned.
Social and Technological Change
Changes in the U.S. population since 1970 that have made census-taking more difficult are well known. They include increased immigration, including illegal immigration, and therefore the existence of communities that are wary of cooperating with the census and in which English is not the primary language. Changing norms in residence and living arrangements have also complicated the concept of a single, usual place of residence for many people, including children in joint custody, people with seasonal homes, commuter workers or couples who maintain a separate workweek residence, and others. More broadly, the public’s willingness to respond to surveys has declined significantly over the past three decades. Given these factors, an incremental approach to census design and an unfocused R&D program are likely to result in a continuation of the pattern of escalating census costs in 2020 with little or no improvement in quality.
Nevertheless, important dynamics that are dramatically changing the environment in which the census is taken offer both challenges and opportunities to make the 2020 census markedly more cost-effective. Administrative record files from government assistance and other programs are improving in quality and completeness, the Internet is now a ubiquitous means for communication, and the proportion of household business that is carried out through the ordinary mail is shrinking. It is difficult to paint a reliable picture now of the United States in 2020, but it is easy to conceive of the Internet as being the primary method for communication and conducting household business for the great majority of residents, and that a large fraction of U.S. households may opt not to receive ordinary mail or may ignore much or all of it. It is also likely that administrative records will provide timely, high-quality, and inexpensive information that would be a useful input to a variety of census operations. Clearly, the design of the 2020 census should be dramatically different to accommodate such changes.
TOWARD A NEW VISION FOR THE 2020 CENSUS
To escape the pattern of incremental and often unfocused research and development for the next census leading to escalating costs with diminishing
returns on quality, the Census Bureau will need to completely overhaul its approach for planning the 2020 census. Planning for 2020 must begin with a set of clear goals for census costs and quality, a limited set of strategic visions that are likely to meet those goals, and an R&D program that is focused on determining which vision has the most promise and how best to implement it. The goal for census costs should recognize that an increase in real dollar per housing unit cost for 2020 over 2010 would be unjustified in comparison with the experience of other developed nations and unacceptable in a time of fiscal imbalance.
Recommendation 2.1: The Census Bureau should approach the design of the 2020 census with the clear and publicly announced goal of reducing the inflation-adjusted per housing unit cost to that of the 2000 census (subtracting the cost of the 2000 census long-form sample), while holding coverage errors (appropriately defined) to approximately 2000 levels.
To be clear, we do not make this recommendation to suggest cutting census costs simply for the sake of cost-cutting. The objective is a more efficient census that, for example, uses lower cost options like the Internet and administrative records to supplement or replace expensive follow-up operations without compromising census quality; effective research and improved cost modeling (see below) are critical to determining to what extent these efficiencies can be realized. The reason for setting a cost goal that is stark, ambitious, and public is to convey commitment to a focused 2020 census planning process.
This 2020 census cost goal should exclude the costs of the ACS, which will increasingly take on a life of its own as a source of intercensal data on a wide range of population characteristics.
Recommendation 3.1: The Census Bureau should immediately develop a limited number of strategic visions for the 2020 census that are likely to meet its announced goals for costs and quality. By strategic visions, we mean start-to-finish strategies for conducting all major census operations in order to confront looming threats and implement new technologies. In addition, evaluation of each vision should include thorough review of the costs and benefits of individual census operations relative to the announced cost and quality goals, to determine whether operations that are not demonstrably cost-effective may be eliminated or scaled back or whether new technologies or practices might usefully be introduced.
Recommendation 3.2: The Census Bureau should develop its 2020 census research and development program by identifying a handful of research questions whose resolution will determine
which of the visions of the next census are feasible and cost-effective and which are not. Priorities for evaluations of the 2010 census should be arrived at consistent with this research program, as should priorities for experiments and tests in the 2011–2018 period.
Recommendation 4.7: The Census Bureau’s planning for the 2020 census, particularly for research in the period 2010–2015, should be designed to permit proper evaluation of significant innovations and alternatives to the current decennial census design that will accomplish substantial cost savings in 2020 without impairing census quality. Otherwise, the census design in 2020 will either be an incremental change from that in 2010 with increased costs, or the Census Bureau may be compelled to implement a poorly evaluated and tested alternative design under severe time and cost constraints with a risk of substantially reduced quality. All involved, including Congress and the administration, should recognize that substantial cost savings in 2020 can be achieved only through effective planning over the course of the 2010–2020 decade and should fund and pursue research efforts commensurately.
PROVIDING TOOLS FOR CENSUS R&D
To make possible a truly focused R&D program for 2020 census planning, the Census Bureau needs to take immediate steps to develop the necessary tools for effective planning and evaluation. These tools include an improved, transparent cost model of census operations and well-documented data from all 2010 census operations in formats suitable for research and evaluation.
Recommendation 3.3: In order to provide early indications of the costs of competing visions for the 2020 census and to support effective planning throughout the decade, the Census Bureau should develop and validate a detailed cost model that not only represents the 2010 census, but also accommodates novel approaches to census-taking, including the use of data capture via the Internet and automated telephone systems, the use of handheld devices in nonresponse follow-up, the use of administrative record information for some types of nonresponse follow-up cases, and innovative mechanisms for reducing the costs of updating the Master Address File between 2010 and 2020. This cost model should be able to assess the implications of introducing specific changes to an existing design, singly and in com-
bination, and to distinguish between the direct and indirect cost effects of specific changes. This cost model should be thoroughly documented and transparent so that the Census Bureau can obtain the benefit of expert advice on cost-effective improvements to census operations.
Recommendation 3.4: The Census Bureau should retain sufficient input and output data, properly linked and documented, from each 2010 census operation to permit adequate evaluation of the contribution of the operation to census costs and data quality to feed into 2020 census planning. For this purpose, the Census Bureau should either establish an internal group or hire a contractor with database management expertise. This group would have the responsibility of retaining and documenting sufficient data from the 2010 census to be able to comprehensively represent the functioning of all census operations. Such a group would also have the responsibility of assisting Bureau research staff, using current database management tools, to produce research files to support the assessment of analytic questions concerning aspects of the 2010 census.
REVITALIZING THE RESEARCH INFRASTRUCTURE
To support a focused, goal-oriented R&D program for the 2020 census and, more broadly, to improve its research capabilities, the Census Bureau needs to take immediate steps to revitalize its research infrastructure. These steps include thorough assessment of the Census Bureau’s posture toward R&D compared with other data collection organizations and the development of means for research to break out of existing organizational “silos.”
Recommendation 4.1: The Census Bureau should comprehensively review the research and development practices and organization in other national statistics offices and in survey organizations in academia and the private sector, with the goal of modernizing and strengthening the Bureau’s own research and development program. Such a review should include assessments of and recommendations about:
How to organize and direct basic and applied methods research to best serve the decennial census and other Census Bureau programs;
How to organize information technology and database management to best serve research and operations, including how to manage the development of new technologies
and ensure access to adequate expertise in these technical areas;
How to operate collaborative project teams to facilitate timely innovation;
How to ensure adequate training in survey methods and related fields;
How to achieve extensive and intensive interaction with external research organizations and academic departments so that Census Bureau researchers and methodologists can benefit from related research work and ideas elsewhere; and
How to fund and establish priorities for research and applied methodology work.
Recommendation 4.4: The Census Bureau should put in place incentives and structures so that research is fully integrated and collaborative not only across programs, but also with operational planning. Research should be responsive to operational needs, and, in turn, research findings should play a primary role in informing operational decision making.
Necessary steps also include reestablishing the position of Associate Director of Methodology and Standards, reestablishing a strong Center for Survey Methods under that associate director, integrating research across programs and with operational planning, integrating census and ACS research, and renewing and refreshing mechanisms for obtaining outside expert advice.
Recommendation 4.2: To carry out the findings from the review recommended above, the Census Bureau should consider reestablishing and filling an associate director–level executive staff position to head the statistical and survey research activities at the Census Bureau, with authority to organize the Bureau’s research and applied methods activities.
Recommendation 4.3: The Census Bureau should give greater emphasis to survey methodology. One possibility for doing so would be to establish a core survey methods research center, staffed by full-time survey researchers and headed by a nationally recognized expert in census and survey data collection instruments. Such a high-profile center could give priority to research on making effective initial contacts with census and survey respondents, including those made with new technologies.
Recommendation 4.5: The Census Bureau should integrate decennial census and American Community Survey research—for
example, by using the ACS methods panel as a test bed for the Internet and other data collection methods to consider in the census and by matching census and ACS records to evaluate coverage in both programs. To support comparative census-ACS research and to inform users, the Census Bureau should carry out analyses that explore, at both the aggregate level and the level of individual households, the degree of differences and the source of differences in demographic characteristics and residence between the ACS and the decennial census.
Recommendation 4.6: The Census Bureau should renew and augment mechanisms for obtaining external expertise from leading researchers and practitioners in survey and census methodology and in relevant computer science fields. These mechanisms might include (1) a more active census professional advisory committee program in which the members have an opportunity to work more closely with Census Bureau staff in developing and evaluating ideas for improved census and survey methods; (2) increased opportunities for sabbaticals at the Bureau for university faculty and other short-term appointments for both senior- and junior-level (graduate student) academics at the Census Bureau; (3) increased opportunities for sabbaticals for Census Bureau staff at academic institutions and private-sector survey organizations; (4) the awarding of design contracts early in the decade to support research and development of innovative technologies for census and survey data collection and processing; and (5) more effective use of contracting processes to obtain expert services.
2020 R&D FOR STAGES OF CENSUS-TAKING
The strategic visions developed for the 2020 census will include different combinations of approaches for more cost-effective address list development, initial contact with and response mechanisms for household and group quarters residents, and follow-up operations. The panel suggests targets of opportunity for focused R&D for each of these stages of census-taking.
Address List Development
A priority for determining a more cost-effective strategy for developing the Master Address File (MAF) between 2010 and 2020 is to conduct a comprehensive evaluation of the accuracy and costs of the 2010 Decennial Master Address File (DMAF) at each stage of its development. This evaluation should determine, by comparison with results from the 2010 cen-
sus, the 2010 ACS, and the 2010 Census Coverage Measurement, which operations—Postal Service updates, Local Update of Census Addresses, block canvass, Vacant/Delete Check, and others—resulted in accurate addresses (occupied and vacant), which operations added nonresidential and nonexistent units, and which operations erroneously deleted valid addresses. The evaluation should examine the comparative performance of various address operations for geographic areas, types of neighborhoods, and types of structures, such as single-family residences, small multiunit structures, and large multiunit structures. It should simulate the effects on census costs and accuracy of deleting one or more address operations nationwide and for types of areas and structures.
Research and development should investigate the use of administrative records from government programs to replace more expensive address updating operations in some areas and the use of information on population and housing trends from the ACS and the Census Bureau’s population estimates program to target areas in which address canvassing may not be needed. R&D should also investigate the costs and benefits of continuous updating of the MAF with the cooperation of some technologically savvy state and local governments and pilot-test a continuous updating program in selected areas. A continuously updated MAF would have benefits throughout the decade for the ACS and other household surveys.
Initial Contact and Response
There are a number of ways of contacting households to let them know how and when to respond to the census, including not only the traditional mailing of a questionnaire but also mailing a postcard, sending an e-mail, or leaving a voice message. Similarly, Internet and interactive voice response could become much more prevalent modes of household response. There are issues of confidentiality protection and ensuring the ability to link a response to a particular address, yet other countries have solved these problems and are making extensive use of Internet response to their censuses.
Only about 3 percent of people reside in institutions and other kinds of group quarters, but in some local areas they constitute a large proportion of the population. Group quarters enumeration poses special challenges—first, of identifying all group quarters and then of gaining access and accurately enumerating group quarters residents. An early priority for R&D on more effective group quarters enumeration should be to investigate the use of administrative records to enumerate the residents of institutional group quarters, such as nursing homes, prisons, and college dormitories.
Nonresponse follow-up and other field operations, such as the Vacant/Delete Check, are time-consuming and add substantially to the costs of the census. Focused R&D for a cost-effective 2020 census should address ways to minimize field operations by making more extensive use of administrative records, as well as ways to reduce the costs of fieldwork, assuming that some level of in-person follow-up will continue to be required.
R&D on minimizing field operations should analyze 2000 and 2010 information to determine the likely effects on costs and data quality of using administrative records to replace some or all of nonresponse follow-up in some or all geographic areas. Options to consider include the use of administrative records to replace current last-resort procedures in late stages of follow-up; the use of administrative records after one or two attempts at in-person nonresponse follow-up; and the use of administrative records at the outset so that nonresponse follow-up is conducted only when there is no match for an address to a record. Options for reducing the cost of in-person follow-up include reducing the number of contact attempts before using last-resort procedures (or administrative records) and the use of handheld technology in place of paper questionnaires.
Cost-Benefit Review of Individual Census Procedures
A close examination of the modern U.S. census reveals the accretion of many individual procedures to address special situations for address list development, initial contact, nonresponse follow-up, and telephone and inperson coverage improvement follow-up. To satisfy a goal of a substantial reduction in real dollar per housing unit cost for the 2020 census, the Census Bureau will need not only to test and develop major changes in census operations, but also to examine thoroughly each individual procedure to determine its contribution to census costs and data quality.
This exercise will require the detailed, transparent cost modeling recommended above, so that stakeholders can be informed regarding trade-offs between cost savings and possible adverse effects on completeness of census coverage. While significant deterioration of data quality should not be contemplated, elimination of procedures that only marginally benefit quality and have significant costs should not be ruled out a priori. The U.S. census needs to be of high quality; it also needs not to consume disproportionate resources. Focused R&D that builds from strategically chosen evaluations of 2010 census results and is supported by detailed cost modeling is the necessary ingredient to achieving a new vision for a cost-effective 2020 census.