In connection with every recent decennial census, the U.S. Census Bureau has carried out experiments and evaluations. A census “experiment” usually involves field data collection during the census in which alternatives to current census processes are assessed for a subset of the population. An “evaluation” is usually a post hoc analysis of data collected as part of the decennial census processing to determine whether individual steps in the census operated as expected. The Census Bureau program for evaluations and experiments for the 2010 decennial census is referred to as the 2010 CPEX Program.
CPEX, like its predecessor programs, has enormous potential to help improve the next census, which is the federal government’s single most important, and most costly, data collection activity. A well-planned and well-executed CPEX is a sound investment to ensure that the 2020 census is as cost-effective as possible.
The Census Bureau is now determining the topics for experiments during the 2010 census. The specific designs of the experiments have to be final by summer 2008 to meet the planning needs for the census. Because the data needed to support census evaluations are typically output files from the census itself, the exact structure of individual evaluations is not yet as time-sensitive as the experiments. However, some early planning for evaluations is crucial so that the necessary data extracts can be prepared and retained. This is especially true because much of the data collection in 2010 will be carried out by contractors, and so data retention requirements need to be arranged with contractors as early as possible.
The Panel on the Design of the 2010 Census Program of Evaluations and Experiments has been broadly charged to review proposed topics for evaluations and experiments and recommend priorities for them for the 2010 census, to consider what can be learned from the 2010 testing cycle to better
plan for the 2020 census, and to assess the Census Bureau’s overall continuing research program for the nation’s decennial censuses.
The primary purpose of this interim report is to help reduce the possible subjects for census experimentation from an initial list of 52 research topics compiled by the Census Bureau to perhaps 6, which is consistent with the size of the experimentation program in 2000. This interim report also offers broad advice on plans for evaluations of the 2010 census. The panel expects to provide fuller details of individual experiments and evaluations in its subsequent reports.
The panel identified three priority experiments for inclusion in the 2010 census to assist 2020 census planning (in one instance, there might be several related experiments): an experiment on the use of the Internet for data collection; an experiment on the use of administrative records for various census purposes; and an experiment (or set of experiments) on features of the census questionnaire.
One important opportunity for improving census quality and possibly reducing census costs in 2020 is the use of the Internet as a means of enumeration. Although Internet response was permitted (but not advertised) in the 2000 census, the Census Bureau has elected not to allow online response in 2010. The panel does not second-guess that decision, but we think that it is essential to have a full and rigorous test of Internet methodologies in the 2010 CPEX. Internet response provides important advantages for data collection, including alternate ways of presenting residence rules and concepts, increased facility for the presentation of questionnaires in foreign languages, and real-time editing. It also has the feature of immediate transmission of data, which has important benefits regarding minimizing the overlap of census data collection operations.
Recommendation 1: The Census Bureau should include, in the 2010 census, a test of Internet data collection as an alternative means of enumeration. Such a test should investigate means of facilitating Internet response and should measure the impact on data quality, the expeditiousness of response, and the impact on the use of foreign language forms.
Another important opportunity for reducing costs and improving data quality is the use of administrative records. These are data collected as a by-product of the management of federal, state, and local governmental programs, such as birth and death records, building permit records, and welfare program records. In 2000, administrative records were the subject of an experiment intended to study their use as a complementary type of enu-
meration (that is, whether person counts for some geographic areas derived from records were consistent with census returns). However, administrative records could be used more broadly to assist a number of census tasks, including such uses as (1) to improve the Master Address File, (2) as an alternative to last-resort proxy response, (3) as an alternative to item and unit imputation, (4) to resolve duplicate search, (5) to validate edit protocols, (6) for coverage measurement and coverage evaluation, (7) for coverage improvement, and (8) to help target households for various purposes. It is important for the Census Bureau to determine, starting now, which of these various potential uses of administrative records would or would not be effective for use in 2020.
Recommendation 2: The Census Bureau should develop an experiment (or evaluation) that assesses the utility of administrative records for assistance in specific census component processes—for example, for improvement of the Master Address File, for nonresponse follow-up, for assessment of duplicate status, and for coverage improvement. In addition, either as an experiment or through evaluations, the Census Bureau should collect sufficient data to support assessment of the degree to which targeting various census processes, using administrative records, could reduce census costs or improve census quality.
Finally, given the crucial importance of the census questionnaire as a driver of census data quality, especially with regard to the nation’s data on race and ethnicity, and to correctly locate each person at the proper census residence, the Census Bureau should conduct either a large experiment or several smaller experiments on the content and method of presentation of the census questionnaire.
Recommendation 3: The Census Bureau should include one or more alternate questionnaire experiments during the 2010 census to examine:
the representation of questions on race and ethnicity on the census questionnaire, particularly asking about race and Hispanic origin as a single question;
the representation of residence rules and concepts on the census questionnaire; and
the usefulness of including new or improved questions or other information on the questionnaire with regard to (1) coverage probes, (2) the motivation of census questions, (3) the request of information on usual home elsewhere on group quarters questionnaires, and (4) deadline messaging and mailing dates for questionnaires.
In such experiments, both the 2000 and the 2010 census questionnaires should be included in the assessments to serve as controls. The Census Bureau should explore the possibility of joining the recommended experiments listed above into a single experiment, through use of fractional factorial experimental designs.
It is important that sufficient data be retained to enable postcensus evaluations of the processes used to update the Master Address File from census to census. The success of a mailout-mailback census is most dependent on the quality of its address list, and therefore understanding the contribution of the various processes used to update the address list, especially Local Update of Census Addresses (LUCA) and address canvassing, is crucially important. In addition, given the expense of address canvassing in all blocks, it is important to be able to ascertain the extent to which canvassing can be targeted to blocks that are likely to have changes. Both administrative records, especially building permit data, and commercial mailing lists may have value in assisting in the targeting of blocks for canvassing.
Recommendation 4: The Census Bureau should design its Master Address File so that the complete operational history—when list-building operations have added, deleted, modified, or simply replicated a particular address record—can be reconstructed. This information will support a comprehensive evaluation of the Local Update of Census Addresses and address canvassing. In addition, sufficient information should be retained, including relevant information from administrative records and the American Community Survey, to support evaluations of methods for targeting blocks that may not benefit from block canvassing. Finally, efforts should be made to obtain addresses from commercial mailing lists to determine whether they also might be able to reduce the need for block canvassing.
More broadly, a master trace sample database could be used to address a substantial number of questions about the functioning of the 2010 census. Such a database would necessitate the retention of the entire census processing history (including the coverage measurement processes) of all addresses for a selected sample of areas, structured in a way to facilitate analysis. For example, such a database would help determine what percentage of census omissions are in partially enumerated households, or it could assess the benefits of the coverage follow-up interview. The panel therefore recommends that the process for creating such a database be initiated.
Recommendation 5: The Census Bureau should initiate efforts now for planning the general design of a master trace sample database and should plan for retention of the necessary information to support its creation.
Also, evaluations should be carried out on the feasibility of coverage measurement through use of a reverse record check based on the American Community Survey. The reverse record check is an alternative method for estimating the completeness of census coverage of the population, which may have advantages over the methods of dual-systems estimation and demographic analysis that have been used for this purpose to date.
Recommendation 6: The Census Bureau, through the use of an evaluation of the 2010 census (or an experiment in the 2010 census) should determine the extent to which the American Community Survey could be used as a means for evaluating the coverage of the decennial census through use of a reverse record check.
Finally, the Census Bureau has no program for assessing the rate of omissions of residents of group quarters in the 2010 census, nor can it assess the rate of placement of group quarters in the wrong census geography. The Census Bureau should therefore take the first steps toward remedying this by collecting sufficient information in 2010 to evaluate ideas on how to include this capability in the 2020 census coverage measurement program.
Recommendation 7: The Census Bureau should collect sufficient data in 2010 to support the evaluation of potential methods for assessing the omission rate of group quarters residents and the rate of locating group quarters in the wrong census geography. This is a step toward the goal of improving the accuracy of group quarters data.
OVERALL CENSUS RESEARCH PROGRAM
It appears that basic census research is not receiving the priority and support needed to best guide census redesign. For example, tests on some topics have been unnecessarily repeated, and previous research has sometimes been ignored in designing newer tests. Also, some topics, by their nature, require a relatively long time to understand and therefore need to be separated from the decennial census operational cycle. The lack of priority of research can also be seen in that the results of the 2006 test census tests were not all completed in time for the design of the 2008 census dress rehearsal. Research continuity is important not only to reduce redundancy and to ensure that findings are known and utilized, but also because there are a number of
issues that come up repeatedly over many censuses that are inherently complex and therefore benefit from testing in a variety of circumstances in an organized way, as unaffected as possible by the census cycle. Finally, given the fielding of the American Community Survey, there is now a real opportunity for research on census and survey methodology to be more continuous.
Recommendation 8: The Census Bureau should support a dedicated research program in census and survey methodology, whose work is relatively unaffected by the cycle of the decennial census. In that way, a body of research findings can be generated that will be relevant to more than one census and to other household surveys.
THE 2010 CENSUS DESIGN
In carrying out our charge to advise on the development of plans for experimentation and evaluation for the 2010 census, and more generally to review the full program of research and testing for improving census methodology, three issues arose that relate to the 2010 census design itself and, consequently, its evaluation. While the panel is aware that most aspects of the 2010 census design have already been decided and cannot be easily changed given time constraints, there remains the possibility that some of the following recommendations may still be able to be acted on prior to 2010.
The first issue is the possibility of the introduction of errors into the data collection transmissions by the handheld computing devices that will be used to follow up households that do not return a mail questionnaire. The second issue is the possibility of interoperability problems in the various software systems constituting the management information system for the 2010 census. The third issue is the role of telephone questionnaire assistance in 2010.
Recommendation 9: The Census Bureau should use dual-recording systems, quantitative validation metrics, dedicated processing systems, periodic system checkpoints, strict control over handheld devices, and related techniques to ensure and then verify the accuracy of the data collected from handheld computing devices.
Recommendation 10: The Census Bureau should provide for a check to ensure that the subsystems of the management information system used in 2010 have no interoperability problems.
Recommendation 11: The Census Bureau should strongly consider, for the 2010 census, explicit encouragement of the collection of all data on the census questionnaire for people using Tele-
phone Questionnaire Assistance. In addition, the Census Bureau should collect sufficient information to estimate the percentage of callers to Telephone Questionnaire Assistance who did not ultimately send back their census questionnaires. This would provide an estimate of the additional costs of nonresponse follow-up due to the failure to collect the entire census questionnaire for those cases. The Census Bureau should also consider carrying out an experiment whereby a sample of callers to Telephone Questionnaire Assistance are asked whether they would mind providing their full information to better estimate the additional resources required as a result of expanding Telephone Questionnaire Assistance in this way.