Research and Experimentation and Data Collection During the Census
As living arrangements, ethnic composition, attitudes toward government, frequency of moving, availability of enumerators, quality of administrative records, and other factors change nationwide, the methods best suited to enumerating the population of the United States also change. In addition, technological innovations (including statistical methods) provide opportunities to improve census methods. Therefore, a cycle of experimentation and data collection during a census, followed by evaluation, further development, and experimentation and testing between censuses is very important. The decennial census provides a unique opportunity to test new methodologies because of its size and its general level of public awareness and acceptance. Plans for research and experimentation and data collection during the 2000 census are now being finalized. These activities begin the process of developing methodologies for 2010.
Role of Research and Experimentation
The Census Bureau used four criteria in evaluating proposals for research and experimentation during the 2000 census. The research (1) must require the census as the test environment, (2) must provide measurable results, (3) must not compromise the success of the census, and (4) should provide information that will assist in planning major components of future censuses. Furthermore, it was recommended that the research minimize adverse effects on respondents and enumerators, provide significant potential benefits, and introduce no or only minor additional burden to respondents. The current budget for the 2000 census
allocates $21 million for research, roughly 0.5 percent of the cost of the census. The panel endorses these criteria and notes that the research budget seems reasonable. The first recommended criterion is extremely important. Research experimentation and data collection should not disturb field operations. Enumerators already have a difficult job, and additional research programs or data collection that complicate their procedures should not be taken on without serious consideration of the benefits and costs. Expanding on the third recommended criterion, the panel also strongly believes that methods that are tested during the census should produce data that are comparable with those collected under current census methodology. Given the use of census data for congressional apportionment and redistricting and the allocation of federal funds, it would be unfair if some areas were penalized or helped by being chosen for census testing.
A concern with respect to testing as part of the decennial census is whether it is possible to predict 12 years in advance of a census (through other testing and experimentation) what methodologies might be effective. After all, technologies change at a rapid pace, and the population itself is dynamic. In response, it is worth noting that in previous censuses, the Census Bureau staff carried out tests that proved useful in advancing census methodology for subsequent censuses. One example is the mailout/mailback procedure, which was tested in 1960 and introduced on a broad scale in 1970. Another is the current testing (initiated in the early 1990s) of various ways to increase response—through redesigning the questionnaire, use of reminder cards, and sending replacement questionnaires—which will likely save substantial funds. The long lead time from one census to the next can actually be beneficial for these changes, since the introduction of a major change to census methodology can have unintended effects on other parts of the census process, and understanding those effects can take a long time and a good deal of careful research.
When unanticipated problems arise during a decennial census that require additional funds, field staff, or other resources, there is an understandable tendency to shift resources from research experimentation and data collection to solving immediate problems. Unfortunately, this may "mortgage the future" of census taking for short-term benefits. Some important issues involving the methodology to be used for the 2000 census would have been clarified if additional data collection had been incorporated into the 1990 census. The Panel on Decennial Census Methodology (which examined census methodology prior to the 1990 census) called specifically for data collection during the 1990 census that the present panel believes would have been useful in planning the 2000 census methodology. A report from that panel (National Research Council, 1988:2–3) states the following:
The panel supports the concept of a master trace sample (MTS) that will facilitate a wide range of detailed studies on the quality of the 1990 census content. As the panel understands the Census Bureau proposal, the MTS will comprise a sample of census records that include not only the final values for each questionnaire item, but also the values for these items at each step in the processing, along with additional information such as whether the respondent mailed back a filled-in questionnaire or responded to telephone or personal follow-up. . . . We applaud the objectives of the MTS and support having as much of the file content as possible available in a public-use format. Such a file would greatly facilitate error analyses of the census. It would support more definitive studies than proved possible for many REX [research, evaluation, and experimental] projects in the 1980 census, in which analysts frequently encountered incomplete and inconsistent field data. . . . One additional application of the MTS that we support is a study of the potential of sampling in the final stages of follow-up to improve the quality of the data. This study would require the capture of data from enumerator records on how many callbacks, by phone and in person, were attempted before an interview was obtained. We understand that these records are not generally well maintained and are for the enumerators' own use. We urge that, for at least a subsample of the MTS, the Census Bureau make an effort to have the enumerators keep good records. These data should then be analyzed to determine the value of sampling.
Unfortunately, due to budget and time constraints, a master trace sample was not collected in 1990.
As in 1990, in 2000 there will be pressures to reduce the number of research experiments, and associated data collection, undertaken during the decennial census. The panel takes this opportunity to suggest priorities for the research experiments, so that if some reductions become necessary, the most promising experiments can be protected. In addition, the fact that nonmonetary resources are constrained during a decennial census argues for focusing on a select number of projects. This concern was expressed by the Panel on Decennial Census Methodology (National Research Council, 1988:3) as another reason for the importance and utility of collecting a master trace sample:
The panel urges that the Census Bureau not include too many separate projects in the REX [research and experimentation] program. Given limited time, staff, and budget resources, it would be far preferable for the Census Bureau to conduct a smaller number of studies well than to attempt a larger number of studies with poor results. In this regard, one advantage of devoting the necessary resources for obtaining a comprehensive and high-quality master trace sample is that the file has the potential for long-range use. It could be analyzed in many different ways throughout the next decade as new ideas and hypotheses about the factors involved in census data quality arise.
The panel notes a final and more general point on the role of research. The 2000 census presents many opportunities for data collection that would be useful to implement to improve census methodology and operations for future censuses. The benefits of experimentation during the census are considerable but are more limited and often carry some risk. Therefore, the first priority should be to understand which data have potential research value and to keep those data in a form that will facilitate later analysis. The second priority is experimentation. In the rest of this chapter we first discuss and assess proposed experiments and then data collection.
Proposed Experiments and Panel's Assessment
Seven research experiments have been proposed by the Census Bureau to be carried out during the 2000 census.1 The panel describes them below in order of our assessment of their priority: high, intermediate, low.
Alternative Questionnaire and Mail Treatment
This experiment has three parts: (1) the single-page format for the decennial census questionnaire that is currently used would be replaced with a booklet version; (2) a reduced set of residence rules would be used in some questionnaires; and (3) the increase in response from a targeted mailing of a replacement questionnaire would be assessed by an unclustered sample of households. We believe the second and third parts are important. The experiment will provide valuable information on the benefits from a reduced set of residence rules and, more importantly, on the value of a targeted replacement questionnaire in a census environment. The testing of a booklet questionnaire seems less important, since experiments on minor modifications to the questionnaire can be conducted outside the decennial census.
Administrative Records Census Experiment
This test explores whether administrative records could be used to acquire high-quality short form information through the merging and unduplication of several national administrative records lists. One por
tion of the test would attempt to provide household assignments, while the other would only collect information on individuals. To improve the quality of the match and as a validation tool, a coverage improvement survey would be used to collect both Social Security numbers and reconciliation information. This experiment will help reveal ways in which administrative records could play a more prominent census role, as well as provide various kinds of assistance for the proposed American Community Survey.
The panel supports three modifications that would enhance the value of this experiment. One problem with the current proposal is that none of the national lists target the non-elderly poor, an especially important source of census undercoverage. To remedy this problem, lists of food stamp recipients should be added to the lists already suggested. The panel also questions the allocation of the majority of the costs of this project to the coverage improvement survey, which is needed to provide Social Security numbers for help in matching. Typically, birth dates and names, both generally available on administrative records systems, are sufficient for high-quality matching. It is possible that the coverage improvement survey might help with information on movers. However, the value of the coverage improvement survey should be revisited given its share of the cost. Instead, the coverage improvement survey might be used only as a reconciliation survey, or the funds that are saved through discontinuation of the survey could be used to increase the size of the experimental area.
Finally, this project provides an ideal opportunity to test triple-system estimation: one list would be the census; the second would be the merged administrative records list; and the third would be the integrated coverage measurement survey list. Triple-system estimation offers clear advantages over dual-system estimation (with respect to addressing correlation bias) if the lists are of high quality, and therefore this methodology is worth examining for use as part of integrated coverage measurement in future censuses. Examination of related approaches could also be supported through the collection of these data, including methods for addressing dependence and heterogeneity (e.g., Darroch et al., 1993), as well as work on individual-level models of undercoverage (e.g., Alho, 1990; Huggins, 1989, 1991; Alho et al.,1993).
Use of Administrative Records for Nonresponse Follow-Up
This test would match (by computer) a sample of census nonrespondents with various national administrative records databases to fill in their short form information. The coverage improvement survey would be used to assess the quality of the information obtained. Discrepancies
would be attributed to either the computer match or the quality of the administrative lists. The panel believes that this is an extremely important experiment that could show how administrative records might be used to provide high-quality information and reduce costs in the census.
Targeted Enhancements to the Master Address File
Seven national administrative records lists would be acquired, merged, and unduplicated to produce a single database of addresses. The list would be used to update the master address file (MAF). The hope is to target the field work supporting updating of the MAF to areas with certain identified features, such as large clusters of unmatched street addresses, multiunit structures, blocks where residences have been abandoned, and mobile home parks. Areas to represent widely diverse situations would be selected for study and then subjected to simulated targeted MAF updates. The results for the precanvass operation would be used to conduct cost/benefit studies to compare tradeoffs between reduced fieldwork and loss of coverage in comparison to full canvassing.
The panel is excited about the possibility of using addresses from administrative records to update the MAF for the variety of purposes for which it is intended, including the decennial census. One suggestion is to tie all updates to the Topologically Integrated Geographic Encoding and Referencing (TIGER) system, because an address not referenced to census geography is of limited value.
Use of the Employee Reliability Inventory File by Nonresponse Follow-Up Enumerators
This test would use noncognitive tests on candidate enumerators to try to determine which of them have the necessary interpersonal skills for successful job performance, supplementing cognitive tests that will likely be in general use for the 2000 census. The panel is unsure of the value of this test. It is possible that a test of noncognitive characteristics might be useful in reducing employee turnover, but the possible benefits and the likelihood that they would be realized are not clear. However, the costs and risks of this project are minimal, and the project would provide some measurement of enumerator productivity, which is useful.
Census Calling Card Incentive Experiment
A sample of all households and a sample of nonresponding households would receive prepaid telephone calling cards (with a value of about $5) along with their questionnaire, with a letter encouraging their response to the questionnaire by telephone. After providing an interview, the calling card would be activated. The panel does not believe that this experiment requires testing during a decennial census. In addition, use of a monetary incentive for what is a legally mandated requirement raises a concern about whether this experiment could be implemented as a regular part of a decennial census.
Social Security Numbers, Privacy Attitudes, and Notification Experiment
To evaluate the degree to which requests for Social Security numbers would be accepted by households, questionnaires would be mailed to a national sample of households to request this information. As an alternative, a sample of census 2000 questionnaires would query respondents on whether they would be willing to provide their numbers. Another part of the experiment would determine the effect on the response rate of a notification on the census form that administrative records will be used to assist the Census Bureau in acquiring information. Finally, the Bureau would make use of a telephone survey to measure broad aspects of current attitudes concerning the computer matching of a household's information.
The panel judges that much of the information gained from this experiment is already in hand, through work on similar issues with respect to household surveys. In addition, it is risky during a decennial census to request information that is not legally required along with information that is so required. This experiment might be successfully conducted during a test census.
Regarding experiments that were not proposed, the panel has only one suggestion: it might be helpful to test how respondents understand and answer questions regarding their usual place of residence. This concept is still confusing in the draft census instructions (see, e.g., Tourangeau et al., 1997) and will likely lead to substantial reporting errors. The Census Bureau's expectation that a person's residence is the place where he or
she lives and sleeps most of the time may not be what is understood by some respondents. However, this expectation could be evaluated during a test census and thus does not require a decennial census environment.
In addition, the panel has some concern that more attention was not devoted to innovative methods for counting hard-to-enumerate populations. Some possibilities might include further work on such ideas as blitz enumeration, where crews of specially trained enumerators work on a compressed time schedule to enumerate an area, and team enumeration, where enumerators work in small groups.
Finally, a study that was described to the panel, the interviewer effects experiment, has been substantially broadened to a 2010 census error modeling and simulation research experiment. The panel did not review this version of the experiment, and so does not provide priorities for its use here.
Recommendations and Suggestions for Data Collection
In addition to a setting for experiments, a decennial census is also an opportunity to collect data on current census procedures. Data collection can be even more valuable than research experimentation during a decennial census. Knowing what happened during a census can help identify useful modifications in processes, and data collection can also support simulation studies that assess the benefits of alternative methodologies and identify needs for further research. The panel offers one recommendation and several suggestions for data collection in the 2000 census.
Collection of a Master Trace Sample
The Census Bureau has designed an extremely innovative data collection system to be used during the 2000 census to provide decision support to all operational managers. The data are summarized to support decisions that need to be made in real time. For future analysis, however, detailed process data on individual enumerations will very likely be required. For example, it will undoubtedly be important to know how many interview attempts were made, how long they took, and whether the respondent was a household member or a proxy respondent. Preservation of these data is not required for operational support, and much of this data would not be captured or even recorded by enumerators. Although the Census Bureau is concerned about the cost and additional workload required for collection of this information, the panel believes that it would be worthwhile to collect it on a sample basis. Furthermore, some of these data items could be preserved for research use, again on a
sample basis, by designing software systems to abstract these items from the existing data processing system without further enumerator effort.
Therefore, the panel strongly supports a renewal and modest expansion of the suggestion by the Panel on Decennial Census Methodology of 10 years ago (National Research Council, 1988) for the collection of a master trace sample. With the various innovations in the 2000 census, such as the possibility of sampling for nonresponse follow-up and alternative methods for enumeration (e.g., ''Be Counted" forms), it would be very useful if the planned data management system could collect a trace sample in, say, 100 census tracts around the country. (Sampling tracts would facilitate study of the effects at the block or interviewer level.) The trace sample would provide information as to what happened in all phases of data collection, which will be instrumental in guiding methodological advances to be used in 2010 and beyond. Specific variables that could be included in the trace sample collection are as follows:
- where the address came from (original master address list, local update, casing check, etc.);
- the type of questionnaire (long or short form), whether, and when it was returned, whether it was the first or a replacement questionnaire (or both), whether respondent-friendly enumeration was (also) used, if the household was a nonrespondent and a member of the nonresponse follow-up sample, then how many approaches for field enumeration were made, when they were they made, which mode was used, whether they were ultimately successful, whether data capture required proxy enumeration and, if so, what type of proxy enumeration, edit failures, and finally whether there were any data differences among duplicate responses for households or individuals; and
- the identification number of the enumerator, to facilitate evaluation of interviewer effects.
Of course, any of the above information that could easily be collected on a 100 percent basis should be.
Additional Suggestions for Data Collection
In addition to the data needed for a trace sample, the following would be valuable to collect:
- All information necessary to estimate the census cost model. This would include the training costs, hourly rate and any other compensation of enumerators, turnover rate, number of hours worked, responses per hour, number of enumeration offices staffed, and how long they were
- open. This should include costs of all sources of updates to the MAF, and advertising. Linking costs to information on census outcomes (i.e., improvements in the quality of the counts) would make it possible to carry out cost-benefit analyses of the various components of the 2000 census, such as the benefits of the reminder postcard and the replacement questionnaire. (Lack of this information complicated evaluation of the 1990 census and the costing for 2000.)
- A measure of interviewer quality. Information on hiring or training tests might help identify the characteristics of those interviewers who were more skilled, which could be linked to the quality of the data collected.
- The amount and type of duplication and the success of unduplication. The amount of duplication with respect to the various forms of questionnaire response should be measured. On a sample basis, telephone or field follow-up could be used to assess the success of unduplication.
- Complete integrated coverage measurement data. All information pertaining to integrated coverage measurement estimation, including, on a sample basis, integrated coverage measurement interviews, should be collected.
- Data needed to estimate and decompose the variance of the final census estimates. Finally, to be able to use total error modeling in the 2000 census, sufficient information to support such modeling should be collected.
Recommendation 5.1: The panel recommends that a trace sample be collected in roughly 100 tracts throughout the United States and saved for research purposes. The trace sample would collect detailed process data on individual enumerations. In addition, similar information on integrated coverage measurement should be collected, on a sample basis if needed. It would be very useful if information could be collected, again on a sample basis, to support complete analysis of the census costs model, all aspects of the amount of duplication and efforts to unduplicate, and information needed to support total error modeling of the 2000 census.
Our review of the Census Bureau's research and data collection plans was hampered without having the 2000 census evaluation plans. Because they have related objectives, plans for data collection, experimentation, and evaluation need to be considered jointly so that all important issues are covered and there is no unnecessary duplication. Ideally, the
Census Bureau needs to collect all data that support any evaluation studies it intends to carry out. More coordination of these activities would be desirable in the future, by having final evaluation plans earlier. Given the uncertainty about overall census plans and budgets for 2000, however, we recognize why coordination of evaluation and research plans was not possible. We hope that future censuses can benefit from longer times for planning.