Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 47
Experimentation and Evaluation Plans for the 2010 Census: Interim Report 4 Considerations for the 2010 Census In carrying out our primary charge regarding the selection of experiments and evaluations for the 2010 census, the panel inevitably had to consider plans for the conduct of the census itself. Moreover, the conduct of every census inevitably affects the Census Bureau’s overall research program for the decennial censuses. Thus, in this chapter the panel presents three recommendations concerning some census operations with a view to their contributions to improvement to census methodology. Although we understand that the design of the 2010 census is relatively fixed, we hope that the material in this chapter may still be of use to the Bureau. TECHNOLOGY The Census Bureau will be using more technology in the 2010 census than in previous censuses, and this has raised some concerns that the panel would like to see addressed in the final plans for 2010. The concerns involve the functioning of the handheld computing devices to collect field enumeration data and the operation of the management information system for the 2010 census. By management information system is meant the various software systems that manage and monitor, somewhat interactively, the mailout-mailback process, nonresponse follow-up, field enumerator hiring and firing and compensation, questionnaire data capture, and other major census processes. We don’t know the full extent to which these systems need to interoperate, but at least some modest degree of interaction is required, for example between the Master Address File (MAF)–TIGER system and the handheld devices in providing electronic maps for the handheld devices to display. The two primary concerns are whether the transmission of data using the handheld computing devices could be compromised in some manner (or could be lost unintentionally through mistakes and technological problems) and whether the needed interoperability of the components of the management information system could be hampered either by the adapting of software or the acquisition of newer software releases for the various components of the system between the dress rehearsal and the 2010 census. With respect to the security of the transmissions of the handheld computing devices, the motivation to do harm to the census counts may be relatively modest given the lack of a financial incentive, and this may result in less chance for a security breach. However, this argument is not compelling. Furthermore, not only is there interest in reducing the opportunity for a security breach, there is also the matter of being able to assure census data users that the counts are valid. To accomplish this, the Census Bureau should carry out an independent validation and verification of the functioning of the handheld devices. This could be accomplished in the following ways, either in the 2008 dress rehearsal or in the 2010 census:
OCR for page 48
Experimentation and Evaluation Plans for the 2010 Census: Interim Report Establish a dual recording stream for all data from mail-in, telephone, or handheld devices: one file to go to the contractors and one to be retained by the Census Bureau. In the event of catastrophic failure by a contractor or a serious challenge to the results, it will be important to have all the raw data in the hands of the Census Bureau. It is practical to develop simple programs, written and run by Census Bureau personnel, that will search large data files for patterns of interest. In this way, unexpected or curious results can be efficiently discovered and checked, and this can contribute to the validation and verification effort. Related to points (1) and (2), the Census Bureau should develop quantitative validation metrics, a priori, to check for data set self-consistency and comparison of redundant data. Other important general operational measures that we recommend for the 2010 census, either to determine whether any security breaches have occurred or to prove that the 2010 Census was secure (and which are probably already carried out), include: Retention of an archive of all raw data with date and time stamps. In the event of serious software failure, it would be important to be able to “replay the census” from these raw data. Use, by the Census Bureau and contractors, of dedicated processing systems that run no other applications and have highly secured network connections and secure accounts. Use of periodic system checkpoints to monitor and analyze software systems for intrusions or unauthorized manipulations of data. Strict control over handheld devices, including their inventory, individual device identification, and permission to operate (turn them on, turn them off, enable data transfer, disable data transfer, etc.). Use of methods to prevent and detect bogus data streams, including data that impersonate handheld devices. With respect to concerns about configuration control of the management information system of the 2010 census, the processing history of the dress rehearsal could be retained and the software systems intended for use in 2010 could be used to “replay” the dress rehearsal soon before the 2010 census to identify any systems that fail to interoperate. That is, assuming that the management information system for the dress rehearsal functions well, saving the processing history would then provide a means for determining whether modifications or updates of components of the management information system between 2008 and 2010 had raised any interoperability problems. (This is referred to as regression testing.) In addition, all information system errors encountered during the dress rehearsal should be captured in a form that allows them to be used during the software development work between the dress rehearsal and the start of the 2010 census. RECOMMENDATION 9: The Census Bureau should use dual-recording systems, quantitative validation metrics, dedicated processing systems, periodic system
OCR for page 49
Experimentation and Evaluation Plans for the 2010 Census: Interim Report checkpoints, strict control over handheld devices, and related techniques to ensure and then verify the accuracy of the data collected from handheld computing devices. RECOMMENDATION 10: The Census Bureau should provide for a check to ensure that the subsystems of the management information system used in 2010 have no interoperability problems. DATA RETENTION BY CENSUS CONTRACTORS Given the very successful use of contractors to carry out several decennial census processes in the 2000 census, it is expected that the use of contractors will be expanded in 2010. The component processes that will be contracted out in 2010 include (1) the decennial response integration system (DRIS), which involves systems management of the process of questionnaire response and data capture; (2) the automation of field data collection (FDCA); (3) the data access and dissemination system II (DADS II); (4) the 2010 census communications campaign; and (5) the printing contract. The fact that these systems will be operated by contractors raises an additional complication. Any data collected as part of developmental or operational testing of these systems prior to their use in 2010, as well as any data collected in monitoring the operations of these systems while in use in 2010, may be viewed as proprietary. This would limit the Census Bureau’s ability to assess the performance of these systems in looking toward 2020. While the contractors themselves may issue their own evaluation studies, this is insufficient given that contractors have a bias in evaluating their own systems. We assume that contractual agreements about the sharing of such data, if they have not already been provided for, are now too late (especially for developmental testing results). In that event, the Census Bureau should try to develop some informal understandings of data sharing with their contractors to address this issue. If it is not too late, such data-sharing clauses should be included in contracts. CENSUS ENUMERATION AS PART OF TELEPHONE QUESTIONNAIRE ASSISTANCE1 The current plans regarding the use of Telephone Questionnaire Assistance (TQA) are for it to function primarily as a means for assisting the public in making correct responses to the census form, in particular for complicated situations involving residence rules or responses to the race and ethnicity questions. In addition, this is a method for people to obtain assistance in filling out the census questionnaire when English is not their primary language. On occasion, this has also been a vehicle for households to provide their responses to the census questionnaire. However, this possibility was not encouraged in 2000. 1 Telephone Questionnaire Assistance was an operation used in the 2000 census in which people could call a toll-free number to get help in filling out their census questionnaire, to arrange to be sent a replacement questionnaire, to arrange to be sent a language guide, or to provide their census questionnaire information in situations in which they were not provided a census questionnaire.
OCR for page 50
Experimentation and Evaluation Plans for the 2010 Census: Interim Report For the 2010 census, we think the Census Bureau should consider making more transparent to respondents this option of collecting the information for the entire census questionnaire over the telephone once someone calls TQA. Specifically, whenever someone connects to TQA, the willingness of the operator to take the complete information, instead of just providing the specific help requested, should be made known to the caller during the initial part of the interaction. Our understanding is that this was not done in previous censuses due to the resources needed, especially the number of operators, and due to the additional procedural complications, especially of providing this opportunity for those receiving the census long form. However, given that this is a short-from-only census, we think that the need to get the information as soon as possible, when possible, should outweigh other concerns about making this option more frequently used. This could be especially important if the hourly wages of field enumerators increase substantially in 2010, since collection of such information may importantly reduce the cost of the nonresponse follow-up. If this change is not implemented in 2010, the Census Bureau should collect sufficient information to carry out an evaluation after the census is completed as to the percentage of callers to TQA who ultimately sent back their census questionnaires to estimate the additional nonresponse follow-up costs due to the lack of collection of the entire census questionnaire over the telephone. Also, a possible experiment that should be considered is to sample the callers and ask those sampled if they would mind providing their information at that time by telephone to better estimate the additional resources required. RECOMMENDATION 11: The Census Bureau should strongly consider, for the 2010 census, explicit encouragement of the collection of all data on the census questionnaire for people using Telephone Questionnaire Assistance. In addition, the Census Bureau should collect sufficient information to estimate the percentage of callers to Telephone Questionnaire Assistance who did not ultimately send back their census questionnaires. This would provide an estimate of the additional costs of nonresponse follow-up due to the failure to collect the entire census questionnaire for those cases. The Census Bureau should also consider carrying out an experiment whereby a sample of callers to Telephone Questionnaire Assistance are asked whether they would mind providing their full information to better estimate the additional resources required as a result of expanding Telephone Questionnaire Assistance in this way. In conclusion, the panel is enthusiastic about the opportunity to collaborate with the Census Bureau on its plans for selecting and designing productive experiments and evaluations in conjunction with the 2010 census and, more broadly, a more productive research program overall. The Census Bureau has a very proud history of innovation, including the development of punch card machines, the first nonmilitary application of computers, survey sampling, hot-deck imputation, FOSDIC (Film Optical Sensing Device for Input to Computers), to name a few, and we hope to help continue this important tradition.