Research Experiments
THE NON-AREX 2000 EXPERIMENTS
The Census Bureau is planning on carrying out five experiments during the 2000 census. The Administrative Records Experiment (AREX 2000) is separately described and discussed below. The four other experiments, for which we have no suggested changes, are briefly described here.
-
Incentives for Alternative Response Modes For a sample of households, the Census Bureau will examine whether offering free time on a telephone calling card increases the frequency of respondents' use of the telephone or the Internet to provide their census responses. There will be a control group and two treatment groups, both of which are offered the calling card. Each group is made up of three panels, one for each response mode: (1) reverse computer- assisted telephone interviewing, (2) automated spoken questionnaire, and (3) the Internet. While households may respond with the paper questionnaire, the calling card is activated after one of the above alternative response modes (selected for each panel) is used.
-
Use of Noncognitive Tests of Enumerators For a sample of enumerators, the Census Bureau will examine whether the use of a test of noncognitive characteristics can help to predict enumerator effectiveness. A sample of local census offices will administer a noncognitive test to a sample of their new hires. (This test will not be used in the selection
-
-
-
process.) The test to be used is the Wonderlic Employee Inventory, which measures long-term job commitment, trustworthiness, courteous job performance, conscientiousness, emotional maturity, safe job performance, and freedom from disruptive substance abuse. The goal of the experiment is to see if there is an association between the results from this test and employee turnover and work performance. The test would have to be judged to be job-relevant, have no between-group differences, and be criterion-related to be used in the selection process.
-
-
-
Alternative Questionnaire Study For a sample of households, the Census Bureau will examine whether alternative questionnaires with changes to one of the following— the overall appearance of the census questionnaire, the order and wording of questions on ethnicity and race, and the wording of the question concerning the definition of household residence—result in improved levels or accuracy of response.
-
Study of Attitudes on Confidentiality and the Use of Social Security Numbers For a sample of households, the Census Bureau will examine attitudes toward requests for social security numbers on census forms and notification of their use in linking to administrative records to support statistical analyses. There are two components. The SSN-notification component examines the effects of a request for social security numbers or a public notification on mail or item response rates in the decennial census. There will be different panels, depending on whether the household receives the long form or the short form, whether there is a request for social security numbers, whether the request is for the entire household or only for the person completing the form, and whether there is a notification that mentions the Census Bureau 's possible use of statistical data from other federal agencies. The SPA 2000 component is a household survey that assesses the public 's attitudes toward the Census Bureau's possible use of administrative records. The objective is to measure the public's opinion (of the federal government and the Census Bureau) of expanding uses of administrative records and of the Census Bureau 's possible interest in collecting social security numbers in the future. SPA 2000 will collect data twice, once before advertising and once immediately after April 1, 2000.
THE AREX 2000 STUDY
Description
The Administrative Records Experiment in the 2000 Census (AREX 2000) is an experiment that will test, in a limited set of sites (Baltimore City, Baltimore County, and three counties in Colorado), the use of administrative records as a data collection method. The goal of this test is to better understand the potential for merged lists of administrative records to deliver content and geographic detail, especially sufficient coverage, that meet reapportionment and redistricting requirements. Two methods, which differ by whether individuals are linked to households, will be used in the experiment to generate block-level population counts. The “top-down” approach merges a number of administrative records files, unduplicates them, and then assigns individuals block-level codes using the MAF/TIGER (Master Address File/Topologically Integrated Geographic Encoding and Referencing) system. The “bottom-up” approach matches residences found on the merged administrative records list with those on the census master address file, resolving any inconsistencies. This approach therefore forms households. The resulting population counts, disaggregated by race, Hispanic origin, and gender, will be compared with census counts.
Four evaluations are planned: (1) comparison of population counts (disaggregated demographically) to census counts; (2) analysis of the components of the implementation of the experiment, including effects on coverage and costs; (3) comparison of household level information for the “bottom-up” method; and (4) assessment of the feasibility of the use of administrative records for the purposes of delivering geographically disaggregated count and content information that meets reapportionment and redistricting requirements. Several eventual uses of merged lists of administrative records are under consideration: nonresponse follow-up, reducing differential undercoverage, imputation for item nonresponse, address list improvement, and special procedures to target individual groups. For nonresponse follow-up, the question to be answered is whether there are advantages to the use of (pre-follow-up) census data and AREX in comparison with the current use of (pre-follow-up) census data combined with nonresponse follow-up data.
Analysis and Recommendation
The panel questions whether the goals for AREX 2000 are consistent with the current plans. Specifically, how will this experiment inform the Census Bureau sufficiently to support more refined testing of administrative records (with respect to any of the above purposes, e.g., for nonresponse follow-up or for reducing differential undercoverage) for 2010? Clearly, assessing coverage of the merged list, possibly in conjunction with the census, is the first crucial step toward any of the possible uses of administrative records. A two-way match of the list of census enumerations and the merged administrative records list is a very important addition to this experiment.
Recommendation: The Census Bureau should modify the plans for AREX 2000 to include a match of the merged administrative recordslist with the 2000 census households. Costs of this new activity could be controlled through the use of field follow-up of unresolved matches on a sample basis.
Without a match, one could imagine a situation in which administrative records were considered deficient due to a lack of coverage, but matching (with additional later work) could have shown that missing people were generally included on an administrative records list that had been omitted in the test. Inclusion of that administrative records list would have eliminated the problem. More generally, a match might indicate why the merged list was inferior. Conversely, one could imagine a situation in which administrative records had excellent coverage in comparison with the census, but the process used to merge the lists was deficient, or the lists were of poor quality, so that a positive assessment of coverage was erroneous and the merged list was not nearly as useful as it seemed.
Furthermore, without a match, how will AREX 2000 adequately demonstrate the value of a merged list of administrative records for nonresponse follow-up? Distributions of information on characteristics will not indicate whether the answers are of high enough accuracy to be used for this purpose. Matching the census and AREX 2000 files will provide important additional information on the ways in which administrative records data are inferior or superior to census data, which could be easily missed by comparison of aggregates. If costs are a primary concern, sampling could be used to limit the costs of field reconciliation of differences. In ACE
block clusters, one could also use matching to determine whether individuals that are enumerated by the ACE survey and missed by the census appear on administrative records lists, which would help to assess whether administrative records could be used to reduce census undercoverage. This matching is a first step toward measurement of correlation bias and is also a preliminary step toward implementation of triple system estimation.
In addition to its recommendation for matching, the panel has six other proposals for AREX 2000. First, the Census Bureau should investigate whether AREX 2000 could be used to assess the possibility of using administrative records, in nonresponse follow-up, to supply census data only for the individuals who have a good match, i.e., those cases with a high likelihood of matching as derived from the Fellegi-Sunter algorithm, reserving field enumeration for the remaining, more difficult cases. The panel has not examined the overall feasibility of this approach; if it can be done with only a modest increase in costs, the Census Bureau should consider it.
Second, given the possibly limited utility of the current combination of administrative records used in AREX 2000 to cover the non-elderly poor, the Census Bureau should explore the process of obtaining approval from each state (and some counties) to acquire their food stamp records, as well as the process of obtaining Medicaid files. While obtaining 50 separate approvals from states for food stamp records may be time consuming, the great majority of them might be straightforward. Moreover, the immediate concern is to acquire files for the test areas, which lie in only two states. It would be extremely useful if the Census Bureau could acquire food stamp and Medicaid files for the test areas to assess the value of these lists for these purposes. The Census Bureau should give high priority to getting approval for the acquisition of these files for the test areas (which has been generally feasible in the past). (Besides providing information on the non-elderly poor, both food stamp and Medicaid files are the two obvious candidate record systems for augmenting Internal Revenue Service [IRS] data files for improving coverage.)
Third, given that facilitating acquisition of administrative records through legislative or legal change can sometimes take a considerable amount of time, the Census Bureau should assign a relatively high priority to beginning these activities as soon as possible. This is especially important if food stamp records are to be used.
Fourth, the second evaluation that is planned—analyzing components of the implementation of the experiment, including effect on coverage and
costs—does not appear to be fully thought out. It has not yet been fully described to the panel, and questions remain as to exactly how the data files are going to be merged, the effects resulting from the current choice of data files, how costs will be measured, etc. There is a concern that AREX 2000 is too specific a test on which to base what might be a stop-go decision with respect to the use of administrative records in the 2010 census. Some of the specificity of this test may stem from what appear to be limited resources allocated to it. The panel is concerned that AREX 2000 might not have adequate staff support. Given the importance and complexity of this experiment and given the additional suggestions made here, the panel believes that additional resources are needed. Therefore, the panel suggests that the Census Bureau consider allocating additional resources to this experiment and that it plan additional tests of administrative records for these purposes to learn as much as possible about the choices that are related to better or worse outcomes.
Fifth, the panel is concerned that the above uses of census data often require information on race and ethnicity. The panel strongly recommends that the Census Bureau, very likely in conjunction with other federal agencies and in conjunction with academia and research organizations through directed research, examine how to improve the race/ethnicity data available from administrative records and what the social and political implications would be as a result of the addition of this information to administrative records.
Sixth, these applications obviously require the matching and unduplication of records both from within a single administrative records system and across systems. Therefore, the Census Bureau should include in AREX 2000 an assessment of the accuracy of the matching and unduplicating operations used and an assessment of which data from duplicate records are the most accurate.