–A–
Past Census Research Programs

The descriptions of census research in this section generally exclude operations and analyses directly related to census coverage measurement—matching of the results of an independent postenumeration survey to census records in order to estimate undercount and overcount. We do, however, try to describe some of the formative work along these lines in the 1950 and 1960 censuses, given the novelty of the approach in those counts. Furthermore, although these descriptions do describe experiment and evaluation work related to census content that was part of the long-form-sample questionnaire prior to 2010, we exclude some work related to areas that are now fully out of scope of the decennial census (in particular, the census of agriculture).

A–1
1950 CENSUS

A–1.a
Principal Pretests and Experiments Conducted Prior to the Census
Supplements to the Current Population Survey (CPS)
  • March 1946 (CPS areas, nationwide): test of collection of both current residence (where the interviewer found a respondent at time of interview) and usual residence (the CPS and decennial census standard) information; provided particular information on enumeration of non-residents staying with households and college students.

  • April 1948 (CPS areas, nationwide): tests of method of obtaining income data and of enumeration of people by both usual and current



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 115
–A– Past Census Research Programs The descriptions of census research in this section generally exclude op- erations and analyses directly related to census coverage measurement— matching of the results of an independent postenumeration survey to census records in order to estimate undercount and overcount. We do, however, try to describe some of the formative work along these lines in the 1950 and 1960 censuses, given the novelty of the approach in those counts. Further- more, although these descriptions do describe experiment and evaluation work related to census content that was part of the long-form-sample ques- tionnaire prior to 2010, we exclude some work related to areas that are now fully out of scope of the decennial census (in particular, the census of agriculture). A–1 1950 CENSUS A–1.a Principal Pretests and Experiments Conducted Prior to the Census Supplements to the Current Population Survey (CPS) • March 1946 (CPS areas, nationwide): test of collection of both current residence (where the interviewer found a respondent at time of in- terview) and usual residence (the CPS and decennial census standard) information; provided particular information on enumeration of non- residents staying with households and college students. • April 1948 (CPS areas, nationwide): tests of method of obtaining in- come data and of enumeration of people by both usual and current 115

OCR for page 115
116 ENVISIONING THE 2020 CENSUS residence. The test resulted in determination of pattern for asking income questions in 1950 as well as decisions (confirming residence rules) on enumeration of nonresidents and college students. • May 1948 (CPS areas, nationwide): test of questions on physical char- acteristics of dwellings; led to revised definition of “dwelling unit.” Experiments Conducted in Special Censuses or Other Surveys • April 1946 (Wilmington, NC): experiment conducted as part of special census, focusing on collection of both current and usual residence in- formation. Changes to enumerator training and questions on general population characteristics were also tested. The test resulted in first draft of population questions for 1950. • February 1948 (Washington, DC): experiment conducted as part of survey conducted for National Park and Planning Commission, Bu- reau of Labor Statistics, and Housing and Home Finance Agency. The experiment focused on questions on income and led to revision in the format of the schedule, specific questions, and instructions. • May 1948 (Little Rock and North Little Rock, AR): experiment con- ducted as part of special census on self-enumeration techniques; test resulted in information on possible response rates and comparative costs. • June 1948 (Philadelphia, PA): experiment attached to survey con- ducted for Interdepartmental Subcommittee on Housing Adequacy on methods of measuring housing quality; test resulted in revised defini- tion of “dilapidation.” • March 1949 (Chicago, IL, and adjacent counties): experiment con- ducted as part of Chicago Community Survey on rostering and obtain- ing complete enumeration of persons in households. Significantly, the experiment tested the use of a household questionnaire rather than ei- ther the master ledger-size schedule then used for census interviewing or individual person questionnaires. • June 1949 (Baltimore, MD): test to check the quality of reported hous- ing data conducted as part of survey conducted for Baltimore Housing Authority; test led to some revision of housing questions. Tests of Specific Phases or Operations of 1950 Censuses • May 1947 (Altoona, PA; Charlotte, NC; Cincinnati, OH; Louisville, KY): test on “document sensing,” or a format for the schedule to enable cards to be punched automatically; test indicated that technique was possible.

OCR for page 115
APPENDIX A 117 • January 1948 (6 southern counties): test of special Landlord-Tenant Operations Questionnaire led to revision of procedures. • August 1949 (33 field offices): test of alternative population and hous- ing schedules provided input to determination of final schedule (ques- tionnaire) for 1950. • September 1949 (Puerto Rico): test of population, housing, and agri- culture questions for enumeration in Puerto Rico. • October 1949 (Raleigh and Roxboro, NC): test of training procedures led to determination of final training plan. • November 1949 (Raleigh, NC): test of questions on separate Survey of Residential Financing to be conducted simultaneously with the 1950 census. • January 1950 (Chicago, IL): test of Survey of Residential Financing questions helped determine final procedures. Dress Rehearsals • April–May 1948 (Cape Girardeau and Perry Counties, MO): test in- tended to (1) compare quality of data obtained from a schedule pared down to very few questions to one with many questions; (2) collect both current and usual residence information; and (3) generally assess quality of data from new questions. The pretest led to the conclusion that the short schedule yielded no substantial gains in quality over the longer, more-questions instrument. It also helped refine residence rules for some census types and specification of the duties of enumer- ation crew leaders. • October 1948 (Oldham and Carroll Counties, KY; Putnam and Union Counties, IL; Minneapolis, MN): test focused on different enumeration procedures (self-enumeration, distribution of materials by post office, etc.) and their effects on data quality. Based on the test, the Census Bureau decided to use self-enumeration in the Census of Agriculture. The test also yielded cost, time, and quality data for the different ap- proaches. • May 1949 (Anderson City, School District 17, and Edgefield County, SC; Atlanta, GA; rural areas near each of 64 CPS field offices): test of training methods and final questionnaires and training procedures led to some modification in procedures (including procedures for ship- ping supplies to local offices) and determination of procedures for the postenumeration survey. SOURCE: Adapted from U.S. Census Bureau (1955:Table B, p. 6), with ad- ditional information from Appendix E of that document.

OCR for page 115
118 ENVISIONING THE 2020 CENSUS A–1.b Research, Experimentation, and Evaluation Program Although the 1950 census was the first to include a structured experi- mental and evaluation program (Goldfield and Pemberton, 2000a), details of the precise evaluations conducted in 1950 are not generally available. Only slightly more information about the shape of specific experiments is detailed in the Census Bureau’s procedural history for the 1950 census. In a short section titled “Experimental Areas,” that history notes (U.S. Census Bureau, 1955:5): A number of variations in the procedures for collecting data were intro- duced in ten District Offices. These variations made possible a compar- ison of procedures under actual census conditions. The experimental areas were located in Ohio and Michigan. In six of these districts, the alternative procedures involved the use of a household schedule (instead of a line schedule for a number of households), of the household as a sampling unit (instead of the person), and of self-enumeration (instead of direct enumeration). In four of the districts, assignments were made to enumerators in such manner that the variation in response could be studied in terms of enumerator differences. Remarking on the self-enumeration portion of the 1950 experiments, the procedural history for the 1960 census clarifies that the test areas were in Columbus, OH, and Lansing, MI, and that part of the experiment requested that individual respondents complete and mail back (on Census Day) the questionnaires left with them by field enumerators (U.S. Census Bureau, 1966:292). The impact of the 1950 experiment on the later adoption of mailout-mailback methodology in the census is discussed by Bailar (2000). A–2 1960 CENSUS A–2.a Principal Pretests and Experiments Conducted Prior to the Census Supplements to the Current Population Survey (CPS) • November 1958: test of definitions for unit of enumeration. • April 1959: test of farm definitions. Experiments Conducted in Special Censuses or Other Surveys • March–April 1957 (Yonkers, NY): major experiment conducted as part of special census. The experiment had two principal objectives: (1) ob- taining data on the cost of a two-visit interview process (and effects on data quality) and (2) testing direct entry of interview information on machine-readable forms, either by enumerators or household respon- dents. In particular, a two-interview approach was tested in which— for roughly every fourth household—a limited amount of informa-

OCR for page 115
APPENDIX A 119 tion was collected in a first interview and a long-form questionnaire was left for completion (and eventual pickup by an enumerator). The test demonstrated that enumerators could readily use the computer- readable schedules in the field but that respondents found them more difficult to follow. The two-visit approaches were found to be very expensive because of the difficulty of finding household members at home when enumerators attempted their visits. However, the idea of independent listing of dwelling units by crew leaders as well as enu- merators was found to have potential value for improving coverage. The computer-readable forms generated by the test also helped with debugging the collection and tabulation routines. The Yonkers test also gave the Census Bureau the chance to test a new question on address of place of work, but the Bureau found the results to be unsatisfactory. • October 1957 (Indianapolis, IN): focused test of several possible cov- erage improvement techniques in one postal zone as part of a special census. Building on the Yonkers test, one approach compared results of a recheck of listed addresses by crew leaders with enumerators’ orig- inal results. Another technique involved preparing postcards based on enumerator interviews, with the postal carriers instructed to report boxes for which there were no cards for further verification (delib- erately withholding a small sample of cards as a test of the carriers). Finally, the Bureau tested the completeness of lists of old-age assistance beneficiaries, juvenile delinquents, and other persons believed to live in low-income families obtained from local authorities, as well as the distribution of census forms in public schools for parents to complete. The tests provided evidence of coverage gains from the post office check and independent crew leader listings and suggested that the lo- cally provided special lists were useful in indicating missed persons. • October 1957 (Philadelphia, PA): census experiment focused on word- ing and placement of questions on labor force status, address of place of work, and date of marriage. Three different schedules were used in the experiment, which deliberately sent inexperienced interviewers to do initial questioning and trained CPS interviewers for verification interviews. One tested approach for the place of work question asked respondents to identify the location on a map of the city. The test re- sulted in revisions of the occupation and industry questions in the final census schedule. • November 1957 (Hartford City, IN): test of mailout census methodol- ogy as part of a special census. A four-page Advance Census Report was mailed to individual household addresses; respondents were asked to complete the form but not to mail it back, holding it for an enumer- ator’s visit instead. In the test, about 40 percent of households had completed part or all of the questionnaire prior to the visit, and the

OCR for page 115
120 ENVISIONING THE 2020 CENSUS advance questionnaire seemed to be particularly useful in improving the quality of data on value of home. January 1958 (Memphis, TN): test of self-enumeration and mailback • methodology as part of special census. Housing units were listed by enumerators and questionnaires distributed, with instructions to com- plete and mail on the census date. Mailed returns were chosen for follow-up verification, some by telephone and others by trained CPS interviewers; the post office check used in the Indianapolis test was also used for portions of Memphis. Part of the Memphis test also ex- perimented with collection of information on visitors present in house- holds on the census date and using reported usual residence informa- tion to try to allocate them to their usual home. Based on this test, a process for collecting “usual home elsewhere” information for some transient populations (enumerated where they are found on Census Day) was used in the 1960 census and in group quarters enumeration in subsequent censuses. Finally, the experiment asked some enumera- tors to query respondents for some information on neighboring living quarters (above or below, right or left, in back or in front); the tech- nique was subsequently adopted as part of the evaluation process of the 1960 census. February 1958 (Lynchburg, VA): further testing of mailout of Advance • Census Reports as part of special census. As in the Memphis test, CPS interviewers and crew leaders were used to reinterview some house- holds to verify the quality of mailback data. Attempts were also made to time interviews to determine whether the Advance Census Report reduced the time spent in enumerator interviewing. In May 1958 enu- merators returned to Lynchburg to recheck and verify the quality of housing data collected in February. March 1958 (Dallas, TX): census experiment on alternative question- • naire forms. Two small samples of households were interviewed using different schedules, specifically trying different methods for collect- ing information on income and place of work, on housing equipment items (e.g., type of heating and presence of air conditioning), and a 5-year versus 1-year migration question. April, June 1958 (Ithaca, NY): test conducted as part of special census • (with staff follow-up) on enumerators’ ability to classify types of living quarters. October 1958 (Martinsburg, WV): test intended to determine which • of three alternative population and housing questionnaires could be used most effectively in the field. The content of the questionnaires was identical, and essentially identical to the questions that would be used in the 1960 census, but the alternative schedules varied the size and structure of the schedules. The Martinsburg test also delivered

OCR for page 115
APPENDIX A 121 Advance Census Reports prior to enumerator visits. Because the test was sufficiently close to the actual census, the test also permitted the Bureau to evaluate the training materials planned for use in 1960 as well as editing and coding routines. • December 1958 (Philadelphia, PA): limited pretest to compare two possible forms with different skip patterns, after a final decision to use some self-enumeration in the 1960 census. • Mid-1959 (800 households in 10 regional headquarters cities): small- scale test following the full-dress exercise in North Carolina (described below), testing variants of some questions to determine whether adding check boxes to routing questions (e.g., “If you do not live in a trailer, check here and continue with the next question.”) promoted fuller response. Questionnaires were also distributed with return en- velopes for mailback. “Informal” Experiments and Pretests for the Census of Housing • June 1957 (Washington, DC): test of procedure for listing structures. • October 1957 (12 standard metropolitan areas): test of collection of data on condition of housing unit in selected 1956 National Housing Inventory segments. • March 1958 (New York, NY): test of classification of living quarters; used to formalize 1960 census definition of housing unit based on sep- arate entrance or separate cooking equipment. • March, May 1958 (Prince George’s County, MD): tests of questions on exterior materials of housing and on basement shelters. • June 1958 (Lynchburg, VA): recheck of data on condition and housing unit from May 1958 pretest. • September 1958 (Port Chester, NY): recheck of data on classification of living quarters. Dress Rehearsal • February–March 1959 (Catawba and Rutherford Counties, NC): final full-scale test prior to the census, making use of the Advance Census Report delivery that would be used in the 1960 count. Households chosen for the long-form sample were asked to complete their ques- tionnaires and mail them to their local census office. SOURCE: Adapted from U.S. Census Bureau (1966:App. B), with supple- mental information from Part III, Chapters 1 and 2 of that document.

OCR for page 115
122 ENVISIONING THE 2020 CENSUS A–2.b Research, Experimentation, and Evaluation Program A long list of possible research studies—based, in part, on input from the Panel of Statistical Consultants (an advisory group to the Census Bureau’s assistant director for research and development)—for the 1960 census was refined to a final list of 22 studies. The Bureau divided these 22 studies into 8 “projects.” A. Measurement of response variability • An expanded version of a study performed in 1950 to study vari- ability in response due to enumerators and census staff; the 1960 version of the “Response Variance Study I” drew sample from all areas where questionnaires were mailed (with enumerators sent to pick them up or conduct interviews), rather than four selected ar- eas as in 1950. The sample for which crew leaders and enumera- tors were assigned in order to estimate these effects included about 320,000 housing units and 1,000,000 persons. • A follow-up study, called “Response Variance Study II,” designed to measure variability in response due to the respondents themselves. A sample of 5,000 households from Response Variance Study I was drawn and enumerators sent to conduct interviews; a second sample of 1,000 housing units were asked to report again using a mailed, self-response questionnaire (with mailback, and interviewer follow-up if necessary). • “Response Variance Study III” looked at variability due to coding and data entry. For a one-fourth sample of households for which data was collected in Response Variance Study I, photocopies of the enumeration books were made; pairs of coders then independently coded and transcribed the data. B. Reverse record checks (undercoverage in general population): a check to see whether persons in an independent sample were enumerated in the 1960 census, in which the independent sample was culled from a mix of past census records and administrative data. Specifically, the sample was constructed from samples from four sources: (1) persons enumerated in the 1950 census; (2) immigrants and aliens registered in January 1960 with the Immigration and Naturalization Service; (3) birth records for children born between the 1950 and 1960 censuses; and (4) persons who were found in the 1950 postenumeration survey but not in the 1950 census. C. Reverse record checks II (undercoverage in specific groups): a check simi- lar to Project B, except that the records-based sample consisted of Social Security beneficiaries and students enrolled at colleges or universities.

OCR for page 115
APPENDIX A 123 D. Reenumerative studies of coverage error • Postenumeration survey based on an area sample: a sample of “segments” that had been independently canvassed by enumera- tors for the Survey of Components of Change and Residential Fi- nance (SCARF), conducted alongside the census. Provided with both 1960 census and SCARF information, enumerators in 2,500 area segments were tasked to search for omitted housing units or structures mistakenly labeled as housing units. About 10,000 hous- ing units were administered a detailed housing questionnaire. • Postenumeration survey based on a list sample: About 15,000 living quarters (both housing units and group quarters) already enumer- ated in the censuses was drawn, representing about 5,000 clusters of about 3 units each, dispersed across 2,400 enumeration districts. For each unit, a detailed reinterview was conducted in order to list persons within living quarters (this follow-up interview was con- ducted in early May 1960, so that not much time had passed since the April 1 Census Day). Enumerators were also asked to list “pre- decessor” and “successor” housing units or group quarters (e.g., neighboring units along a specified path of travel) in order to fur- ther check on missed housing units. E. Measurement of content error in data collection • Reinterview of about 5,000 households in the long-form sample for the 1960 census, using specially trained enumerators: a first phase (about 1,500 households in July 1960) sent the enumerators to conduct blind reinterviews; in the second phase (October 1960), dependent interviewing using the household’s reported 1960 cen- sus information was conducted for half of the remaining sample and for the other half blind interviewing, as in the first phase. • Reenumerative study focusing on housing characteristics, adminis- tering a battery of housing characteristics: about half of the 10,000 housing units in the study received the short form in the 1960 cen- sus, and the others were in the long-form sample (and hence had already reported some of the housing items). • Match of records from the 1960 census long-form sample and the Current Population Survey’s March or April 1960 samples. • Match of respondent-provided occupation and industry informa- tion with data collected directly from employers. • Match of about 10,000 sampled Internal Revenue Service returns to census records (although only about one-fourth of these were studied in depth regarding the consistency of reported income).

OCR for page 115
124 ENVISIONING THE 2020 CENSUS F. Studies of processing error • A September 1963 detailed review of a sample of individually filled Advance Census Reports, Household Questionnaires (enumerator schedules), and the coded computer-readable forms in order to es- timate transcription and other response errors. • Coding error study in which a set of long-form questionnaires were separately coded by three coding clerks but only one was the designated “census coder” whose work went onto the final questionnaire. • Study of the accuracy of automated editing rules in the microfilm– computer system for data collection and tabulation. G. Analytical studies: general studies of census quality and coverage, includ- ing comparison of census counts with demographic analysis estimates. H. Post Office coverage improvement study: postcensus postal check mak- ing use of postal employees and resources. A sample area of 10,000– 15,000 housing units was selected in each of the 15 postal regions in the continental United States. These areas were matched to census enumer- ation districts. Postal carriers were asked to review name and address cards completed by enumerators for every counted household, making new cards for households on their routes that were not included in the census. SOURCE: Adapted from U.S. Census Bureau (1963). A–3 1970 CENSUS A–3.a Principal Pretests and Experiments Conducted Prior to the Census Tests Conducted as Part of Special Census or Other Survey • August 1961 (Fort Smith, AR): test conducted as part of special census to compare an address register compiled through enumerator visits with a register based on 1960 census records, new building permits, and a postal check. The separate listings were “found to be just as complete.” Enumerator-visited households were also left with a brief questionnaire and asked to return the form by mail. Pretests of Census Operations and Questionnaires • June 1962 (Fort Smith, AR, and Skokie, IL): tests of address list updat- ing using building permits and postal checks (updating the Fort Smith list from the August 1961 special census and repeating the methodol- ogy in Skokie). Short questionnaires were mailed to all households for response by mail, yielding 71–72 percent return rates.

OCR for page 115
APPENDIX A 125 • April 1963 (Huntington, NY): further testing of address listing and mailout-mailback methods, now focused on a larger city—deemed to be a rapidly growing area and including a mix of urban, suburban, and rural housing types. The test also included administering a long-form questionnaire to approximately 25 percent of households. In nonmail delivery areas, address lists were built through enumerator canvass and questionnaires deposited at the households for mailback. • May–June 1964 (Louisville, KY, Standard Metropolitan Statistical Area [SMSA]): based on the Fort Smith, Skokie, and Huntington experi- ences, the Census Bureau requested and received funds for larger-scale testing of mailout-mailback methods (experimental censuses) in 1964 and 1965. The first of these was to be conducted in an area of about 750,000 population with a large central city; the Louisville area was selected (with the test being conducted from a special census office in Louisville, rather than the Bureau’s nearby processing center in Jef- fersonville, IN). The Louisville test added separate listing and contact strategies for “special places” (mainly group quarters), based on in- put from a task force on difficult-to-enumerate areas that had been convened in Louisville in late 1963. In this test, the Census Bureau concluded that a computer-generated address register based on pre- vious census records and building permits was more complete than listing books completed by enumerators. In particular, “using a list based on enumerator canvass did not seem the ideal way to insure complete coverage in multiunit structures in city delivery areas.” The Bureau further concluded “that at the current state of development of procedures the cost of a mail census, including certain important im- provements over the 1960 approach, would not exceed the cost of a census by enumerator canvass.” • April 1965 (Cleveland, OH): Cleveland was chosen as the site for the second experimental mailout-mailback census both for its big-city na- ture and for its perceived enumeration problems in the 1960 census. For this test, the Census Bureau experimented with using a commercial mailing list as the base for the address register; the test was also the first to be geocoded by computer based on an address coding guide (i.e., coded information on address ranges on odd and even sides of street segments). As in Louisville, both short- and long-form question- naires were distributed. The Cleveland test also centralized editing procedures, assigning completeness checks of returned questionnaires to district office clerks rather than individual enumerators. However, the Bureau’s attempts to predesignate hard-to-enumerate areas were deemed to be less successful than was the case in Louisville, and probe questions added to the questionnaire on other households at the same street address were generally found to be unclear and confusing. Fur-

OCR for page 115
170 ENVISIONING THE 2020 CENSUS determining final housing unit inventory, done by merging infor- mation on the processed Decennial Response File (census returns) with the Decennial MAF. – Beta Site: operational analysis of software evaluation facility within the Census Bureau that is responsible for integrating soft- ware systems of the census as well as to conduct security testing. – Cancelled: Invalid Return Detection. • Quality Assurance Evaluations (Series M) – Evaluation of Quality Assurance Philosophy and Approach Used for the Address List Development and Enumeration Operations: document operational experiences with quality assurance ap- proach. – Effectiveness of Existing Variables in the Model Used to Detect Discrepancies During Reinterview, and the Identification of New Variables: document specific quality assurance measure used in cases in which enumerators’ work suggested discrepancies. • A.C.E. Survey Operations (Series N) – Contamination of Census Data Collected in A.C.E. Blocks: study intended to determine success in keeping census and A.C.E. op- erations independent. – Discrepant Results in A.C.E.: examine how quality assurance steps identified interviewers who entered discrepant data in the A.C.E. interview. – Evaluation of Matching Error: assess error in the matching pro- cess used to identify missed or erroneously enumerated persons between the census and the A.C.E. – Targeted Extended Search Block Cluster Analysis: study result of expending search for possible matches from adjoining blocks for all sample clusters (as in 1990) to a broader search (still tar- geted to clusters judged most likely to benefit from additional searching. – Field Operations and Instruments for A.C.E.: assess quality of housing unit and person coverage in A.C.E. operations by exam- ining the quality of the (independent) address listing, effect of follow-up interviewing, and noninterview rates. – Cancelled (16, most converted to separate A.C.E. evaluation pro- gram): Analysis of Listing Future Construction and Multi-Units in Special Places; Analysis of Relisted Blocks; Analysis of Blocks With No Housing Unit Matching; Analysis of Blocks Sent Di- rectly for Housing Unit Follow-Up; Analysis of Person Interview With Unresolved Housing Unit Status; Analysis on the Effects of Census Questionnaire Data Capture in A.C.E.; Analysis of

OCR for page 115
APPENDIX A 171 the Census Residence Questions Used in A.C.E.; Analysis of the Person Interview Process; Extended Roster Analysis; Matching Stages Analysis; Analysis of Unresolved Codes in Person Match- ing; Outlier Analysis in the A.C.E.; Impact of Targeted Extended Search; Effect of Late Census Data on Final Estimates; Group Quarters Analysis; Analysis of Mobile Homes. • Coverage Evaluations of the Census and of the A.C.E. Survey (Series O) – Housing Unit Coverage Study: examine net coverage rate, gross omission rate, and erroneous enumeration rate of housing units, at various geographic levels as well as by A.C.E. poststrata. – Analysis of Conflicting Households: report on resolution of cases in A.C.E. housing unit matching when the census and the A.C.E. listed two entirely different families for the same unit. – Analysis of Proxy Data in the A.C.E.: study accuracy (match rates and erroneous enumeration rates) of data collected from proxy respondents: persons who are not members of the household, such as neighbors or landlords. – Housing Unit Duplication in Census 2000: study characteristics of duplicate housing units, attempting to identify census opera- tions most likely to produce housing unit duplication. – Analysis of Deleted and Added Housing Units in the Census Mea- sured by the A.C.E.: evaluate housing unit coverage on the early Decennial MAF. – Consistency of Census Estimates with Demographic Benchmarks: comparison of census results with demographic analysis bench- marks. – Cancelled (20, most converted to separate A.C.E. evaluation pro- gram or combined with other evaluations): Type of Enumeration Area Summary; Coverage of Housing Units in the Early Decen- nial MAF; P-Sample Nonmatches Analysis; Person Coverage in Puerto Rico; Housing Unit Coverage in Puerto Rico; Geocoding Error Analysis; E-Sample Erroneous Enumeration Analysis; Anal- ysis of Nonmatches and Erroneous Enumerations Using Logistic Regression; Analysis of Various Household Types and Long-Form Variables; Measurement Error Reinterview Analysis; Impact of Housing Unit Coverage on Person Coverage Analysis; Person Du- plication; Analysis of Households Removed Because Everyone in the Household Is Under 16 Years of Age; Synthesis of What We Know About Missed Census People; Implications of Net Cen- sus Undercount on Demographic Measures and Program Uses; Evaluation of Housing Units Coded as Erroneous Enumerations; Analysis of Insufficient Information for Matching and Follow-Up;

OCR for page 115
172 ENVISIONING THE 2020 CENSUS Evaluation of Lack of Balance and Geographic Errors Affecting Person Estimates; Mover Analysis; Analysis of Balancing in the Targeted Extended Search. • A.C.E. Survey Statistical Design and Estimation (Series P) – Cancelled (5, most converted to separate A.C.E. evaluation pro- gram): Measurement of Bias and Uncertainty Associated with Ap- plication of the Missing Data Procedures; Synthetic Design Re- search/Correlation Bias; Variance of Dual System Estimates and Adjustment Factors; Overall Measures of A.C.E. Quality; Total Error Analysis. • Organization, Budget, and Management Information System (Series Q) – Management Processes and Systems: study the Census Bureau’s organizational structure and decision-making processes, as well as its interaction with the Census Monitoring Board, Congress, the General Accounting Office, and other outside interests. • Automation of Census Processes (Series R); reviews of technical systems conducted by Titan Corporation – Telephone Questionnaire Assistance: toll-free service provided by a commercial phone center to answer questions about the census and the questionnaire. – Coverage Edit Follow-Up: program to resolve count discrepan- cies and obtain missing data for large households. – Internet Questionnaire Assistance: system that allowed respon- dents to use the Census Bureau’s Internet site to ask questions and receive answers about the census questionnaire or other census- related information. – Internet Data Collection: system that offered census short-form respondents the opportunity to respond via the Internet, using the 22-digit ID number found on their mailed census form. – Operations Control System 2000: system for control, tracking, and progress reporting for all field operations conducted for the census. – Laptop Computers for A.C.E.: systems used in A.C.E. follow-up interviewing. – A.C.E. Control System: tracking and control system behind the A.C.E. field operations. – Matching and Review Coding System for A.C.E.: system used to match A.C.E. returns to census records. – Pre-Appointment Management System/Automated Decennial Ad- ministrative Management System: system used for administrative management in the census, including tracking and processing of temporary enumerators, payroll, and background checks.

OCR for page 115
APPENDIX A 173 – American FactFinder: systems developed to provide access to census data via the Internet at http://factfinder.census.gov. – Management Information System 2000: systems used to manage census operations, including tracking of dates and budgets and creation of progress reports of current status during census oper- ations. – Data Capture: systems developed for full electronic data capture and imaging of census questionnaires, using optical mark and op- tical character recognition. SOURCE: Adapted from National Research Council (2004a:App. I). A–7 PRINCIPAL PRETESTS AND EXPERIMENTS CONDUCTED PRIOR TO THE 2010 CENSUS A–7.a Pretests of Census Operations and Questionnaires • 2002 (Gloucester County, VA): pilot testing of use of Census TIGER maps on a handheld computer (Pocket PC-class) by census interview- ers. The tests involved only locating particular address features on the small-screen map, not using the computer map to navigate a route or the collection of GPS coordinates. • February–April 2003 (national sample): the 2003 National Census Test was administered to a national sample of about 250,000 house- holds, drawn from the set of households that was enumerated us- ing mailout-mailback methodology in the 2000 census. Strictly mail- based, the 2003 test involved no field follow-up component. The test focused primarily on two issues: – Response mode and contact strategies: different experimental groups were offered the opportunity to reply by mail (traditional method), Internet, or interactive voice response (IVR, an auto- mated telephone system). Groups also varied as to whether these response modes were offered as a choice or whether they were “pushed” (e.g., providing Internet directions but no actual paper questionnaire in the mailing). Finally, contact strategies (includ- ing targeted replacement questionnaires and reminder postcards) were also varied. This component of the test involved eight ex- perimental groups, one with 20,000 households and the other seven with 10,000 households each. – Race and ethnicity (Hispanic-origin) question wording: seven treatment groups of 20,000 households each received different variations on the wording and arrangement of questions on race and Hispanic origin. Experimental settings included whether “some other race” was offered as a choice in the categories for

OCR for page 115
174 ENVISIONING THE 2020 CENSUS race, whether wording was slightly revised to ask respondents if they are “Hispanic” or if they are “of Hispanic origin,” and whether instructions explicitly directed respondents to answer both questions. The test was rounded out by a control group of 20,000 households; this group’s questionnaire included the race and Hispanic-origin ques- tions worded as they were in the 2000 census (unlike the 2000 census context, the 2003 test control group households were eligible for a replacement questionnaire in lieu of nonresponse follow-up). The samples for all groups were stratified by response rate in the 2000 census, in which the classification was a grouping into “high” and “low” response groups based on a selected cut-off. Martin et al. (2003:11) comment that the low-response strata “included areas with high proportions of Blacks and Hispanics and renter-occupied hous- ing units” and further comment that addresses in low-response areas were oversampled. Still, it is unclear whether the sample design gener- ated enough coverage in Hispanic communities to facilitate conclusive comparisons—that is, whether it reached enough of a cross-section of the populace and a sufficiently heterogeneous mix of Hispanic nation- alities and origins to gauge sensitivity to very slight and subtle changes in question wording. With regard to the response mode and contact strategy portion of the test, results reported by Treat et al. (2003) suggest that multi- ple response mode options may change the distribution of responses by mode—shifting some would-be mail responses to Internet, for ex- ample. However, the addition of choices does not generally increase cooperation overall. The experience of the 2003 test suggests serious difficulties with the interactive voice response option; 17–22 percent of IVR attempts had to be transferred to a human agent when the system detected that the respondent was having difficulty progressing through the IVR questionnaire. Moreover, rates of item nonresponse were greater for IVR returns than for the (paper response) control group. Internet returns, by comparison, experienced higher item re- sponse rates than the control. As indicated in past research, reminder postcards and replacement questionnaires had a positive effect on re- sponse. Martin et al. (2003) report that the race and Hispanic-origin ques- tion segment of the test showed mixed results. Predictably, elimination of “some other race” as a response category reduced “some other race” responses considerably, by 17.6 percent (i.e., Hispanic respondents apparently declined to write in a generic response like “Hispanic” or “other” if “some other race” was not a formal choice). The Bu-

OCR for page 115
APPENDIX A 175 reau concluded that the 17.6 percent decline in generic race reporting “more than offset” the impact of a 6.4 percent increase in the esti- mated number of Hispanics declining to answer the race question al- together (Martin et al., 2003:15). Adding examples of ancestry groups (e.g., Salvadoran, Mexican, Japanese, Korean) boosted the reporting of detailed origins among Hispanics, Asians, and Native Hawaiian or Other Pacific Islanders. Treatment groups for which instructions were revised, instructing respondents to answer both the race and Hispanic- origin questions, produced the most puzzling results; levels of missing data on one or both questions increased, as did the percentage report- ing themselves as Native Hawaiian or Other Pacific Islanders (relative to the control group). • February–July 2004 (7 neighborhoods of Queens County, NY [Asto- ria, Corona, Elmhurst, East Elmhurst, Jackson Heights, Long Island City, and part of Woodside]; Colquitt, Thomas, and Tift Counties, GA): the 2004 test was intended to test a package of new procedures and technologies in both a high-density urban site and a rural site. Lake County, IL, was originally designated a test site but was dropped due to constraints in funding for fiscal year 2004. Intended to include approximately 200,000 housing units, the test centered around a few major topics: – Handheld devices: the test marked the Census Bureau’s first at- tempt to use handheld computers, equipped with GPS receivers, for nonresponse follow-up interviewing. The handhelds devel- oped for use in the 2004 test were pieced together from com- mercial, off-the-shelf components; although the test included some advanced workflows (e.g., transmitting questionnaire data directly from enumerators’ devices to headquarters and pushing enumerator assignments directly to individual handhelds, with- out filtering through regional or local offices), some parts of case management and assignment were still done using paper reports. – Further testing of race and Hispanic-origin question wording: the 2004 test permitted continuation of work from the 2003 test. – Special place/group quarters definition: the 2004 test field exer- cises allowed Bureau staff to test the aptness of revised definitions of group quarters (nonhousehold) populations. In addition to dropping the Illinois test site, other planned parts of the 2004 test were eliminated from the test plan, including an attempt to target a mailing of dual-language (English and Spanish) questionnaires to certain households and to target canvassing methods for updating the MAF.

OCR for page 115
176 ENVISIONING THE 2020 CENSUS The 2004 field test also provided the first chance to test a ver- sion of the Bureau’s planned Coverage Follow-Up (CFU) operation, a combination of coverage improvement programs from previous cen- suses and computerized matching to detect potential census duplicates. Given the relatively small size of the test, the Bureau was able to per- form follow-up on all eligible cases and to conduct a post hoc clerical review of cases to specify the likely source of duplication. Follow-up interviews were conducted both by telephone (experienced interview- ers) and field staff (temporary, relatively novice interviewers). • July 2004 (France, Kuwait, and Mexico): Overseas Enumeration Test meant to assess the feasibility of a count of all Americans living over- seas, motivated by congressional interest. Interest in the issue had been heightened by one of Utah’s legal challenges to the 2000 census (having been edged for the 435th seat in the U.S. House of Represen- tatives by North Carolina) concerning the counting of missionaries of the Church of Jesus Christ of Latter-day Saints. To carry out the test, the Bureau adopted techniques similar to those used to elicit overseas resident counts in the 1960 and 1970 censuses: persons living over- seas had to make contact with a U.S. embassy or consular office in order to obtain the census questionnaire. Some publicity about the test in the selected countries (France, Kuwait, and Mexico) was made in English-language newspapers and media. Costing approximately $8 million, the test yielded very low response: 5,390 questionnaires total, compared to 520,000 questionnaires printed for the test and rough es- timates of on the order of 1.15 million American citizens in the test countries. Although the 2004 effort was originally intended to be re- peated in 2006, no funding for the overseas test was included in the Bureau’s appropriations for that year. • August–September 2005 (national sample): a second, mail-only Na- tional Census Test in 2005 involved only variations in questionnaire design. The test reached a sample of about 420,000 households and was intended to simultaneously address several major objectives: (1) test revised questions on tenure, relationship, age, date of birth, race, and Hispanic origin; (2) test respondent friendliness of new designs of mail and Internet questionnaires; (3) compare revised versions of the Question 1 household count item, including residence rules instruc- tions, and further test coverage probe questions selected from those tested in 2003; (4) test use of a replacement questionnaire; and (5) test use of a bilingual English-Spanish questionnaire. All of these objectives were intended to be covered by a set of 20 experimental treatment groups within a single-panel test; the control group included some items worded as they had been in the 2000 census, but other items used versions tested in 2003 or 2004. Although there was no field

OCR for page 115
APPENDIX A 177 follow-up, a sample of respondents was reinterviewed by telephone to assess the consistency of household rostering. Logistically, the test encountered problems as it coincided with the impact of Hurricanes Katrina and Rita and the disruption or suspension of mail service in the Gulf Coast area; the test was also complicated by the decision to keep the Internet questionnaire English-only and in a single for- mat that lacked the other experimental features, preventing Internet returns from being used in studying some treatments. • Early 2006 (national sample): ad hoc Short Form Mail Experiment with a planned sample size of about 24,000 addresses from the U.S. Postal Service’s Delivery Sequence File. Not originally part of the test- ing plan for the 2010 census, the Census Bureau sought authority for the test in a Federal Register notice in October 2005. As described in that notice (Federal Register, October 5, 2005, p. 58181), this special mailout test was developed based on three objectives: – “Evaluate the effects of the wording of the instruction about who to list as Person 1” on the questionnaire (the householder, with whom the reported relationships of other household members are defined); – “Evaluate the proportion of respondents who forget to enumer- ate themselves by asking them to provide their personal infor- mation at the end of the form” (with “personal information” described as name, phone number, and proxy status—that is, whether the respondent is completing a form for someone else); and – “Evaluate how a compressed schedule with a fixed due date im- pacts unit response patterns.” (The notice specified only that the “compressed schedule” would change from questionnaires being “mailed 2 weeks before ‘Census Day’ ” to “households receiv[ing] the questionnaires a few days before ‘Census Day.’ ”) The Census Bureau divided the sample into four treatment groups (Federal Register, October 5, 2005, p. 58181): – Group 1. Housing units in this treatment group will receive questionnaires with the same wording for the Person 1 in- struction that we used in the Census 2000 questionnaire. In the Final Question, respondents will be asked to provide their name, telephone number and proxy information. The mail out schedule will be the conventional schedule. The ques- tionnaire will be mailed two weeks before “Census Day”, and there will be no explicit deadline. – Group 2. Housing units in this treatment group will receive questionnaires with the revised wording for the Person 1 in-

OCR for page 115
178 ENVISIONING THE 2020 CENSUS struction. In the Final Question, respondents will be asked to provide their name, telephone number and proxy informa- tion. The mailout schedule will be the conventional schedule. The questionnaire will be mailed two weeks before “Census Day” and there will be no explicit deadline. – Group 3. Housing units in this treatment group will receive questionnaires with the revised wording for the Person 1 in- struction. In the Final Question, respondents will be asked to check over their answers before considering the survey com- plete. The mailout schedule will be the conventional schedule. The questionnaire will be mailed two weeks before “Census Day” and there will be no explicit deadline. – Group 4. Housing units in this treatment group will receive questionnaires with the revised wording for the Person 1 in- struction. In the Final Question, respondents will be asked to check over their answers before considering the survey com- plete. The mailout schedule will be compressed, so that the survey is received closer to “Census Day” and an explicit due date will be provided. • April 2006 (part of Travis County, TX, including the cities of Austin and Pflugerville; Cheyenne River Indian Reservation, SD): a second oper- ational test involving field work, the 2006 Census Test used mailout- mailback (with field nonresponse follow-up) in the Texas site and enu- merator visits (no mail) in the South Dakota site. As in 2004, handheld computers were used for enumerator interviews and were assembled from off-the-shelf components; in 2006, the new aspect of handhelds being tested was the delivery of maps to enumerators via the devices. The Census Bureau’s list of definitions of group quarters was expanded following the 2004 test, and the usefulness of this revised set was as- sessed in Travis County. A–7.b Dress Rehearsal • April 2008 (San Joaquin County, CA; nine-county area surrounding Fayetteville, NC [Chatham, Cumberland, Harnett, Hoke, Lee, Mont- gomery, Moore, Richmond, and Scotland Counties, including Fort Bragg and Pope Air Force Base]): as a full rehearsal of census activi- ties, the 2008 dress rehearsal actually began operations in spring 2007 with address canvassing. This activity was the first operational trial of custom handheld devices designed by Harris Corporation and its subcontractors under the Field Data Collection Automation (FDCA) contract; problems encountered in use of the handhelds precipitated a crisis in funding and a “replan” of FDCA and core census operations in the first half of 2008.

OCR for page 115
APPENDIX A 179 SOURCES: Adapted from “2010 Census: How We Prepare for 2010” (http://2010.census.gov/2010census/about_2010_census/007623.html); for the 2006 Short Form Mail Experiment, Federal Register, October 5, 2005, pp. 58180–58182; Hill et al. (2006); Karl et al. (2005); Knight et al. (2005); National Research Council (2004b:Boxes 9.1 and 9.2); Pennington (2005); Tancreto (2006).

OCR for page 115