–3–
Census Bureau Research, Past and Present

HAVING CONCLUDED IN CHAPTER 2 that serious attention to cost and quality must drive the planning for the 2020 census, we describe our recommendations in the following two chapters. In this chapter, we critique the Census Bureau’s existing program for research—exemplified by the 2010 Census Program of Experiments and Evaluations (CPEX)—both by comparison with the Bureau’s past efforts and through articulation of the gaps in its current strategies toward research. This chapter then provides general guidance for rethinking census research (and, with it, the approach to the 2020 census); Chapter 4 turns to the practical issues of structuring and scheduling research in advance of the decennial census.

Section 3–A presents our critique of current and recent trends by the Census Bureau in its research programs; it builds from the context provided by two detailed appendices at the end of this report. Appendix A describes the precensus testing programs and formal research (experimentation and evaluation) programs of the 1950–2000 censuses. It also describes the testing earlier in this decade related to the 2010 census. Appendix B then describes the 2010 CPEX program in detail. Section 3–B rounds out this chapter by laying out what we think are key steps in improving the substance of Census Bureau operational research.

It is important to note two caveats about the historical review of research in Appendix A (and, indeed, throughout this report). First, in summarizing research activities in the appendix, our concern is more in inventorying the types and varieties of activities that have characterized Census Bureau re-



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 59
–3– Census Bureau Research, Past and Present H C HAPTER 2 that serious attention to cost AVING CONCLUDED IN and quality must drive the planning for the 2020 census, we de- scribe our recommendations in the following two chapters. In this chapter, we critique the Census Bureau’s existing program for research— exemplified by the 2010 Census Program of Experiments and Evaluations (CPEX)—both by comparison with the Bureau’s past efforts and through ar- ticulation of the gaps in its current strategies toward research. This chapter then provides general guidance for rethinking census research (and, with it, the approach to the 2020 census); Chapter 4 turns to the practical issues of structuring and scheduling research in advance of the decennial census. Section 3–A presents our critique of current and recent trends by the Census Bureau in its research programs; it builds from the context pro- vided by two detailed appendices at the end of this report. Appendix A describes the precensus testing programs and formal research (experimenta- tion and evaluation) programs of the 1950–2000 censuses. It also describes the testing earlier in this decade related to the 2010 census. Appendix B then describes the 2010 CPEX program in detail. Section 3–B rounds out this chapter by laying out what we think are key steps in improving the substance of Census Bureau operational research. It is important to note two caveats about the historical review of research in Appendix A (and, indeed, throughout this report). First, in summarizing research activities in the appendix, our concern is more in inventorying the types and varieties of activities that have characterized Census Bureau re- 59

OCR for page 59
60 ENVISIONING THE 2020 CENSUS search and less in completely documenting their results. This is partly due to availability of information—particularly for the earlier censuses, method- ological details are scant in the literature—and partly because documenting the full “results” of major field tests such as census dress rehearsals is simply beyond the scope of our project. Our interest in the appendix and this chap- ter is in describing general contours and features of census research and not on assessing the merits (or lack thereof) of each specific activity. Second—and more fundamentally—we deliberately do not delve into the details of the coverage measurement programs that have accompanied the decennial censuses (save for formative developments in the earliest decades). We concur with our predecessor National Research Council panels that have found the Census Bureau’s coverage measurement programs to be generally of high quality. In particular, the Panel to Review the 2000 Census con- cluded that coverage measurement work in 2000 “exhibited an outstanding level of creativity and productivity devoted to a very complex problem” and that it showed “praiseworthy thoroughness of documentation and explana- tion for every step of the effort” (National Research Council, 2004a:244, 245). We also direct interested readers to the final report of another Na- tional Research Council (2008a) panel that had coverage measurement re- search as its explicit charge, which is not the case for our panel. Given our charge, we have focused our own analysis principally on research and development related to census operations; accordingly, it is important to note that our comments on “census research” in what follows should not be interpreted as applying to the Census Bureau’s extensive body of coverage measurement research. 3–A CURRENT RESEARCH: UNFOCUSED AND INEFFECTIVE In our assessment, the Census Bureau’s current approach to research and development (R&D), as it applies to decennial census operations, is unfo- cused and ineffective. The Census Bureau’s most recent research programs suffer from what one of our predecessor panels described as a “serious dis- connect between research and operations in the census processes” (National Research Council, 2004b:45): Put another way, the Census Bureau’s planning and research entities op- erate too often at either a very high level of focus (e.g., articulation of the “three-legged stool” concept for the 2010 census) or at a microlevel that tends toward detailed accounting without much analysis. . . . What is lacking is research, evaluation, and planning that bridges these two levels, synthesizing the detailed results in order to determine their im- plications for planning while structuring high-level operations in order to facilitate meaningful detailed analysis. Justifying and sustaining the 2010 census plan requires both research that is forward-looking and

OCR for page 59
CENSUS BUREAU RESEARCH, PAST AND PRESENT 61 strongly tied to planning objectives, and rigorous evaluation that plays a central role in operations rather than being relegated to a peripheral, post hoc role. In our assessment, the 2010 CPEX exemplifies these problems, although the problem is broader than the specific experiments, evaluations, and assess- ments outlined in Appendix B. As the quotation from the predecessor panel suggests, our critique applies to Census Bureau research writ larger, includ- ing the census tests conducted between 2000 and 2010 and the evaluations and experiments of the 2000 census. 3–A.1 Legacy of Research In Appendix A, we outline the precensus testing activities and formal research, experimentation, and evaluation programs of the 1950–2000 cen- suses. Together, they provide a picture of the Census Bureau’s major re- search programs on the decennial census, from the buildup to the 1950 cen- sus through the 2008 dress rehearsal for the 2010 census. We refer to spe- cific points in these narratives throughout this chapter, and also note some general impressions from the flow of census research over the years. The first such observation from past census research is that the lack of focus evi- dent to us in the Bureau’s current strategy was not always the case. Indeed, the Census Bureau has in the past been a place where major technological improvements and major data collection changes have been successfully exe- cuted through careful (but innovative) research and pathbreaking theoretical work. Arguably the best example of R&D driving change in the process of census-taking—a string of related research projects building toward a major operational goal—is the switch from enumerator-conducted personal visits to mailed, self-response questionnaires as the primary mode of data collec- tion. Now that mailout-mailback methodology has become so ingrained, it can be difficult to fully grasp how seismic a shift in methodology the change to mail was for the census. However, the magnitude of the shift can be inferred from chronicling the careful, deliberate program of testing and ex- perimentation that preceded the change. As the Census Bureau’s procedu- ral history of the 1970 census (U.S. Census Bureau, 1976) notes, mail was used for some Census Bureau activities as early as 1890—predating the es- tablishment of the Bureau as a permanent office. In 1890, questionnaires concerning residential finance were mailed to households with a request for mail return; the same was repeated in 1920, and a similar mail-based pro- gram of income and finance questions was also used in 1950. Supplemental information on the blind and the deaf was requested by mail in the 1910, 1920, and 1930 censuses, and a mail-based “Absent Family Schedule” was used for some follow-up work in 1910, 1930, and 1940. The direct path

OCR for page 59
62 ENVISIONING THE 2020 CENSUS to mail as the primary mode for census collection probably begins with the “Advance Schedule of Population” that was delivered to households in 1910; this form was meant to acquaint households with the topics of the census but was not meant to be completed by the householders (a similar advance form was used in the agriculture census conducted in 1910). Following World War II, and often in conjunction with special censuses requested by cities and towns, the Census Bureau initiated a set of experiments and tests of mailout or mailback methods; one such test was conducted in 1948 (Lit- tle Rock, AR; Section A–1.a), another as a formal experiment of the 1950 census in which households in Columbus, OH, and Lansing, MI, had ques- tionnaires distributed to them by enumerators prior to the 1950 census with instructions to complete and mail them on Census Day (U.S. Census Bureau, 1966:292; see also U.S. Census Bureau, 1955:5). Similar tests were per- formed in 1957, 1958, and 1959, with the January 1958 test in Memphis, TN, adding field follow-up of mailed census returns as a check on quality (Section A–2.a). In the 1960 census, households were mailed an “Advance Census Re- port,” which they were asked to fill out but not return by mail. Instead, enumerators visited the household to collect the forms and transcribe the information onto forms more conducive to the optical film reader then used to process census data. If a household had not completed the advance form, the residents were interviewed directly by the enumerator. Based on the successful use of the mailout questionnaire in 1960, Congress enacted a brief but powerful amendment to census law in 1964: P 88-530 struck .L. the requirement that decennial census enumerators must personally visit ev- ery census household. Even though mail methods had been tested and the required legal authorization had been obtained, mailout-mailback methods were subjected to further testing prior to the 1970 census, as Section A–3.a describes. These designed tests of mail procedures escalated in size and com- plexity from a relatively small community (Fort Smith, AR) to a large central city (Louisville, KY), to known hard-to-count areas (parts of Cleveland, OH, that experienced enumeration problems in 1960 and ethnic communities in Minnesota and New York). In the 1970 census, questionnaires were mailed to about 60 percent of all housing units, focusing on major urbanized ar- eas; a formal experiment conducted during the 1970 census (Section A–3.b) expanded mailout-mailback methods to more rural areas in 10 local offices, anticipating wider use of mail methods. The percentage of the population in the mailout-mailback universe has grown in subsequent censuses to include 81 percent of the population in 2000, with another 16 percent receiving questionnaires from census enumerators to be mailed back (see below). Other notable examples in which R&D (in the form of census tests, ex- periments, and evaluations) drove important developments in the census process include:

OCR for page 59
CENSUS BUREAU RESEARCH, PAST AND PRESENT 63 • Refinement of residence rules for college students: Prior to the 1950 census, the results of test questions in Current Population Survey (CPS) supplements and special censuses of cities contributed to the Bureau’s reversing its rule on counting college students. As described by the Na- tional Research Council (2006:Sec. 3–B.1), census practice since 1880 had favored counting college students at their parental homes. The test results prior to 1950 contributed to the conclusion that college students enrolled at schools away from home were frequently omit- ted in parental household listings, so the Census Bureau reverted to the 1850 census approach of counting college students at their school location. • Development of enumeration strategies for nonmail areas: In the 2000 census, blocks and other geographic regions were designated to be cov- ered by one of nine types of enumeration areas (TEAs)—essentially, the method for making initial contact with census respondents—with mailout-mailback being the most common TEA. Update-leave enu- meration for areas without city-style addresses, in which enumerators checked and updated address list entries during their visits but simply left a census questionnaire for respondents to return by mail, was first tested in a significant way in one of the experiments of the 1980 cen- sus; five matched pairs of district offices were selected for comparison using this technique (Section A–4.b). The March 1988 dress rehearsal in St. Louis, MO, added a variant of the strategy—urban update/leave, targeting hard-to-enumerate areas for personal enumerator visit—that was added to the 1990 census, evaluated, and judged to be a valuable technique (Sections A–5.a and A–5.b). The Bureau’s response to an un- foreseen problem in the same 1988 dress rehearsal also led to enduring changes in practice. Nine counties in the east central Missouri test site were initially thought to be amenable to mailout-mailback but a pre- census check suggested high levels of undeliverable addresses. Hence, the Bureau swapped strategies for such areas; the same flexibility in approach applied to the 2000 census, in which mailout-mailback con- version to update-leave was one of the TEAs. 3–A.2 Flaws in Current Census Research and the 2010 CPEX In this section, we briefly describe the principal deficiencies that we ob- serve in the Census Bureau’s current approach to research and in the 2010 CPEX in particular. In our assessment, shortcomings in the Census Bureau’s research strategy need to be overcome immediately in order to foster an effective research program for the 2020 census. However, at the outset of this discussion, it is important to note—and commend—revisions to 2010 census research that were announced as this

OCR for page 59
64 ENVISIONING THE 2020 CENSUS report was in the late stages of preparation. In congressional testimony in October 2009, Census Bureau Director Robert Groves (2009:9) described three research programs that he initiated following his own evaluation of 2010 census preparations: We will develop and implement a Master Trace Project to follow cases throughout the decennial census cycle from address listing through tabulation so that we have a better research base for planning the 2020 Census. We also will be conducting an Internet measurement re- interview study, focused on how differently people answer questions on a web instrument from a paper questionnaire. Finally, we will mount a post-hoc administrative records census, using administrative records available to the Census Bureau. All of this will better position us for the developmental work we must conduct to improve future decennial census operations. In committing to retain 2010 census operational data and to more aggres- sively evaluate the quality of administrative records data relative to census returns, the director’s proposed programs are responsive to recommenda- tions made in our letter report (Part III of this volume). The proposed Inter- net reinterview study stops short of testing Internet response in the census, but does at least put the Bureau in the position of testing Internet response to a census instrument. We commend these developments and look forward to their completion. We also note that they are also partially responsive to the recommendations and guidance in the balance of this chapter. That said, important problems remain in the Bureau’s general approach to research, as we now describe. Lack of Relevance to Cost and Quality Issues Although effects on cost and quality were listed by the Bureau as primary criteria for choosing studies for the 2010 CPEX, the final slate of experi- ments and evaluations described in Appendix B and analyzed in our letter report (Part III of this volume) seem ill-suited to inform choices that would meaningfully affect either cost or quality. Of the experiments: • Only the nonresponse follow-up (NRFU) Contact Strategy Experiment appears clearly motivated by an attempt to reduce the cost of a high- cost operation without impairing census quality. However, even that experiment promises to stop short of providing comprehensive infor- mation on the cost-benefit trade-offs of suspending follow-up contacts after some number (more or less than 4–6) of attempts. Because enu- merators will know in advance how many attempts are possible for a household, the experiment presents the opportunity for enumera- tors to game the system: to try particularly hard in early approaches at a 4-contact household or be slightly more casual in early attempts

OCR for page 59
CENSUS BUREAU RESEARCH, PAST AND PRESENT 65 at a 6-contact household. The experiment may provide insight into how enumerators deal with preset rules and conditions but not a true measure of NRFU yields at each possible contact opportunity. • The Deadline Messaging/Compressed Schedule Experiment may rightly be said to have some bearing on census costs, to the extent that it helps provide mailing package cues and prompts that may boost the mail return rate (and thus reduce the more costly NRFU workload). The compressed schedule portion of the experiment could be argued to promote higher quality by pushing data collection closer to the actual census reference date of April 1, but the impact of a one-week shift on the quality of resulting data is most likely negligible.1 • The Alternative Questionnaire Experiment (AQE) is heavily focused on refinements to the measurement of race and Hispanic origin— important data items to be sure, but ones for which an objective truth is both unknown and unknowable, subject as it is to individual concepts of self-identity. Hence, the experiment may suggest whether differ- ent treatments yield different levels of reporting in specific categories, but it is impossible to say whether “different” is the same as “higher quality.” By comparison, only a single panel in the AQE focuses on the quality of information about residence and household count—the information that represents the constitutional mandate for the census. • As we noted in our letter report, the Confidentiality/Privacy Notifi- cation Experiment is, if anything, contrary to the goals of reducing cost and improving quality. Its paragraph treatment raises the possi- bility of mixing census information with data from other government agencies—i.e., the use of administrative records in census processes— in an ominous manner. Since the experiment includes only the single alternative wording, it creates a situation where respondents may re- act negatively but relatively little is learned about public sensitivity to records use. Undue Focus on “Omnibus” Testing Slots One observation that is clear from comparing previous census research programs (e.g., Sections A–1.a–A–4.b) with the 2000 and 2010 research pro- 1 A treatment group in the 2006 Short Form Mail Experiment (see Section A–7) that used a compressed mailing schedule showed similar levels of nonresponse to questionnaire items on housing tenure, age, Hispanic origin, race, and sex compared with a control group. The com- pressed schedule group had a statistically significant difference (decrease) in leaving the total household count blank compared with the control, and also appeared to increase reporting of new babies and reduce the tendency for respondents to omit themselves from the questionnaire (Martin, 2007), although how these effects are specifically generated by a one-week difference in questionnaire mailout is unclear.

OCR for page 59
66 ENVISIONING THE 2020 CENSUS grams is that the Bureau used to be considerably more flexible in the forms of research studies it undertook. Small, targeted tests in selected sites used to be more frequent; again, the example of the final gear-up to mailout- mailback methodology in 1970 (Section A–3.b) is instructive, with a series of tests escalating in size and scope from small communities to dense urban centers. The Census Bureau also made considerable use of special censuses commissioned by individual localities as experimental test beds; costs of the tests were thus shared by the Census Bureau and the sponsoring locality, and the locality had a tangible product—fresh population data—as an incentive for cooperating with the experimental measures. The use of special censuses for such purposes seems to have ended—perhaps understandably so—when the city of Camden, NJ, sued the Census Bureau over the results of a Septem- ber 1976 test census in that city, occasioning several years of legal wrangling (Section A–4.a). In the past, the Census Bureau was also more willing to use other surveys for testing purposes—particularly the Bureau-conducted Current Population Survey, which was used to test items for the 1950 and 1960 censuses. In the most recent rounds of research and experimentation, selected studies seem to have been chosen more based on the availability of testing “slots” (e.g., something that could be tested using only the mail as part of a single, omnibus, mail-only experiment in a year ending in 3) than on loom- ing questions and operational interests. The recent cycle of mail-only tests in years ending in 3 or 5, tests involving a field component in years 4 and 6, and a dress rehearsal in year 8 has the advantage of keeping the various parts of census processing in fairly constant operation, so that there is no need to completely rebuild field operations from scratch. But a too-strong focus on these single-shot testing slots has led to poor design choices. For example: • The National Research Council (2004b:227) argued that the Census Bureau’s decision to fuse a test of alternative response technologies (i.e., paper, Internet, or telephone) to a mail questionnaire with nu- merous modules on race and Hispanic origin question wording in the 2003 National Census Test “was likely one of convenience” rather than one intended to produce meaningful results. The availability of a nationally representative sample seemed to have trumped attention to “power analysis to determine the optimal sample sizes needed to mea- sure effects to desired precision” or “more refined targeting of pre- dominantly minority and Hispanic neighborhoods” where the revised race questions would provide the most information.

OCR for page 59
CENSUS BUREAU RESEARCH, PAST AND PRESENT 67 • Early plans for the mail-only 2005 National Census Test included ex- perimental panels of different presentations of the instructions and wording of census Question 1 (household count). However, those early plans failed to include any relevant control group—either the question as presented in the 2000 census or the modified question used in a 2004 test—making it impossible to judge effectiveness of the experimental treatments compared with a baseline. After this defi- ciency was identified at a meeting of the Panel on Residence Rules in the Decennial Census, the test plan was altered to include a control (National Research Council, 2006:205). In a regime of large omnibus tests, topics that might best or more accu- rately be handled in a series of smaller, focused tests are forced into a larger design, without promise that the omnibus test will be able to distinguish be- tween fine-grained alternatives. Those omnibus census tests that involve a field component also suffer from an important limitation. They are meant to be census-like to the greatest extent possible in order to utilize the com- plete census machinery. But an explicit proviso of the modern census tests is that no products are released, most likely to maintain consistency with the Bureau’s reluctance in recent decades to use locally sponsored special cen- suses as experimental opportunities. But the result of this practice is a major operational test that is “census”-like save for the fact that it is not actually a census: such trials provide participating localities with no tangible product or benefit. Try though the tests do to create census-type conditions, localities have little incentive to provide unfettered support other than a sense of civic duty. Finally, the shift in recent years to omnibus tests has created another fundamental flaw in Census Bureau research: almost of necessity, the tests can not build from each other. In previous decades, “chains” of related tests can be seen. For instance, the major census tests in Yonkers, Indianapolis, and Memphis in 1957–1958 (Section A–2.a) all involved use of a two-stage interview process, with another enumerator or supervisor rechecking results; based on experience in one of the tests, approaches in the later tests were varied. By comparison, the large-scale tests of recent years take longer to design, longer to field, and longer to analyze—and leave few resources for subsequent tests. With few exceptions, the results of recent census tests have been unable to follow directly from the experience of their predecessors simply because the results of the earlier tests had not been processed. Failure to Utilize Current Methods in Experimental Design A criticism related to the increased reliance on a smaller number of large, omnibus tests is a lack of attention to some fundamentals of experimental design. In an attempt to be all-inclusive, the experimental designs of recent

OCR for page 59
68 ENVISIONING THE 2020 CENSUS decennial census experiments—including those of the 2010 CPEX—do not take proper account of factors affecting the response and strength of the expected treatment effect, and, as a result, the findings from some experi- ments have been inconclusive. As we have already noted, the 2003 National Census Test fused together two broad topics—alternative response method- ologies and variants on race and Hispanic origin questions—mainly to fill topic “slots” on the planned mailout-only test. Combining the two and dis- tributing the sample led not only to the comparison of extremely subtle treatments in the race and Hispanic-origin segments, but also to the omis- sion of relevant treatment groups on the response method portion (i.e., a treatment to “push” Internet use exclusively, instead of a group encouraging either Internet or telephone response). There are several commonly used techniques that the Census Bureau does not typically employ in its experiments and tests that could provide im- portant advantages over the methods currently used. For example, fractional factorial designs are extremely useful in simultaneously testing a number of innovations while (often) maintaining the capability of separately identifying the individual contributions to a response of interest. Such a methodology would be well suited to the problem of census innovation, since there are typically a small number of replications and a relatively large number of factors being modified. This problem is very typical of the large census tests that often need to examine a number of simultaneous changes due to limited testing opportunities. The problem of confounding is well known, yet there are examples of experiments carried out by the Census Bureau, either during the decennial census or during large-scale census tests, in which the experiments have gen- erated very uncertain results due to the simultaneous varying of design fac- tors in addition to those of central interest. For example, the ad hoc Short Form Mail Experiment in 2006 (see Section A–7) took as a main objec- tive determining whether a compressed mailing schedule and specification of a “due date” hastened response, but—by design—the experiment treated deadline and compressed scheduling as a single, combined factor and so could not provide insight as to which change was most effective.2 The pro- posed Deadline Messaging/Compressed Schedule experiment in the 2010 CPEX shows similar features and flaws. The message treatments shown in Table B-1 test subtle variations of an appeal made in a short paragraph in a cover letter and reminder postcard with blunter changes made to other parts of the mailing package. Whether a quicker response is due to the appeal to save taxpayer funds by mailing the questionnaire or to the explicit “Mail by 2 Describing the design, Martin (2007:12) argues that, “practically speaking, it is not fea- sible to implement a deadline without also moving up the mailing schedule. It does not make much sense to provide a deadline that is in the distant future, nor does it make sense to have a deadline that is before Census Day.”

OCR for page 59
CENSUS BUREAU RESEARCH, PAST AND PRESENT 69 April 5” advisory—printed on the outside envelope of all the experimental treatments—is a question that the experiment will not be able to answer. More fundamentally, experiments and census tests are rarely sized through arguments based on the power needed in support of the statisti- cal tests used to compare alternatives. As noted in Section B–1.c, the cri- tique in our letter report of the lack of a power analysis for the 2010 CPEX experiments—particularly the Deadline Messaging/Compressed Schedule experiment—was answered by the Census Bureau with an appeal to two internal memoranda and an arbitrary doubling of the sample size, with no insight as to how either the original or doubled sample sizes had been de- rived (U.S. Census Bureau, 2009a). All of this argues for greater attention to standard techniques of statistical experimental design in the planning of census experiments and intercensal tests. It follows that making improvements in the Bureau’s experimental design and testing areas depends on bolstering the technical capability and research leadership of its staff; see Chapter 4 for further discussion of such organiza- tional features. Lack of Strategy in Selecting and Specifying Tests, Experiments, and Evaluations In addition to not providing direct information on cost and quality, the experiments and evaluations in the 2010 CPEX show little sign of antic- ipating or furthering future methodology that could yield a 2020 census that is reflective of contemporary attitudes, technologies, and available in- formation sources. The choices in the 2010 CPEX seem more suggestive of a “bottom-up” approach—looking at highly specific parts of the census process and making small adjustments—than a more visionary “top-down” approach that takes major improvement in the cost-effectiveness of the cen- sus (and such wholesale change in operations as is necessary to achieve that improvement) as a guiding principle.3 Such a top-down approach would be predicated on alternative visions for the conduct of a census—general di- rections that might be capable of significant effects on census cost or quality. Then, even though an experiment in the 2010 census would not be capable of fully assessing such a vision, the topics for experimentation would relate to those visions: chosen strategically, they could provide bits of preliminary information to guide later work over the course of the subsequent decade (or decades). 3 Labels like “top-down” or “bottom-up” approaches to planning are, necessarily, oversim- plifications. Both approaches—as pure strategies—have value, and the ideal is undoubtedly some combination; we use the labels here to be evocative and to suggest a recent focus on small iterations.

OCR for page 59
84 ENVISIONING THE 2020 CENSUS • Position the MAF as a national resource: The MAF benefits other federal government agencies through the major demographic surveys (done under contract with the Census Bureau) that use the MAF as a sampling frame. It is also developed, in some respects, in partnership with the U.S. Postal Service because the postal DSFs are a major in- put source to the MAF. It stands to reason, then, that a useful area of research concerning the MAF would be whether it satisfies the needs of its major stakeholders and insights that other agencies may have on the frame-building process. In the case of the Postal Service, our re- view of past decades’ research programs—replete with intensive use of postal checks and use of information collected directly from local letter carriers—is a reminder that establishing and maintaining a re- search partnership with the Postal Service is vital. For example, it should be determined whether the Census Bureau’s adaptation of the regular DSF updates makes use of the full range of address informa- tion on those databases and whether other postal data could further improve the MAF or TIGER. It should be noted that broader use of the MAF would be likely to require action by Congress (akin to the 1994 act that permitted both LUCA and DSF updating) because of the U.S. Supreme Court’s interpretation that Census Bureau address lists fall under the confidentiality provisions of Title 13 of the U.S. Code (Baldrige v. Shapiro, 455 U.S. 345, 1982). • Interface with the commercial sector: Just as it is important to com- pare the address coverage from compiled administrative records files with the existing MAF, it would also be worthwhile to study the qual- ity and coverage of commercially available mailing lists (even if such commercial lists do not become an input source to the MAF). In partic- ular, it would be useful to learn from private-sector practices in build- ing, maintaining, and filtering mailing lists, as well as how private- sector firms have developed other frames, such as e-mail and telephone listings. • Assess integration of the “household” MAF with group quarters: One of the improvements promised in the MTEP was the merger of MAF, TIGER, and group quarters information into a common database structure. For the 2010 census, group quarters validation will still be done as a separate operation, following up with structures and fa- cilities labeled as “other living quarters” in the complete address can- vassing operation. Study of group quarters validation will be useful for judging the effectiveness of the merger of these lists and the ability of flags in the address records to distinguish between group quarters and regular household populations. This kind of study and research is par- ticularly important given the sometimes blurred line between conven-

OCR for page 59
CENSUS BUREAU RESEARCH, PAST AND PRESENT 85 tional household and group quarters structures, such as group homes and health care facilities that may combine “outpatient,” independent living, assisted living, or full-time nursing care functions within the same structures. Better and Less Expensive Data Collection: Toward a “Paperless” Census An original goal of the incorporation of new technology into the census process for 2010 was to reduce the use, storage, and movement of paper in the census. The use of paper translates directly to both cost and time; while the use of document scanning greatly reduces the time that needs to be spent handling the paper (keying information directly from forms), the reliance on paper at all stages has serious implications for the size and scope of the local census offices and data capture centers that must be equipped for the census. The use of technology is an area in which the 2010 census is likely to be remembered for some strides but probably more for the costly and embarrassing collapse of the plans for use of handheld computers in NRFU interviewing and reversion to paper-and-pencil methods. That the 2010 census will fall short of its original goals for reducing pa- per use is not a failure of vision—the goal was a good and laudable one—but a failure to execute and fully articulate that vision. The idea itself was not enough to guarantee success: the idea had to be matched by a set of re- search and testing activities designed to propel the larger task of technology development forward and specify the requirements for the technology. Going forward, it is difficult to imagine a plan for the 2020 census that can substantially reduce costs or increase quality without a major emphasis on developing and integrating new technology. As a bold statement of mis- sion, we encourage the Census Bureau to go further than to think of simply getting the development process of handheld computers for NRFU in shape. Rather, we suggest a broader examination of all steps in the census process with the public, stated goal of making the 2020 census as “paperless” as is practicable. Further reasons why an effort to move the census in a paperless direc- tion is a critical strategic issue for 2020 include the implications for quality. Indeed, we think that experience in the general survey research community suggests that the gains in accuracy from electronic methods for data collec- tion may be more important than cost reductions. The Census Bureau’s own work in the 2003 and 2005 tests suggested that Internet responses were typi- cally of higher quality than responses by other modes; edit routines and skip patterns in electronic questionnaires can promote accuracy in subtle but im- portant ways. Secondary gains in accuracy are not hard to imagine. Nonpa- per formats would not have the same hard space limits as paper forms, thus reducing the need for follow-up with large households for whom reported

OCR for page 59
86 ENVISIONING THE 2020 CENSUS information simply does not fit on the paper form. Questionnaires could also be made directly accessible in a wide variety of foreign languages, with- out the strong filter of recent censuses in which a call to an assistance cen- ter was necessary to request a foreign language form. The age distribution and attitudes of the population also make a higher tech, relatively paperless census a key strategic issue; new generations are arguably more conversant with electronic media than paper media, and a “green” census (saving paper) might serve as a strong incentive to boost participation. However, one of the strongest arguments for a heightened focus on use of technology leading to the 2020 census is simple perception, exactly the reason why the 2010 cen- sus looks odd relative to other national censuses and surveys that are now turning toward Internet data collection. That is, it would simply look foolish and out of step to try to force 2020 census technology into the 2010 mold rather than aggressively studying and rebuilding systems. The guidance by the Panel on Research on Future Census Methods in its final report (National Research Council, 2004b) on developing and im- plementing a technical infrastructure remains valid. It also follows that movement toward census processes that are highly automated and as pa- perless as possible heightens the importance of ensuring that those processes have an audit trail—that they include outlets for retention, archival, and analysis of operational data such as we recommend for the 2010 census in Section 3–B.3. Having already described many of the points raised in that report, we do not expound on them further. In brief, some other selected issues and possibilities for technology research over the next decade include: • Boosting response using paperless methods: One of the valid arguments raised by the Census Bureau for not permitting online response in the 2010 census is that their experience suggests that permitting electronic response options along with a paper questionnaire does not seem to el- evate overall response. That is, it does not seem to produce original re- spondents: it may sway some people who might otherwise respond by mail to return the form electronically, but it does not convert probable nonrespondents to respondents. This observation is not unique to the Census Bureau’s experience; other national statistical offices and sur- vey research organizations have encountered the same phenomenon. Developments in questionnaire design and approach strategies should be pursued by the Bureau in cooperation with these other groups. • Security and confidentiality concerns: In overcoming the concerns about computer and network security that led it to disallow online response in 2010, the Census Bureau would benefit from in-depth study of the security mechanisms used in other censuses and surveys. It would also benefit from examples in the electronic implementation of other government forms, such as tax returns.

OCR for page 59
CENSUS BUREAU RESEARCH, PAST AND PRESENT 87 • Mode effects: A perennial concern in survey methodology with the adoption of new survey response types is the difference in consistency of response across the different types. Response mode differences are not inherently good or bad but need to be understood and doc- umented. Better and Less Expensive Secondary Contact: Nonresponse Follow-Up Reexamining assumptions and strategies for NRFU operations is a key strategic operation because of the significant costs of mobilizing the massive temporary enumerator corps and making contacts at households that, for whatever reason, do not respond to lower cost initial contacts. Research on the possible role of administrative records in NRFU processes is particularly critical to achieving better and less expensive secondary con- tact with respondents. With some additional evaluative and follow-up com- ponents, the telephone-based 2010 CFU operation could provide some use- ful insight to start such research. One of the possible planned sources of household or address records being submitted to the CFU operation is a search of census returns against the Census Bureau’s database of adminis- trative records compiled from other federal agencies, a database currently known as StARS (Statistical Administrative Records System) or e-StARS. As we have also described, the retention of operational and procedural data during the 2010 census also has the potential to yield very valuable information; these data snapshots should be able to support a post hoc examination—as a research question—of the impact on census costs and quality if administrative records had been used as supplemental enumera- tions at various stages of NRFU work. All stages—from near-original enu- meration of nonresponding households to use of records as a last resort measure rather than proxy information—should be considered and investi- gated. To be clear, the use of administrative records should not be seen as a panacea for all census problems, and we do not cast it as such. Sheer nu- meric counts aside, the quality and timeliness of administrative records data for even short-form data items, such as race and Hispanic origin and rela- tionship within households, remain open and important questions. Wider use of administrative records in the census also faces formidable legal hur- dles, not the least of which are inherent conflicts between the confidentiality and data access provisions in census law (Title 13 of the U.S. Code) and the Internal Revenue Code (Title 26), given the prominence of tax return data in the administrative files. Still, just as it is difficult to imagine a 2020 planning effort that seriously addresses cost and quality issues without ag- gressive planning for use and testing of new technology, it is also difficult

OCR for page 59
88 ENVISIONING THE 2020 CENSUS to imagine such an effort without a meaningful examination of the role of administrative records. Other key strategic issues for NRFU-related research include: • Investigation of state and local administrative data sources: As men- tioned above, the Census Bureau’s current StARS administrative records database is built annually from major administrative data sources maintained by federal agencies, including the Internal Revenue Service. Particularly as the idea of using administrative records in a va- riety of census operations (such as geographic resource updates) is con- sidered, the Census Bureau should explore the quality and availabil- ity of data files maintained by state and local governments, including local property files, records for “E-911” conversion from rural non- city-style addresses to easier-to-locate addresses, and state and county assessors’ offices. • Optimal pacing and timing of contacts: The NRFU Contact Strat- egy Experiment of the 2010 CPEX varies the number of contacts allowed for nonresponding households. As we have already noted, there is something slightly off in the specification of the experiment— capping the number of visits and making this known to the interview- ers presents opportunities for gaming the system. But the optimal number of attempted NRFU contacts—based on yields of completed interviews and quality of information—is an important parameter to resolve. So, too, is work on best ways to structure local census of- fice and enumerator workload in order to maximize the chances of successful interview completion. • Efficacy of telephone follow-up: Along the same lines, a research ques- tion that has been touched on in past census research but that is worth revisiting in the 2020 climate is use of telephone (or electronic) follow- up rather than personal visit. The effectiveness of the telephone-based CFU operation in 2010 may provide initial insight on the feasibility of conducting such operations on an even larger scale. Much may also be learned about the effectiveness of telephone-based follow-up in the census context by studying and evaluating its use in the ACS and other Census Bureau surveys. • Reducing NRFU workload by shifting some burden: Clearly, a crit- ical determinant of the cost of NRFU operations is the number of nonresponding households that are followed up. The U.S. Supreme Court’s 1999 decision and current wording of census law reinforce that reducing that workload by following up with only a sample of households is not permissible. But research efforts on another an- gle to cut into the overall NRFU workload—promoting late response to the census—may be worthwhile. This could involve, for example,

OCR for page 59
CENSUS BUREAU RESEARCH, PAST AND PRESENT 89 extending publicity and outreach campaigns (and possibly some shift in message—emphasizing that “it isn’t too late” to respond) and “Be Counted”–type programs; such a message was employed in the 1980 census (see Table 2-5). The effectiveness of such an approach would depend on the level of automation of census operations (e.g., the abil- ity to transmit quickly revised enumerator assignments) and the time demands for data capture from paper forms. Still, the costs and ben- efits are worth exploring—these efforts might not sway some truly hard-to-count respondents, but they could elicit responses from some reluctant or forgetful households. • Examining relative quality of “last resort” enumeration: In those cases in which contact simply cannot be made, the relative quality of dif- ferent options for filling the blanks—for example, proxy information, imputation, and use of administrative records—should be quantified and evaluated. • Quality of interviews as a function of time from Census Day: It is gen- erally well understood that follow-up interviews (as well as indepen- dent interviews such as the postenumeration survey that is the heart of coverage measurement operations) are best done as close as possible to the census reference date. Doing so helps to curb discrepancies due to people moving to different households or switching between “per- manent” and seasonal residences. It is also generally well understood that there is decay in interview quality and consistency with length of time from the survey. This is arguably more an issue for time-sensitive information (i.e., exact knowledge of monthly utility bills) or recall of numerous events than for the items on a short-form-only census. Still, a body of quantitative evidence on recall and decay effects on short-form items (including number of persons in and composition of the household)—and key long-form items currently collected on the American Community Survey—as they vary with time from Census Day would be very useful in revisiting such assumptions as optimal timing of determining the start of NRFU operations. Rethinking the Basic Census Objective: Getting Residence Right The basic constitutional mandate of the decennial census is to provide an accurate resident count. Accordingly, in terms of setting basic strategy for the 2020 census, the concept of residence and collecting accurate resi- dence information is vitally important. Census residence rules and concepts have been easy to pigeonhole as a questionnaire design issue—the search for the right phrasing and ordering of words and instructions at the start of the census form in order to prod respondents to follow particular con- cepts. These questionnaire design matters are important, but the issues are

OCR for page 59
90 ENVISIONING THE 2020 CENSUS much broader—a thorough examination of residence merits attention to the basic unit of analysis of the census, to the implications of residence con- cepts for data processing design of operations, and to tailoring enumera- tion approaches to different levels of attachment to a single “usual” place of residence. The National Research Council (2006) Panel on Residence Rules in the Decennial Census discussed a wide range of research ideas under the gen- eral heading of residence; these generally remain as applicable to the 2010– 2020 planning period as they were to 2000–2010. In terms of questionnaire design, these include further research on replacing the current instruction- heavy approach to the basic household count question with a set of smaller, more intuitive questions and more effective presentation of the rationale for the census and specific questions. Other specific research suggestions (many of which draw from the same panel’s recommendations) include: • Quality of facility records for group quarters enumeration: Residence concepts and group quarters enumeration are impossible to disentan- gle because the nonhousehold, group quarters population includes major cases in which determination of a single “usual” residence is difficult: college students away from home, persons under correc- tional supervision, persons in nursing or other health care facilities, and so on. Hence, attention to getting residence concepts right de- mands attention to methods for enumerating group quarters. In the 2000 census, about half of the returns for the group quarters popula- tion were filled through reference to administrative or facility records rather than direct interview. Yet much remains unknown about the accuracy and capability of facility records and data systems to fill even the short-form data items, let alone their ability to provide the kind of alternative residence information that would be necessary to inform analysis of census duplicates and omissions. The National Research Council (2006:240) suggested that the Census Bureau study the data systems and records of group quarters facilities on a continuous ba- sis, akin to its efforts to continuously update the MAF. This is clearly a major endeavor, but one that is particularly important because of the inclusion of group quarters in the ACS. Even the records systems of state-level correctional systems will vary, let alone the plethora of systems maintained by individual colleges or health care facilities. But research toward a continuous inventory would benefit other surveys of group quarters populations between censuses and may suggest meth- ods for more efficient and accurate data collection from the relatively small but very policy-relevant group quarters population. • Residence standard mismatch between the census and the ACS: While the decennial census uses a de jure “usual residence” concept, the long-

OCR for page 59
CENSUS BUREAU RESEARCH, PAST AND PRESENT 91 form-replacement ACS uses a 2-month rule—effectively, a de facto or “current residence” standard. The exact ramifications of this differ- ence in residence standards are unknown, and, indeed, they may be relatively slight, particularly given the pooling of ACS data to produce multiyear average estimates. But more precise empirical understanding of the possible differences introduced by differing residence standards in the Census Bureau’s flagship products would bolster the ACS’s cred- ibility. In the next chapter (and in Recommendation 4.5 in particular) we discuss the need for integration of research between the decennial census and the ACS; a matching study of the census to ACS returns near April 2010 would be an ideal step in that regard to study possible differences in residence specifications and household rostering. • Revamping “service-based enumeration”: In the 2000 census, the Cen- sus Bureau’s efforts to count transient populations, including per- sons experiencing homelessness, were combined into an operation known as service-based enumeration. This operation—to be repeated in 2010—relies principally on contacts at locally provided lists of shel- ters and facilities providing food or temporary shelter services. Just as group quarters enumeration would benefit from sustained research effort and attention over the decade, so, too, would outreach efforts to best cover the service-based population. Such effort should include collaboration with subject-matter experts, local government author- ities, private service providers, and other agencies (such as the U.S. Department of Housing and Urban Development) that have periodi- cally attempted measures of the number of homeless persons at points in time. It should also include focused, relatively small surveys to com- pare the efficacy of sample-based measures of the homeless population compared with census-type canvasses. • Revisiting the “resident” count: Because of the legal climate surround- ing the 2010 census, the Census Bureau may face pressure to conduct research on components that are currently included or not included in the census “resident” count. It should prepare accordingly. In partic- ular, its research program should give some thought to studying the effects on response and cooperation by including questions on citizen- ship or immigration status. The arguments that such questions could seriously dampen response and hurt the image of the decennial census as an objective operation are straightforward to make and, we think, are basically compelling, but empirical evidence is important to build- ing the case. With regard to counting Americans overseas, the experi- ence of the 2004 Overseas Enumeration Test is very useful, but here, too, additional quantitative evidence would be useful. In particular, it would be useful to examine and critique information resources that

OCR for page 59
92 ENVISIONING THE 2020 CENSUS may be available from the U.S. Department of State (e.g., contacts with overseas embassies or consulates) to estimate the level of coverage in such files; it would also be useful to evaluate and assess the quality and timeliness of the data files that the Census Bureau already uses from the U.S. Department of Defense and other federal agencies with employees stationed overseas (for inclusion in apportionment totals). Planning Now for the Census Beyond 2020 The history of past census research that we have outlined in Ap- pendix A and described in this chapter—particularly the successful adop- tion of mailout-mailback methods—suggests that truly massive change in approach to the census can take decades of planned research to be fully re- alized. It is vitally important in 2009 and 2010 for the Census Bureau to be thinking of and planning for the 2020 census, but it is also appropriate to be thinking now of even broader changes that may apply even further in the future. A sample of such broader, fundamental issues that should be considered for long-term research efforts include the following: • A census without mail: Arguably the boldest area for research in this direction is the concept of a census without primary reliance on mailout-mailback methods. Given the difficult fiscal circumstances of the U.S. Postal Service and major effects that electronic commerce and e-mail have had on regular physical mail volume, means for making initial contact with the national population other than mailed letters or questionnaires may have to be considered in future censuses. • Change to the unit of enumeration: Since the act authorizing the 1790 census required the count to be counted at their “usual place of abode,” the decennial census has used the household as its basic unit of enumeration. In the modern census context, this has involved associating households with specific addresses and, through those ad- dresses, with specific geographic locations. Just as the core assumption of mailout-mailback methodology is one that should be probed and re- considered in coming years, so too is the unit of enumeration worthy of research and examination. For example, it is important to consider how a census using the individual person as the unit of analysis can be analyzed and tabulated, as well as the extent to which households, families, or larger constructs can be reconstructed from a person-level census.

OCR for page 59
CENSUS BUREAU RESEARCH, PAST AND PRESENT 93 • Interaction between the census and the American Community Survey: We discuss the integration of the census and the ACS further in Chap- ter 4, but the topic is a critical long-term research enterprise. In its early period of development, it is both appropriate and important to focus on the properties of the ACS as a replacement for the long- form sample of previous censuses—whether ACS tabulations can sat- isfy both user needs and the myriad legal and regulatory demands for demographic data. Going forward, the capacity of the ACS as a unique survey platform in its own right must be explored, including ways for the census and the ACS to support each other: for example, use of parts of the ACS sample as a test bed for experimental census concept and questions.

OCR for page 59