4
Improved Survey Methods: Making It Easier to Respond

In order to achieve its goals--to reduce costs and respondent burden, to do a better job of counting traditionally undercounted groups, and to improve data quality--the Census Bureau tried five new survey methods in the 1995 census test: (1) development of respondent-friendly questionnaires; (2) provision of multiple modes for response; (3) mailing Spanish-language forms, in addition to English ones, to targeted areas; (4) targeted promotional campaigns; and (5) new approaches to enumerating people with no usual residence, service-based enumeration. Evaluation of these methods showed that some were successful, some showed promise but needed changes, and others did not appear to have any positive effect on either the response rate or data quality. The goals also sometimes proved to be conflicting, making decisions about implementation more difficult.

RESPONDENT-FRIENDLY QUESTIONNAIRES

Improvement in the census questionnaire has the potential to reduce costs by increasing mail response, by far the least expensive way to count household members. In past censuses, the questionnaire's design was based on a need for easy data capture, without much concern for its effect on respondents. Over the last couple of decades, however, considerable research has shown that the appearance of a survey questionnaire and the form of the appeals made for cooperation can substantially increase response rates. This approach, called the total design method, has a carefully prescribed series of steps for presentation of a questionnaire to respondents. The Census Bureau began small-scale tests and focus group evaluations of these methods in 1992, and the new procedures were implemented in the 1995 census test (Treat, 1995a,b). Research in this area by the Bureau is continuing.

One of the major ways that the total design method differs from previous census operations is that it makes multiple mail contacts with a household. Plans for the 2000 census now include sending an advance letter a few days before, and a reminder postcard a few days after, the census questionnaire arrives by mail. In addition, a replacement questionnaire will be sent a couple of weeks after the initial questionnaire mailing to those households that have not yet responded. The test results indicate that this is an especially fruitful strategy. Evaluations in the early tests estimated that the prenotice letter and post mailing postcard increased the mail response rate by about 12.7 percent, and the replacement questionnaire produced an additional gain of about 10 percentage points.

If both effects persist in the actual census, they would produce a cost savings of about half a billion dollars (Edmonston and Schultz, 1995), minus the added cost of the



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 24
Preparing for the 2000 Census: Interim Report II 4 Improved Survey Methods: Making It Easier to Respond In order to achieve its goals--to reduce costs and respondent burden, to do a better job of counting traditionally undercounted groups, and to improve data quality--the Census Bureau tried five new survey methods in the 1995 census test: (1) development of respondent-friendly questionnaires; (2) provision of multiple modes for response; (3) mailing Spanish-language forms, in addition to English ones, to targeted areas; (4) targeted promotional campaigns; and (5) new approaches to enumerating people with no usual residence, service-based enumeration. Evaluation of these methods showed that some were successful, some showed promise but needed changes, and others did not appear to have any positive effect on either the response rate or data quality. The goals also sometimes proved to be conflicting, making decisions about implementation more difficult. RESPONDENT-FRIENDLY QUESTIONNAIRES Improvement in the census questionnaire has the potential to reduce costs by increasing mail response, by far the least expensive way to count household members. In past censuses, the questionnaire's design was based on a need for easy data capture, without much concern for its effect on respondents. Over the last couple of decades, however, considerable research has shown that the appearance of a survey questionnaire and the form of the appeals made for cooperation can substantially increase response rates. This approach, called the total design method, has a carefully prescribed series of steps for presentation of a questionnaire to respondents. The Census Bureau began small-scale tests and focus group evaluations of these methods in 1992, and the new procedures were implemented in the 1995 census test (Treat, 1995a,b). Research in this area by the Bureau is continuing. One of the major ways that the total design method differs from previous census operations is that it makes multiple mail contacts with a household. Plans for the 2000 census now include sending an advance letter a few days before, and a reminder postcard a few days after, the census questionnaire arrives by mail. In addition, a replacement questionnaire will be sent a couple of weeks after the initial questionnaire mailing to those households that have not yet responded. The test results indicate that this is an especially fruitful strategy. Evaluations in the early tests estimated that the prenotice letter and post mailing postcard increased the mail response rate by about 12.7 percent, and the replacement questionnaire produced an additional gain of about 10 percentage points. If both effects persist in the actual census, they would produce a cost savings of about half a billion dollars (Edmonston and Schultz, 1995), minus the added cost of the

OCR for page 24
Preparing for the 2000 Census: Interim Report II additional mail contacts.3 Savings estimates based solely on the 1995 census test are probably too optimistic, since the effect of any questionnaire improvements are likely to be less in 2000 than in the 1995 census test because the initial response rates will be higher. At higher response rates, every additional percentage point of improvement is more costly to achieve. There would also be considerable operational complexity in sending replacement questionnaires to nonresponding households over the entire United States, as would be required in 2000. If this constraint proves prohibitive, one possibility is to target only areas with low response rates for replacement questionnaires. These areas could be identified either in advance, based on 1990 block characteristics, or on early mail response rates in 2000. Another feature of the total design method is to take account of the general appearance and form of the survey questionnaire, as well as to follow guidelines on the wording and order of the questions themselves. These guidelines were followed in the development of the questionnaire for the 1995 census test. Analysis of the results showed some shortcomings of the questionnaire items, such as apparent confusion about the meaning of some coverage questions, whose purpose was to encourage respondents to assess whether they had correctly enumerated all household members. The Census Bureau is continuing to pursue research on questionnaire design, including research into improvement of the long form.4 Since research on questionnaire design is relatively inexpensive and has high potential payoff, we believe that this effort is well placed. In addition to research to design respondent-friendly forms, the Bureau has recently announced the transfer of five items from the short to the long form. This shortening of the form may act to increase mail return rates, although at the cost of reduced small-area data. MULTIPLE MODES FOR RESPONSE The plans for the 2000 census provide a number of new ways for respondents to be enumerated. The innovations were intended to reduce respondent burden and to improve coverage of traditionally undercounted groups. One of these changes is the introduction of the "Be Counted" forms: these forms are placed in public locations so that people who did not receive a census form or do not believe they were counted can easily report their information. This change in the census design was adopted from a suggestion presented to Congress in a report from the U.S. General Accounting Office (1992). The forms were available in the 1995 census test in five languages and at a 3   We do not have data to estimate the added costs, but they are likely to be considerably less than the estimated savings. 4   The long form sent to 1 of 6 sample of households in 1990, collects additional data beyond the minimum required for reapportionment and redistricting. These data are of intense interest to many census users.

OCR for page 24
Preparing for the 2000 Census: Interim Report II information by telephone to a toll-free number. Individuals could also request by telephone a census form in any available language. Unlike the "Were You Counted" campaigns of the 1980 and 1990 censuses, which took place after nonresponse follow-up and did not include reporting by telephone, the "Be Counted" campaign is concurrent with the census. The 1995 census test suggests that the "Be Counted" forms did not achieve the goal of improving coverage of groups who have traditionally been undercounted in the census. The number of people responding by these forms was quite small, less than 1 percent of the 1990 population in all three 1995 census test sites: 0.8% for Oakland and Paterson, 0.5% for Louisiana. In addition, the hard-to-enumerate demographic subgroups--including people younger than 30 years old and blacks and Hispanics--tended to be underrepresented in the "Be Counted" population in comparison with their numbers in the 1990 census in those sites. Thus, as a method of reducing the differential undercount, the "Be Counted" campaign does not seem promising. However, most of the time a ''Be Counted" form was received, it was the only form received for that housing unit. Only 15.2 percent of the households with a person who sent in the form had another census form (mail or enumerator-returned) (Ammenhauser and Lucas, 1996). In addition, the public relations benefits from such a campaign might be substantial. For the 1995 census test, a wide variety of types of sites were used for distribution of the forms, including city halls, motor vehicle offices, and community centers. Only a few types of site resulted in a significant number of used forms. It seems that some reduction in cost could be realized by reducing the number of sites used for distribution of the "Be Counted" forms. The toll-free telephone number was an especially effective mode for collecting the "Be Counted" data in the 1995 census test. Over 42 percent of the people enumerated through the forms initiated the interview by telephone in the two urban test sites. In a related development, there is now discussion within the Census Bureau of allowing reporting electronically, through the Internet. Allowing response by Internet will surely not increase participation by historically undercounted groups, but it might also provide the benefit of improved public relations. There are some technical complexities of processing "Be Counted" forms that are not yet resolved and need further research (Ammenhauser and Lucas, 1996). One is that respondents are required to report on the form whether their response is for a whole household or a partial household. Exclusion of the address from the nonresponse followup universe is dependent on the answer to this question. However, given the difficulty that respondents appear to have with the concept of usual residence, as indicated in the analysis of the coverage question on the mail-back census form, it seems unlikely that the information received for this item is very accurate. Thus, this additional complexity in the processing procedure seems error prone. A simpler procedure would be to include either none or all of the addresses in the nonresponse follow-up universe: if none, it would leave unreported people in households with a "Be Counted" person to be estimated by integrated coverage measurement; if all, it would increase the nonresponse follow-up workload, but not by much because of the small number of forms. Either approach would require careful thought about sample design and estimation issues. For example, if the households are not included in the

OCR for page 24
Preparing for the 2000 Census: Interim Report II nonresponse follow-up universe, it might be advantageous if they constituted their own set of strata for integrated coverage measurement estimation. If all the households are included in the nonresponse follow-up universe, they might best be sampled at a different rate than other nonresponding households. Furthermore, auxiliary information from the "Be Counted" forms themselves--which would be available for both sampled and nonsampled households--could be used in estimation. To our knowledge, these issues have not yet been considered by the Census Bureau. Whenever more than one questionnaire is made available to a household, as will occur with either "Be Counted" forms or replacement questionnaires, there must be some means of ensuring that people who are counted more than once can be accurately "unduplicated." The 1995 census test showed that the number of duplicated responses resulting from replacement forms was small, and their unduplication was a feasible procedure (Ammenhauser, 1996; Hill and Leslie, 1996). In the actual census, there might be more motivation for people to try to inflate the count intentionally through use of multiple forms, especially the "Be Counted" forms. The Census Bureau should be prepared for this possibility. TARGETING FOR SPANISH-LANGUAGE QUESTIONNAIRES The Census Bureau has conducted two tests of the effectiveness of mailing Spanish-language questionnaires, in addition to English ones, to households in areas with high linguistic isolation in 1990, defined as areas in which more than 30 percent of the households include no one over 14 who speaks English very well or at all. The goal of mailing Spanish-language questionnaires to these areas was to increase the response rate and decrease the differential undercount. The Spanish Forms Availability Test, conducted in fall 1993, showed that offering a Spanish-language form increased the completion rate by an estimated 2-6 percentage points. However, the households that completed the Spanish questionnaire showed significantly higher item nonresponse rates than the households that completed an English questionnaire, thus appearing to reduce progress towards the stated goal of improving data quality. In addition, the majority of Hispanic households that returned a questionnaire returned an English form. The 1995 census test investigated the effect of sending two questionnaires, one in English and one in Spanish, to targeted areas in a census environment, where greater publicity may reduce the differential nonresponse rates. In this test, approximately 60 percent of the Hispanic households in the targeted areas returned Spanish questionnaires. This proportion remained constant whether the targeted area had a moderately high (between 15% and 30%) or high (more than 30%) level of linguistic isolation, based on 1990 data. Thus, the linguistic isolation variable does not appear to be a particularly good predictor of the usefulness of a Spanish-language questionnaire. This result could have occurred because the 1990 data were out of date by 1995, which suggests that they will be even less useful in 2000. The result is important because the Census Bureau is planning to expand the non-English forms distribution to include non-English languages in addition to Spanish.

OCR for page 24
Preparing for the 2000 Census: Interim Report II Recommendation: The Census Bureau should monitor the use of Spanish-language and other non-English language forms and the 800 telephone number during the 1998 dress rehearsal to refine criteria for forms distribution in 2000. TARGETED PROMOTIONAL CAMPAIGNS The Census Bureau is planning a major shift in the focus of its marketing plan for the 2000 census. The major change is that geographic areas and demographic subgroups will be targeted with different publicity and outreach campaigns, rather than the uniform approach used in previous censuses. The goal of this approach is to reduce the differential undercount. Another change is that the campaign will use paid advertising, rather than public service announcements, as in previous censuses. Since the Census Bureau lacks in-house expertise in this area, the plan is to contract out this program. Although this new approach seems reasonable, the panel has not seen evidence of its effectiveness. In fact, the only analysis undertaken to date has been through focus groups, whose results are mixed. It may not be possible to obtain experimental evidence of the effectiveness of the planned promotional campaign in advance of the census, but we believe it is important that measures of its success be specified and data collected to assess its performance for use in future censuses. Recommendation: The Census Bureau should establish measures and specify the data to be collected to assess and evaluate the performance of the various aspects of its promotional program. SERVICE-BASED ENUMERATION The Census Bureau has designed the service-based enumeration program to enumerate clients in emergency shelters and soup kitchens. This program replaces the Snight enumeration of the homeless, which was developed for the 1990 census in collaboration with experts in the field of homelessness. The central feature of the new program, in the 1995 census test, was to enumerate clients at all service locations. Efforts were first made to obtain service provider cooperation, and enumerators who were familiar with the service locations were used. Enumeration in this environment proved to have many difficulties, and the 1995 test identified a number of procedures that could be simplified. One of the more problematic aspects of the 1995 test program was the reinterview attempted at a sample of provider locations. The purpose of this reinterview was to provide data that would allow estimation of coverage, using a dual-system methodology. Respondents were confused about the reason that they were being reinterviewed and were therefore reluctant to complete the questionnaire again, which resulted in high refusal and item nonresponse rates. This in turn caused difficulty with matching to the initial questionnaire. Alternative methods of estimation of coverage were also attempted. One of these, known as multiplicity estimation, required data from only a single visit, but

OCR for page 24
Preparing for the 2000 Census: Interim Report II relied on information provided by the respondent about how many days in the previous seven that services had been used. The methodology for the multiplicity estimator is similar to that of the Politz-Simmons estimator (Politz and Simmons, 1949, 1950). Both the dual-system estimator and the multiplicity estimator rely on several assumptions that are questionable in this application. These coverage measurement methods were examined further in the 1996 test, but results from it were not available for this report. Recommendation: The Census Bureau should continue research to determine the best means to enumerate people with no usual residence. This effort should be part of the Census Bureau's research agenda, with specific goals for both pre-2000 research and to learn from 2000 for the 2010 census.