Click for next page ( 28


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 27
27 schools, libraries, and other public places; however, it is Web-based surveys have advantages over telephone sur- unlikely that many respondents would make the extra effort veys in terms of unit nonresponse. Web surveys do not suf- to go to one of these locations explicitly for the purpose of fer as much as telephone surveys from the issue of high unit completing a transit survey. Despite the high penetration nonresponse rates as a result of call-monitoring techniques rates, when conducting a study that will use web-based such as answering machines and caller ID (1,4,9). This is surveys, either as the only survey method or as part of a because there is much less active screening of e-mails than multi-method survey design, it is important to ensure that of telephone calls (passive spam filtering, which is a serious the population targeted for the study's web sampling frame problem, is described later). Therefore, if an e-mail arrives in is on the high side of these Internet usage statistics. For a respondent's inbox with a subject of interest, that individ- many of these populations, such as suburban and urban ual may be more likely to respond to it than to a telephone adults less than 64 years of age, either college educated or call with an unknown number. with incomes of more than $30,000 per year, the incidence rate of Internet access is between 75% and 94%. These inci- However, spam is a very serious issue for e-mail and has dence numbers are high and for certain subpopulations are become increasingly problematic over the last 5 years. As approaching the national telephone incidence rate of 97.6% most users of e-mail understand through experience, various (U.S. Census). Therefore, web coverage, although not quite types of spam filters are becoming standard at most compa- as good as telephone coverage, is inclusive for many popu- nies and organizations, as well as at home, through a variety lations. This synthesis will further explore and discuss cov- of software products. Therefore, spam issues are a concern erage error in the following section, where it will be shown that must be addressed when conducting web-based surveys. that coverage error, although higher for web-based surveys The most important issue for web-based survey research than telephone surveys, may be mitigated by lower non- regarding spam is to avoid survey invitations being tagged as response error in web-based surveys compared with tele- spam and filtered out of a respondent's e-mail in-box before phone surveys. it is even seen. There are a variety of methods to increase e-mail delivery NONRESPONSE ERROR IN WEB SURVEYS rates and avoid false positives (i.e., a message being tagged as spam although it is legitimate). First, all bulk e-mail The two types of nonresponse error--unit nonresponse and senders should adhere to the industry accepted, and federally item nonresponse--are also of concern to researchers. mandated, e-mail practices outlined in the CAN-SPAM Act of 2003: Unit Nonresponse in Web Surveys 1. Bulk e-mail must clearly identify the sender--including Unit nonresponse error occurs when survey respondents dif- a physical address. fer from nonrespondents in a way that is significant to the 2. Bulk e-mail must contain a valid subject line and valid study. For example, if low-income transit riders respond to routing information. a telephone survey in disproportionately higher numbers 3. Bulk e-mail must contain a working opt-out mechanism. than high-income riders there would be nonresponse error for higher income riders. Often nonresponse error can be Second, many companies offer paid white-listing ser- mitigated through procedures that weight up underrepre- vices, promising increased delivery through partnerships sented groups and weight down overrepresented groups. with e-mail providers. Senders often undergo an e-mail- Weighting may successfully mitigate nonresponse error as practice audit and, if accepted into the program, are added to long as there are enough underrepresented respondents to a white-list used by e-mail providers to allow delivery with- provide reasonable statistical confidence, assuming the peo- out spam filtering. Third, avoiding certain words and phrases ple who responded are similar to those who did not respond. common to spam will decrease the likelihood that legitimate e-mail will be incorrectly marked as spam. Words and However, if nonresponse error results from significant phrases such as "free," "easy money," "gamble," "money," underrepresentation of a particular subpopulation of the sam- and "rich" are commonly found in spam, and should be ple (e.g., high-income transit riders who only use transit in avoided if at all possible. Lastly, ensure that the source of the the evening, and thus were unavailable when the telephone e-mail (ISP) does not tolerate or conduct business with survey was being conducted), the more serious issue arises known spammers. An ISP can have its e-mail servers black- of systematically excluding a particular subpopulation of the listed across the Internet for doing business with known sample whose behavior is different in a way that is important spammers. Anyone sending e-mail using the same system to the study. In this example, an effort would need to be made will be subject to the blacklist rules in effect and the e-mail to ensure that the high-income, evening-travel subpopulation will never reach prospective users. would somehow be included in the study. One method might be for the transit agency to offer a web-based survey option Spam filters are not the only nonresponse issue related to in addition to the telephone option. web-based surveys. Many respondents receive significant

OCR for page 27
28 legitimate e-mail as well as significant spam. This means that item nonresponse can occur as a result of a different form of many potential respondents are fairly ruthless about what item nonresponse called "break-off," where respondents e-mail they read versus e-mail they discard (6). Therefore, simply fail to complete the questionnaire. the key for transit researchers is to ensure that their e-mail invitations are to the point and understand what will be of There are a variety of ways to mitigate item nonresponse interest to respondents so that these invitations are read and and a large body of literature exists on the topic (1, p. 529; 6, acted on (e.g., "Improve your commute" or "Tired of sitting p. 555). One method to reduce item nonresponse is to ensure in traffic?" or "Contribute your opinion on a new transit a high level of interest among potential respondents to the alternative") (1). survey (1). Fortunately, there is often a high level of interest in transit and other transportation surveys because respon- Another nonresponse issue with web-based surveys is that dents have a strong desire to improve their commutes and some respondents simply do not check their e-mail very other travel. often, or at least not the e-mail address that they provided the transit researcher. Still other issues include multiple e-mail Another way to mitigate item nonresponse is to remind addresses, undeliverable e-mail addresses, server errors (e.g., respondents who have started a questionnaire that they the respondent's ISP happens to be conducting server main- have not finished and should continue on and complete it. tenance when the e-mail is sent and therefore it bounces This is a major strength of web-based surveys compared back). with telephone or mail surveys. The costs of e-mailing a reminder are very small, and there is minimal concern Clearly there are a number of issues and concerns regard- that the respondent is not receiving the reminder e-mail ing nonresponse in web-based surveys. However, assuming because they have already responded to the questionnaire the researcher's e-mail invitation is not tagged as spam invitation. (which as explained previously there are ways to mitigate), the e-mail address is correct, the topic is of interest to the One of the benefits of web-based surveys (recorded by respondent, and the invitation is concisely and clearly writ- 71% of respondents to the synthesis survey) is their ability to ten, then unit nonresponse error can be significantly miti- obtain clean data through consistency checks and validation gated and respondents should at least begin the survey of user responses, essentially eliminating item nonresponse (whether they complete the survey is the issue of item non- for those respondents who complete the questionnaire (1). At response, discussed in the next sections). the same time however it is important that real-time editing and response validation in web surveys do not dissuade the Nonresponse can be reduced significantly if researchers respondent from continuing their questionnaire because the are diligent in managing their customers' e-mail address checking and editing become too onerous. The difficulty of lists, such that they contain only valid addresses (or at least balancing validation while encouraging respondents to com- e-mails that have not been returned to sender). If researchers' plete questionnaires can be seen when online geocoding is e-mail lists are valid and current, their respondents will be used in web-based surveys. familiar with the organization from previous e-mail corre- spondence and/or web-based research and may be more Online geocoding is an important benefit for transit and inclined to open and respond to the researchers' e-mail transportation applications because it yields precise and val- requests. Response rates of 50% and higher are not uncom- idated address information, often critically important to tran- mon for well-managed lists or panels. It should be noted, sit researchers and almost impossible to collect accurately however, that the researcher must not create a self-selecting using other survey methods. However, online geocoding is list of those with a greater propensity to respond by simply still not perfect. Occasionally, a respondent may enter their throwing out e-mails of those who did not respond to prior home address into a survey only to find that the survey invitations. (through a real-time geocoding check) insists their home address does not exist (owing, for example, to an outdated GIS data set). The respondent checks their typing, tries to Item Nonresponse in Web-Based Surveys proceed, and again is told the address does not exist. This will often cause the respondent to become frustrated and abandon Item nonresponse refers to the issue of missing or incorrect the survey, either because there is no way to proceed or data items in questionnaires. Item nonresponse occurs when because it is simply too onerous for them to proceed by using respondents skip questions or fail to complete a question- the alternatives presented. The fundamental challenge for naire. Self-administered questionnaires, such as web-based any web-based survey is to guarantee a balance between and paper-based surveys, typically have more item non- validating data to reduce item nonresponse and allowing response than questionnaires administered using interview- respondents a way to proceed through difficult questions ers (4). Although web-survey questions are often validated, without causing break-off, which is a different form of item making it hard or in some cases impossible to skip a question, nonresponse.