Click for next page ( 23

The National Academies of Sciences, Engineering, and Medicine
500 Fifth St. N.W. | Washington, D.C. 20001

Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 22
22 of the questionnaire remains. Experience with the status bar the latest features. However, somewhere a line must be has proven to be mixed; with some respondents appreciating drawn as to the browser version that a web questionnaire will it and others finding it distracting and not particularly infor- support. Recent experience suggests supporting browsers mative owing to its heuristic properties. Based on the equivalent to Microsoft Internet Explorer Version 5 or later research team's experience, the status bar does not do harm (browsers that are seven years old) is sufficient to capture the per se; however, it does require significant programming vast majority of Internet users. effort for what appears to be little gain for respondents and the response rate. Do not require respondents to provide an answer to each question before being allowed to answer any subse- Restrain the use of color so that readability is main- quent ones. tained, navigational flow is unimpeded, and measure- ment properties of questions are maintained. The difficulties in resolving item nonresponse are examined later in this chapter. However, item nonresponse in web- The basic point of this guideline is to ensure that colors are based surveys is a very important aspect to web survey only used for the purpose of making questions clearer and design. If respondents are not permitted to skip questions, do not affect the way a question might be interpreted (e.g., survey "drop out" rates can be very high. Conversely, if a satisfaction scale using colors, which might cause mea- respondents are allowed to skip questions, missing data from surement bias). Web-based questionnaires make it easy to item nonresponse is an issue. These tradeoffs are discussed add color and other formatting techniques that are more dif- in greater depth later in this chapter and must be considered ficult when using paper instruments. Before implementing by all web-based researchers. such additions, the researcher is advised to be sure that the color improves the clarity of the situation and does not make Besides these important guidelines from Dillman, there things worse. are a few other design and formatting issues about web-based questionnaires worth mentioning. One has been alluded to Avoid differences in the visual appearance of questions earlier; cascading style sheets (CSS). CSS is a power tool that that result from different screen configurations, operating disassociates the content of the questionnaire from its for- systems, browsers, etc. matting. This allows the researcher to focus on two impor- tant issues separately: first, designing the survey content, and This guideline is very important and is a continual challenge second, being able to format that content easily later using for researchers conducting web-based surveys. There are CSS techniques. Examples of how CSS can take the same some techniques and trends that are improving these issues, content and format it very differently are shown effectively including: at the following link: Using the technique of relative (proportional) HTML Finally, it is important to note that web-based surveys table sizing instead of absolute sizing. This technique can provide access to people with disabilities. By using can significantly help with the issue of different screen readers and other devices, those who are deaf, blind, screen resolution settings. and otherwise disabled can access surveys that they could Although there are more browsers than ever before, not previously using other survey media. However, to most will read proper HTML code correctly in any allow these devices to work correctly, the web designer operating system, particularly if they are the equiva- must ensure that the web-based survey follows the Section lent of Microsoft Internet Explorer Version 5 (intro- 508 guidelines of the Rehabilitation Act requiring federal duced in 1999) browsers or newer. Based on the agencies to make their electronic and information technol- research team's experience, almost all respondents to ogy accessible to those with disabilities. Among many web-based surveys have browsers that are equivalent other things, Section 508 guidelines include requirements or newer. such as putting text tags on images. One advantage of Automatic updates are now standard on all browsers, using CSS is that it can make the process of complying primarily for security purposes. This means that with 508 guidelines significantly easier for the web survey Internet users are more likely than just a few years designer. ago to have browsers that are current with the latest technologies. COVERAGE AND UNIT NONRESPONSE ERROR Monitors, in general, are getting larger and cheaper. AMONG DIFFERENT SURVEY TYPES Many more respondents have monitors that are now large enough to support higher resolutions and pro- Although this synthesis report is focused on web-based sur- vide more screen space. veys, it is important for transit researchers to understand when it makes the most sense to incorporate a web-based These points are not to suggest that web survey designers survey into their research. In making this decision, an under- should assume that most respondents have browsers with all standing of the strengths and limitations of different survey

OCR for page 22
23 TABLE 5 STRENGTHS AND LIMITATIONS OF DIFFERENT SURVEY TYPES CATI (Computer- Computer- assisted Based Paper-Based, telephone Surveys, Not via Hand or In-Person Considerations Survey Methods: Online Web interview) Online Mail-out Interviews Coverage Rate of Population 97.6% [U.S. population (Census)] x 72% (Pew Internet & American Life Project) x Not applicable x x x Strengths Coverage Wide coverage of most U.S. adults (growing) x Administration Self-administered, giving user flexibility for when they respond x x x Administered via an operator interview with the ability to guide respondents through the questionnaire x Administered via interviewer with ability to guide respondent through questionnaire; therefore, low respondent burden x Inexpensive and easy to contact respondents when an e-mail address is known x With interceptor staff present on site, immediate survey or technological help is available x x Low nonresponse error because respondent "can't say no" in person x Survey design Ability to provide interactive content like maps, customized screens, etc. (not possible with non-web-based survey methods) x x Allows complex questions to be asked while keeping the survey simple for the respondent x x Multi-method and validated geocoding x x Error checking x x x Sampling Allows for targeted sampling of a population x x x x x "Captive audience" with face-to-face contact x x Data collection Centralized data collection x x Respondent keys data x x Interviewer collects data: therefore, low respondent burden x Technology Technology is provided for the respondent x Technology (paper/speech) is universal and built into the survey instrument x x (continued on next page) methods is necessary. Table 5 describes the strengths and survey method. However, although the coverage error is limitations regarding coverage, costs, survey design, unit lower in telephone surveys, unit nonresponse error is large nonresponse errors, language requirements, sampling restric- and growing, because a large percentage of telephone cus- tions, and ease of administration of the following survey tomers screen their calls (1,4,5). In urban areas, the number methods: web-based surveys, CATI (computer-assisted tele- of mobile-phone-only households is increasing at a signifi- phone interviews) surveys, paper-based surveys, computer- cant rate, and currently researchers are by law not permitted based surveys (not online), and in-person interview surveys. to call these households for the purpose of administering By considering the strengths and limitations of each survey CATI surveys, because the recipient of the call will be type, the transit researcher can develop an understanding of charged for the call. Even if this barrier is overcome, mobile each method and how best to utilize them in their research phone numbers are not geographically representative the context. way households with landline phones are. If mobile phone numbers are eventually allowed to be called using CATI As shown in this table, there are strengths and weaknesses techniques, geographical representation will still be a major inherent in every survey method. For example, it is still the issue. case (although this is changing rapidly) that telephone surveys can reach a larger portion of households than web- For any study, the researcher must review all sources of based surveys. This is clearly a strength of the telephone error and not dwell solely on one type. For example, it is

OCR for page 22
24 TABLE 5 STRENGTHS AND LIMITATIONS OF DIFFERENT SURVEY TYPES (continued) CATI (Computer- Computer- assisted Based Paper-Based, telephone Surveys, Not via Hand or In-Person Considerations Survey Methods: Online Web interview) Online Mail-out Interviews Limitations Coverage While improving, coverage issues are still a problem x Limited ways to randomly sample a known geographic area x Coverage error is a problem among very low income populations and is a growing issue for other populations (often in urban transit environments such as major metropolitan areas) due to "mobile- phone-only" households who make up an increasing share of the U.S. households. This issue is particularly concentrated among young and mobile people, causing coverage issues that are becoming significant as households drop their land lines. Mobile phone lines are not included in CATI sampling frames. x Only allows for a targeted segment of the population and cannot be used for wide, random geographic sampling x x x Administration Delivery to respondent not as efficient as electronic delivery x x x x Requires computer proficiency x x Respondent must have the time to respond at time of contact x x x Language Requires literacy x x x Those without computer literacy are less likely to respond x x Spoken language/native tongue issues can be problematic x x Nonresponse Error Nonresponse issues due to spam filters and the abundance of spam messages that do not get filtered, causing potential respondents to ignore many e-mail messages x Call screening is a significant non-response issue that may systematically exclude various subpopulations x Data Data quality can be low due to the inability to validate user input x Requires that data be coded again into digital form, adding further input cost and error x x Static format limits the types of questions that can be asked. x x Costs Range from low to high depending on the complexity of the survey and the method of recruitment. Costs can be very low for recurring surveys, as the marginal costs of re-contacting a respondent are extremely low. x Range from low to high cost depending on the extent of the sampling frame. Costs can be particularly high when contacting potential respondents multiple times through reminders and pre-survey instruments to try and encourage response. x Typically expensive due to the high cost of reaching respondents and because it uses an interviewer to administer the survey. x Typically expensive, as requires on-site staff x x clear that the synthesis survey found that coverage error was follows later in this chapter: Multi-Method Surveys to Mit- a major concern on web-based surveys. However, also con- igate Coverage Error. Budget constraints on a study will sider that web-based surveys appear to have an advantage determine which methods to use; however, if optimal sur- over CATI surveys in terms of nonresponse error. vey methods are considered, costs should be mitigated somewhat by using the most efficient survey method(s); There has never been one survey method with the abil- controlling costs may only be a matter of managing the ity to reach all households equally. Therefore, studies with number of survey methods. multi-method approaches and that use the optimal survey method(s) to target the sampling population are the "best" Responses to the synthesis survey indicated that optimized practice. A detailed discussion of multi-method approaches multi-method surveys are the current state of practice, with