National Academies Press: OpenBook

Web-Based Survey Techniques (2006)

Chapter: Chapter Four - Web-Based Survey Methodologies and Successful Practices

« Previous: Chapter Three - Current State of Practice for Web-Based Transit Surveys
Page 20
Suggested Citation:"Chapter Four - Web-Based Survey Methodologies and Successful Practices." National Academies of Sciences, Engineering, and Medicine. 2006. Web-Based Survey Techniques. Washington, DC: The National Academies Press. doi: 10.17226/14028.
×
Page 20
Page 21
Suggested Citation:"Chapter Four - Web-Based Survey Methodologies and Successful Practices." National Academies of Sciences, Engineering, and Medicine. 2006. Web-Based Survey Techniques. Washington, DC: The National Academies Press. doi: 10.17226/14028.
×
Page 21
Page 22
Suggested Citation:"Chapter Four - Web-Based Survey Methodologies and Successful Practices." National Academies of Sciences, Engineering, and Medicine. 2006. Web-Based Survey Techniques. Washington, DC: The National Academies Press. doi: 10.17226/14028.
×
Page 22
Page 23
Suggested Citation:"Chapter Four - Web-Based Survey Methodologies and Successful Practices." National Academies of Sciences, Engineering, and Medicine. 2006. Web-Based Survey Techniques. Washington, DC: The National Academies Press. doi: 10.17226/14028.
×
Page 23
Page 24
Suggested Citation:"Chapter Four - Web-Based Survey Methodologies and Successful Practices." National Academies of Sciences, Engineering, and Medicine. 2006. Web-Based Survey Techniques. Washington, DC: The National Academies Press. doi: 10.17226/14028.
×
Page 24
Page 25
Suggested Citation:"Chapter Four - Web-Based Survey Methodologies and Successful Practices." National Academies of Sciences, Engineering, and Medicine. 2006. Web-Based Survey Techniques. Washington, DC: The National Academies Press. doi: 10.17226/14028.
×
Page 25
Page 26
Suggested Citation:"Chapter Four - Web-Based Survey Methodologies and Successful Practices." National Academies of Sciences, Engineering, and Medicine. 2006. Web-Based Survey Techniques. Washington, DC: The National Academies Press. doi: 10.17226/14028.
×
Page 26
Page 27
Suggested Citation:"Chapter Four - Web-Based Survey Methodologies and Successful Practices." National Academies of Sciences, Engineering, and Medicine. 2006. Web-Based Survey Techniques. Washington, DC: The National Academies Press. doi: 10.17226/14028.
×
Page 27
Page 28
Suggested Citation:"Chapter Four - Web-Based Survey Methodologies and Successful Practices." National Academies of Sciences, Engineering, and Medicine. 2006. Web-Based Survey Techniques. Washington, DC: The National Academies Press. doi: 10.17226/14028.
×
Page 28
Page 29
Suggested Citation:"Chapter Four - Web-Based Survey Methodologies and Successful Practices." National Academies of Sciences, Engineering, and Medicine. 2006. Web-Based Survey Techniques. Washington, DC: The National Academies Press. doi: 10.17226/14028.
×
Page 29
Page 30
Suggested Citation:"Chapter Four - Web-Based Survey Methodologies and Successful Practices." National Academies of Sciences, Engineering, and Medicine. 2006. Web-Based Survey Techniques. Washington, DC: The National Academies Press. doi: 10.17226/14028.
×
Page 30
Page 31
Suggested Citation:"Chapter Four - Web-Based Survey Methodologies and Successful Practices." National Academies of Sciences, Engineering, and Medicine. 2006. Web-Based Survey Techniques. Washington, DC: The National Academies Press. doi: 10.17226/14028.
×
Page 31
Page 32
Suggested Citation:"Chapter Four - Web-Based Survey Methodologies and Successful Practices." National Academies of Sciences, Engineering, and Medicine. 2006. Web-Based Survey Techniques. Washington, DC: The National Academies Press. doi: 10.17226/14028.
×
Page 32
Page 33
Suggested Citation:"Chapter Four - Web-Based Survey Methodologies and Successful Practices." National Academies of Sciences, Engineering, and Medicine. 2006. Web-Based Survey Techniques. Washington, DC: The National Academies Press. doi: 10.17226/14028.
×
Page 33

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

This chapter first discusses successful techniques for and issues related to design and formatting of web-based surveys. Next, it compares the strengths and limitations of other sur- vey methods with the strengths and limitations particular to web-based surveys, so that the transit researcher can evalu- ate whether and how to best conduct a web-based survey. It then describes ways to handle survey errors in web-based transit surveys, which include coverage error (which occurs when a portion of respondent population is not reached), unit nonresponse error (when there are significant differences in results owing to over- or underrepresentation of groups within the sampling frame), and item nonresponse error (which occurs when respondents skip questions or fail to complete a questionnaire). Finally, the chapter describes suc- cessful practices and challenges faced when incorporating web-based surveys into transit research. This includes dis- cussion of the advanced capabilities of web-based surveys, particularly those that improve the data and information nec- essary for transit research. QUESTIONNAIRE DESIGN AND FORMATTING There is no single way to design and format a questionnaire. As with good books that can be written using different styles, for- mats, and methods, so can questionnaires. If the questionnaire is well-constructed and clear, it will be an effective survey instrument that encourages potential respondents to participate in the study. That said, there are some fundamental principles and techniques that questionnaire writers should understand and be familiar with. These techniques are different from ques- tionnaires that use paper, the telephone, or other media, and are presented in this section to help the transit researcher under- stand what issues need to be addressed to create an effective web-based questionnaire. The individual who has conducted some of the most significant research into what is considered good ques- tionnaire design and formatting for web-based surveys is Dr. Donald Dillman. In this section, we will review the guidelines that Dillman has outlined as to what constitutes good questionnaire formatting and design for web-based surveys as described in his book Mail and Internet Surveys: The Tailored Design Method (1). For each guideline, an example will be provided when appropriate, and commen- tary on various experiences implementing web-based sur- veys will also be provided. Not all guidelines by Dillman 20 are always adhered to exactly by experienced web survey researchers, and any reasons for diverting from his sugges- tions will be discussed as well. Some of Dillman’s most important guidelines are: • Use a welcome screen that is motivational, emphasizes the ease of responding, and that instructs users how to proceed. An example of this technique was employed in the survey for this synthesis (see Figure 15). Research team experience on many web-based surveys suggests that this is an important aspect of a questionnaire and a good way to ease respondents into the survey instrument. Graphics and graphic design that are interesting, eye catching, and relevant to the topic matter give the respondent the impression that the questionnaire is legitimate and worth taking. • Use a password or personal identification number to restrict access to the survey. This technique is important to control access to the survey and ensure one person/one response. This technique was used in this synthesis survey. One effective way to make password protection easy for respondents is to embed the password into the link to the survey in the invitation e-mail as was done for this survey. By using this embedding tech- nique, respondents do not have to type any password or code into a password screen and are taken directly to the welcome screen. • Present an initial question that is interesting and easy for respondent to answer and that does not require any scrolling. As seen in Figure 16, the first question of the synthesis survey was easy for respondents to answer, because it was asking for what type of organization they work. Often respondents find talking about themselves and what they do interesting. In addition, the question is short enough where it does not, for typical screen resolutions, require scrolling. • Present each question in a conventional format that is similar to paper-based self-administered questionnaires. Dillman recommends treating a web-based questionnaire like a paper-based questionnaire, where there are many CHAPTER FOUR WEB-BASED SURVEY METHODOLOGIES AND SUCCESSFUL PRACTICES

21 questions on each page and where branching is done explic- itly by telling the respondent what question to go to next. (Dillman suggests that the web survey provide a hyperlink; however, the respondent must still actively click on it to link with branch.) However, the technique employed by the research team for its surveys that has been found successful is using one question per screen, which keeps it simple for the respon- dent, as there is only one question, and it also means that scrolling is reduced or eliminated altogether. When the respondent clicks “next,” all branching is done automati- cally using this technique. This technique runs somewhat contrary to Dillman’s recommendation to treat a web-based questionnaire like a paper-based questionnaire; however, the research team’s experience suggests that web surveys work better by making the branching seamless so that the respondent does not even notice it. This is accomplished effectively by using the one-question-per-page method and programming any required branching logic in the underly- ing web-based survey code. Dillman also recommends numbering questions so that the respondent understands where the question begins and where it ends. The research team did not do this because the often complex branching employed in questionnaires can make it difficult to know what the exact number of the ques- tion is for any respondent and because the one-question- per-page method makes numbering unnecessary. However, as seen in Figures 15 and 16, employing a status bar (also suggested by Dillman), which is heuristic and not precise for complex surveys, gives the respondent an idea of how much FIGURE 16 First question of this synthesis survey (Topic SH-07). FIGURE 15 Welcome screen for this synthesis survey (Topic SH-07).

of the questionnaire remains. Experience with the status bar has proven to be mixed; with some respondents appreciating it and others finding it distracting and not particularly infor- mative owing to its heuristic properties. Based on the research team’s experience, the status bar does not do harm per se; however, it does require significant programming effort for what appears to be little gain for respondents and the response rate. • Restrain the use of color so that readability is main- tained, navigational flow is unimpeded, and measure- ment properties of questions are maintained. The basic point of this guideline is to ensure that colors are only used for the purpose of making questions clearer and do not affect the way a question might be interpreted (e.g., a satisfaction scale using colors, which might cause mea- surement bias). Web-based questionnaires make it easy to add color and other formatting techniques that are more dif- ficult when using paper instruments. Before implementing such additions, the researcher is advised to be sure that the color improves the clarity of the situation and does not make things worse. • Avoid differences in the visual appearance of questions that result from different screen configurations, operating systems, browsers, etc. This guideline is very important and is a continual challenge for researchers conducting web-based surveys. There are some techniques and trends that are improving these issues, including: – Using the technique of relative (proportional) HTML table sizing instead of absolute sizing. This technique can significantly help with the issue of different screen resolution settings. – Although there are more browsers than ever before, most will read proper HTML code correctly in any operating system, particularly if they are the equiva- lent of Microsoft Internet Explorer Version 5 (intro- duced in 1999) browsers or newer. Based on the research team’s experience, almost all respondents to web-based surveys have browsers that are equivalent or newer. – Automatic updates are now standard on all browsers, primarily for security purposes. This means that Internet users are more likely than just a few years ago to have browsers that are current with the latest technologies. – Monitors, in general, are getting larger and cheaper. Many more respondents have monitors that are now large enough to support higher resolutions and pro- vide more screen space. These points are not to suggest that web survey designers should assume that most respondents have browsers with all 22 the latest features. However, somewhere a line must be drawn as to the browser version that a web questionnaire will support. Recent experience suggests supporting browsers equivalent to Microsoft Internet Explorer Version 5 or later (browsers that are seven years old) is sufficient to capture the vast majority of Internet users. • Do not require respondents to provide an answer to each question before being allowed to answer any subse- quent ones. The difficulties in resolving item nonresponse are examined later in this chapter. However, item nonresponse in web- based surveys is a very important aspect to web survey design. If respondents are not permitted to skip questions, survey “drop out” rates can be very high. Conversely, if respondents are allowed to skip questions, missing data from item nonresponse is an issue. These tradeoffs are discussed in greater depth later in this chapter and must be considered by all web-based researchers. Besides these important guidelines from Dillman, there are a few other design and formatting issues about web-based questionnaires worth mentioning. One has been alluded to earlier; cascading style sheets (CSS). CSS is a power tool that disassociates the content of the questionnaire from its for- matting. This allows the researcher to focus on two impor- tant issues separately: first, designing the survey content, and second, being able to format that content easily later using CSS techniques. Examples of how CSS can take the same content and format it very differently are shown effectively at the following link: http://www.csszengarden.com/. Finally, it is important to note that web-based surveys can provide access to people with disabilities. By using screen readers and other devices, those who are deaf, blind, and otherwise disabled can access surveys that they could not previously using other survey media. However, to allow these devices to work correctly, the web designer must ensure that the web-based survey follows the Section 508 guidelines of the Rehabilitation Act requiring federal agencies to make their electronic and information technol- ogy accessible to those with disabilities. Among many other things, Section 508 guidelines include requirements such as putting text tags on images. One advantage of using CSS is that it can make the process of complying with 508 guidelines significantly easier for the web survey designer. COVERAGE AND UNIT NONRESPONSE ERROR AMONG DIFFERENT SURVEY TYPES Although this synthesis report is focused on web-based sur- veys, it is important for transit researchers to understand when it makes the most sense to incorporate a web-based survey into their research. In making this decision, an under- standing of the strengths and limitations of different survey

23 methods is necessary. Table 5 describes the strengths and limitations regarding coverage, costs, survey design, unit nonresponse errors, language requirements, sampling restric- tions, and ease of administration of the following survey methods: web-based surveys, CATI (computer-assisted tele- phone interviews) surveys, paper-based surveys, computer- based surveys (not online), and in-person interview surveys. By considering the strengths and limitations of each survey type, the transit researcher can develop an understanding of each method and how best to utilize them in their research context. As shown in this table, there are strengths and weaknesses inherent in every survey method. For example, it is still the case (although this is changing rapidly) that telephone surveys can reach a larger portion of households than web- based surveys. This is clearly a strength of the telephone survey method. However, although the coverage error is lower in telephone surveys, unit nonresponse error is large and growing, because a large percentage of telephone cus- tomers screen their calls (1,4,5). In urban areas, the number of mobile-phone-only households is increasing at a signifi- cant rate, and currently researchers are by law not permitted to call these households for the purpose of administering CATI surveys, because the recipient of the call will be charged for the call. Even if this barrier is overcome, mobile phone numbers are not geographically representative the way households with landline phones are. If mobile phone numbers are eventually allowed to be called using CATI techniques, geographical representation will still be a major issue. For any study, the researcher must review all sources of error and not dwell solely on one type. For example, it is Considerations Survey Methods: Online Web CATI (Computer- assisted telephone interview) Computer- Based Surveys, Not Online Paper-Based, via Hand or Mail-out In-Person Interviews Coverage Rate of Population 97.6% [U.S. population (Census)] 72% (Pew Internet & American Life Project) x Not applicable x x x Strengths Coverage Wide coverage of most U.S. adults (growing) x x Administration Self-administered, giving user flexibility for when they respond x x x x Administered via an operator interview with the ability to guide respondents through the questionnaire x Administered via interviewer with ability to guide respondent through questionnaire; therefore, low respondent burden x Inexpensive and easy to contact respondents when an e-mail address is known With interceptor staff present on site, immediate survey or technological help is available x x Low nonresponse error because respondent "can't say no" in person x Survey design Ability to provide interactive content like maps, customized screens, etc. (not possible with non-web-based survey methods) x x Allows complex questions to be asked while keeping the survey simple for the respondent x x x Multi-method and validated geocoding Error checking x x x x Sampling Allows for targeted sampling of a population x x x x x "Captive audience" with face-to-face contact x x Data collection Centralized data collection x x Respondent keys data x x Interviewer collects data: therefore, low respondent burden x Technology Technology is provided for the respondent x Technology (paper/speech) is universal and built into the survey instrument x x TABLE 5 STRENGTHS AND LIMITATIONS OF DIFFERENT SURVEY TYPES (continued on next page)

clear that the synthesis survey found that coverage error was a major concern on web-based surveys. However, also con- sider that web-based surveys appear to have an advantage over CATI surveys in terms of nonresponse error. There has never been one survey method with the abil- ity to reach all households equally. Therefore, studies with multi-method approaches and that use the optimal survey method(s) to target the sampling population are the “best” practice. A detailed discussion of multi-method approaches 24 follows later in this chapter: Multi-Method Surveys to Mit- igate Coverage Error. Budget constraints on a study will determine which methods to use; however, if optimal sur- vey methods are considered, costs should be mitigated somewhat by using the most efficient survey method(s); controlling costs may only be a matter of managing the number of survey methods. Responses to the synthesis survey indicated that optimized multi-method surveys are the current state of practice, with Considerations Survey Methods: Online Web CATI (Computer- assisted telephone interview) Computer- Based Surveys, Not Online Paper-Based, via Hand or Mail-out In-Person Interviews Limitations Coverage While improving, coverage issues are still a problem x Limited ways to randomly sample a known geographic area x Coverage error is a problem among very low income populations and is a growing issue for other populations (often in urban transit environments such as major metropolitan areas) due to “mobile- phone-only” households who make up an increasing share of the U.S. households. This issue is particularly concentrated among young and mobile people, causing coverage issues that are becoming significant as households drop their land lines. Mobile phone lines are not included in CATI sampling frames. x Only allows for a targeted segment of the population and cannot be used for wide, random geographic sampling x x x Administration Delivery to respondent not as efficient as electronic delivery x x x x Requires computer proficiency x x Respondent must have the time to respond at time of contact x x x Language Requires literacy x x x Those without computer literacy are less likely to respond x x Spoken language/native tongue issues can be problematic x x Nonresponse Error Nonresponse issues due to spam filters and the abundance of spam messages that do not get filtered, causing potential respondents to ignore many e-mail messages x Call screening is a significant non-response issue that may systematically exclude various subpopulations x Data Data quality can be low due to the inability to validate user input x Requires that data be coded again into digital form, adding further input cost and error x x Static format limits the types of questions that can be asked. x x Costs Range from low to high depending on the complexity of the survey and the method of recruitment. Costs can be very low for recurring surveys, as the marginal costs of re-contacting a respondent are extremely low. x Range from low to high cost depending on the extent of the sampling frame. Costs can be particularly high when contacting potential respondents multiple times through reminders and pre-survey instruments to try and encourage response. x Typically expensive due to the high cost of reaching respondents and because it uses an interviewer to administer the survey. x Typically expensive, as requires on-site staff x x TABLE 5 STRENGTHS AND LIMITATIONS OF DIFFERENT SURVEY TYPES (continued)

25 two-thirds of those conducting web surveys implementing multi-method surveys to improve the response of their sam- pling population. SURVEY ERROR CONSIDERATIONS IN WEB-BASED TRANSIT SURVEYS There are a variety of survey error issues for researchers and agencies to consider when using web-based transit surveys. All survey methods have survey errors; therefore, each survey method’s errors must be understood in the broad context of all available survey methods, so that the transit researcher can understand which survey method is best in a given situation. It is also important to know when it is appropriate to use mul- tiple survey methods in a study to improve response and to mitigate and minimize survey error for the entire study. Sur- vey error can include coverage error, nonresponse error (both unit nonresponse and item nonresponse), measurement error, and sampling error. This chapter focuses on coverage error and nonresponse error, which are both seen as critical issues for web-based transit surveys based on the results of the syn- thesis survey. Measurement error is discussed briefly as part of the multi-method survey approach. However, the primary concern of transit researchers is coverage error in web-based transit surveys and therefore first addresses this topic. COVERAGE ERROR Results of the synthesis survey make it clear that potential bias from coverage error in web-based transit surveys is a primary concern of transit researchers and agencies. When asked “What do you think are the disadvantages of web-based sur- veys?,” all of the transit researchers currently conducting web surveys cited coverage error and/or sampling bias owing to coverage concerns. When asked “What do you feel are the rea- sons your organization does not conduct web-based surveys?,” two-thirds of researchers not currently conducting web sur- veys also cited coverage error/sampling bias as reasons. Coverage error occurs when a potential respondent within a population cannot be accessed by the survey method being used. Good sampling practice aims to ensure that all mem- bers of the population of interest have a chance of being sam- pled for the study. For example, absence of Internet access for a potential respondent to a web-based survey would be considered coverage error, as would lack of telephone access for a potential respondent to a telephone survey (1,6). In light of concerns about coverage error in web-based surveys, transit researchers must be able to measure potential coverage error in their target populations and understand how much importance to place on coverage error when choosing a survey method for their study. Coverage error varies depending on the respondents being targeted and the survey method. Coverage error can be measured using pri- mary data and/or secondary data. Measuring Coverage Error Using Primary Data The ideal way to measure coverage error is with primary data (i.e., information regarding the actual survey administered) from the sampling population that is being targeted. The typ- ical sampling frame for most transit agency researchers is their current and potential ridership; primarily people within the geographic area in which they operate. Information about web penetration rates for people in the sampling frame can help determine whether there is reason to consider using web-based research and how much of a concern coverage error should be. A major finding from this synthesis research is that, for any research the transit agency conducts, web- based or otherwise, respondents be asked the following: whether they have web access; if that access is at home, at work, or both; and the speed of their web connection (this can help to understand potential nonresponse as a result of slow connections). If they do have access, it is also critical for the transit agency to collect their e-mail addresses so that they can be put into a customer database that can easily be tapped to survey customers again in the future. Twenty-seven of the 36 respondents who completed the survey conducted for the synthesis were transit agencies. Of those 27 agency respondents, 6 provided data on their customers’ web penetration and the remaining 21 (77% of the agencies surveyed) answered “did not know” to the Inter- net penetration question, which asked, “Do you know what percent of your customers have Internet access?” (Another 11 nonagency transit researchers who completed the survey were not applicable to this analysis.) The average customers’ web penetration reported by transit agencies in the survey was 71% (ranging from 50% to 90%), which is nearly iden- tical to current national statistics that report web penetration at approximately 72% (7). It is interesting to note that some transit markets reported very high web penetration, up to 90%, and the Tri-County Transportation District of Oregon (TriMet) noted that its research found web penetration levels were higher for transit riders than for non-riders. Therefore, transit researchers must not assume that web-based surveys of their sample populations will necessarily result in high coverage error. That the remaining 21 agency respondents answered “did not know” to the Internet penetration question likely repre- sents the more important statistic of this analysis, because it shows that many agencies have not yet conducted research to determine their customers’ web penetration numbers, includ- ing some large urban agencies. Measuring Coverage Error Using Secondary Data If primary, or internal, data are incomplete or unavailable, potential coverage error in web-based surveys can be deter- mined using secondary research on web penetration. This research, often national in scope, can be particularly helpful in determining web penetration of non-rider populations, a group

26 that most transit researchers may find more difficult to conve- niently sample as opposed to sampling their own riders. A rea- sonable understanding of web penetration rates can be found from U.S. Census data (by state) and other secondary data, as well as from anecdotal research in the transit agency’s geo- graphic area (e.g., research by businesses that have workers with web access, etc.). Businesses targeted as potential sources of new riders will often be able to inform the transit researcher about employees’ web access at work. There are a variety of sources for web penetration data, the most comprehensive being the U.S. Census’ Computer Use and Ownership from the Current Population Survey. Although comprehensive, the Census data tends to be older than other sources, making it less useful than more current data sources. Internet usage is growing at such a significant rate that even 3-year-old data may be considered out of date. Furthermore, Census data only tracks computer/Internet usage at either home or work and does not provide one number that includes total access penetration regardless of location, thereby underestimating the population’s overall access rate. One very credible data source for the United States is the Pew Internet & American Life Project, which tracks total cover- age wherever this usage (access) occurs (7). Their September 2005 Tracking Survey contains statistics on Internet usage (see Table 6). As can be seen from this table, there are income, geo- graphic, race, age, and gender factors in Internet penetration data; however, overall access penetration is relatively high at 72% and growing quickly (Figure 17). Actual effective access penetration may be slightly higher, as potential respondents may have Internet access at Source: Pew Internet & American Life Project September 2005 Tracking Survey (7). Total Adults Women Men 18–29 30–49 50–64 65+ White, Non-Hispanic Black, Non-Hispanic English-speaking Hispanic Urban Suburban Rural Less than $30,000/yr $30,000–$49,999 $50,000–$74,999 $75,000+ Less than High School High School Some College College+ Dial-Up High-Speed Home Internet Users 39% 59% 92 Educational Attainment 38 62 82 54 78 87 94 75 73 65 Household Income 73 60 79 Community Type 83 71 30 Race/Ethnicity 69 75 Age 84 Demographics of Internet Users Use the Internet (%) 72 TABLE 6 U.S. INTERNET USAGE BY DEMOGRAPHICS FIGURE 17 Computer and Internet access data from the U.S. Census (8).

27 schools, libraries, and other public places; however, it is unlikely that many respondents would make the extra effort to go to one of these locations explicitly for the purpose of completing a transit survey. Despite the high penetration rates, when conducting a study that will use web-based surveys, either as the only survey method or as part of a multi-method survey design, it is important to ensure that the population targeted for the study’s web sampling frame is on the high side of these Internet usage statistics. For many of these populations, such as suburban and urban adults less than 64 years of age, either college educated or with incomes of more than $30,000 per year, the incidence rate of Internet access is between 75% and 94%. These inci- dence numbers are high and for certain subpopulations are approaching the national telephone incidence rate of 97.6% (U.S. Census). Therefore, web coverage, although not quite as good as telephone coverage, is inclusive for many popu- lations. This synthesis will further explore and discuss cov- erage error in the following section, where it will be shown that coverage error, although higher for web-based surveys than telephone surveys, may be mitigated by lower non- response error in web-based surveys compared with tele- phone surveys. NONRESPONSE ERROR IN WEB SURVEYS The two types of nonresponse error—unit nonresponse and item nonresponse—are also of concern to researchers. Unit Nonresponse in Web Surveys Unit nonresponse error occurs when survey respondents dif- fer from nonrespondents in a way that is significant to the study. For example, if low-income transit riders respond to a telephone survey in disproportionately higher numbers than high-income riders there would be nonresponse error for higher income riders. Often nonresponse error can be mitigated through procedures that weight up underrepre- sented groups and weight down overrepresented groups. Weighting may successfully mitigate nonresponse error as long as there are enough underrepresented respondents to provide reasonable statistical confidence, assuming the peo- ple who responded are similar to those who did not respond. However, if nonresponse error results from significant underrepresentation of a particular subpopulation of the sam- ple (e.g., high-income transit riders who only use transit in the evening, and thus were unavailable when the telephone survey was being conducted), the more serious issue arises of systematically excluding a particular subpopulation of the sample whose behavior is different in a way that is important to the study. In this example, an effort would need to be made to ensure that the high-income, evening-travel subpopulation would somehow be included in the study. One method might be for the transit agency to offer a web-based survey option in addition to the telephone option. Web-based surveys have advantages over telephone sur- veys in terms of unit nonresponse. Web surveys do not suf- fer as much as telephone surveys from the issue of high unit nonresponse rates as a result of call-monitoring techniques such as answering machines and caller ID (1,4,9). This is because there is much less active screening of e-mails than of telephone calls (passive spam filtering, which is a serious problem, is described later). Therefore, if an e-mail arrives in a respondent’s inbox with a subject of interest, that individ- ual may be more likely to respond to it than to a telephone call with an unknown number. However, spam is a very serious issue for e-mail and has become increasingly problematic over the last 5 years. As most users of e-mail understand through experience, various types of spam filters are becoming standard at most compa- nies and organizations, as well as at home, through a variety of software products. Therefore, spam issues are a concern that must be addressed when conducting web-based surveys. The most important issue for web-based survey research regarding spam is to avoid survey invitations being tagged as spam and filtered out of a respondent’s e-mail in-box before it is even seen. There are a variety of methods to increase e-mail delivery rates and avoid false positives (i.e., a message being tagged as spam although it is legitimate). First, all bulk e-mail senders should adhere to the industry accepted, and federally mandated, e-mail practices outlined in the CAN-SPAM Act of 2003: 1. Bulk e-mail must clearly identify the sender—including a physical address. 2. Bulk e-mail must contain a valid subject line and valid routing information. 3. Bulk e-mail must contain a working opt-out mechanism. Second, many companies offer paid white-listing ser- vices, promising increased delivery through partnerships with e-mail providers. Senders often undergo an e-mail- practice audit and, if accepted into the program, are added to a white-list used by e-mail providers to allow delivery with- out spam filtering. Third, avoiding certain words and phrases common to spam will decrease the likelihood that legitimate e-mail will be incorrectly marked as spam. Words and phrases such as “free,” “easy money,” “gamble,” “money,” and “rich” are commonly found in spam, and should be avoided if at all possible. Lastly, ensure that the source of the e-mail (ISP) does not tolerate or conduct business with known spammers. An ISP can have its e-mail servers black- listed across the Internet for doing business with known spammers. Anyone sending e-mail using the same system will be subject to the blacklist rules in effect and the e-mail will never reach prospective users. Spam filters are not the only nonresponse issue related to web-based surveys. Many respondents receive significant

legitimate e-mail as well as significant spam. This means that many potential respondents are fairly ruthless about what e-mail they read versus e-mail they discard (6). Therefore, the key for transit researchers is to ensure that their e-mail invitations are to the point and understand what will be of interest to respondents so that these invitations are read and acted on (e.g., “Improve your commute” or “Tired of sitting in traffic?” or “Contribute your opinion on a new transit alternative”) (1). Another nonresponse issue with web-based surveys is that some respondents simply do not check their e-mail very often, or at least not the e-mail address that they provided the transit researcher. Still other issues include multiple e-mail addresses, undeliverable e-mail addresses, server errors (e.g., the respondent’s ISP happens to be conducting server main- tenance when the e-mail is sent and therefore it bounces back). Clearly there are a number of issues and concerns regard- ing nonresponse in web-based surveys. However, assuming the researcher’s e-mail invitation is not tagged as spam (which as explained previously there are ways to mitigate), the e-mail address is correct, the topic is of interest to the respondent, and the invitation is concisely and clearly writ- ten, then unit nonresponse error can be significantly miti- gated and respondents should at least begin the survey (whether they complete the survey is the issue of item non- response, discussed in the next sections). Nonresponse can be reduced significantly if researchers are diligent in managing their customers’ e-mail address lists, such that they contain only valid addresses (or at least e-mails that have not been returned to sender). If researchers’ e-mail lists are valid and current, their respondents will be familiar with the organization from previous e-mail corre- spondence and/or web-based research and may be more inclined to open and respond to the researchers’ e-mail requests. Response rates of 50% and higher are not uncom- mon for well-managed lists or panels. It should be noted, however, that the researcher must not create a self-selecting list of those with a greater propensity to respond by simply throwing out e-mails of those who did not respond to prior invitations. Item Nonresponse in Web-Based Surveys Item nonresponse refers to the issue of missing or incorrect data items in questionnaires. Item nonresponse occurs when respondents skip questions or fail to complete a question- naire. Self-administered questionnaires, such as web-based and paper-based surveys, typically have more item non- response than questionnaires administered using interview- ers (4). Although web-survey questions are often validated, making it hard or in some cases impossible to skip a question, 28 item nonresponse can occur as a result of a different form of item nonresponse called “break-off,” where respondents simply fail to complete the questionnaire. There are a variety of ways to mitigate item nonresponse and a large body of literature exists on the topic (1, p. 529; 6, p. 555). One method to reduce item nonresponse is to ensure a high level of interest among potential respondents to the survey (1). Fortunately, there is often a high level of interest in transit and other transportation surveys because respon- dents have a strong desire to improve their commutes and other travel. Another way to mitigate item nonresponse is to remind respondents who have started a questionnaire that they have not finished and should continue on and complete it. This is a major strength of web-based surveys compared with telephone or mail surveys. The costs of e-mailing a reminder are very small, and there is minimal concern that the respondent is not receiving the reminder e-mail because they have already responded to the questionnaire invitation. One of the benefits of web-based surveys (recorded by 71% of respondents to the synthesis survey) is their ability to obtain clean data through consistency checks and validation of user responses, essentially eliminating item nonresponse for those respondents who complete the questionnaire (1). At the same time however it is important that real-time editing and response validation in web surveys do not dissuade the respondent from continuing their questionnaire because the checking and editing become too onerous. The difficulty of balancing validation while encouraging respondents to com- plete questionnaires can be seen when online geocoding is used in web-based surveys. Online geocoding is an important benefit for transit and transportation applications because it yields precise and val- idated address information, often critically important to tran- sit researchers and almost impossible to collect accurately using other survey methods. However, online geocoding is still not perfect. Occasionally, a respondent may enter their home address into a survey only to find that the survey (through a real-time geocoding check) insists their home address does not exist (owing, for example, to an outdated GIS data set). The respondent checks their typing, tries to proceed, and again is told the address does not exist. This will often cause the respondent to become frustrated and abandon the survey, either because there is no way to proceed or because it is simply too onerous for them to proceed by using the alternatives presented. The fundamental challenge for any web-based survey is to guarantee a balance between validating data to reduce item nonresponse and allowing respondents a way to proceed through difficult questions without causing break-off, which is a different form of item nonresponse.

29 This is not an easy balance to strike, because a successful geocode may be imperative to construct a stated-preference experiment for the respondent later in the survey. Therefore, the survey must collect geographical data or it loses signifi- cant value for the researcher. One technique to resolve the geocoding issue is to ask the respondent twice for their address to ensure it is typed correctly. If the database still fails to find it, the respondent may then be automatically taken to a map screen and asked to indicate their address using the map tool that cannot fail as the result of an incomplete database. This technique has been employed in a number of transporta- tion web-based surveys, including a significant transit study currently underway evaluating a new transit service between Lower Manhattan and JFK Airport. The following list presents some key considerations for web-based researchers to use when designing the validation rules in their surveys (10): • Type of edits (e.g., format, conditional, and consistency edits), • Number of edits (i.e., determination of priorities), • Optimal timing of edits (after each question or just before the questionnaire is completed) with respect to recall versus burden, • Presentation of edit failures to respondents, • Wording of error message (particularly ones calculated by the system), • Design and format of the message (e.g., color and background), • Use of hard edits (forced to fix in expected manner) or soft edits (either reconcile the error or provide comments), • Design and management of previous or complementary external information, and • Help facilities provided (e.g., additional instructions, telephone support, and e-mail responses). Although there are issues with item nonresponse error in web-based surveys, real-time validation and editing remain as major benefits of conducting web-based surveys. The details of how, when, and whether to validate are important and must be considered by all web researchers (transit or otherwise). SUCCESSFUL PRACTICES AND CHALLENGES IN CONDUCTING WEB-BASED TRANSIT SURVEYS The previous sections are intended to provide the researcher with an understanding of the strengths and limi- tations of different survey methods compared with web- based surveys and of the various issues to consider when choosing which survey method(s) to use to best serve their study. This section discusses how to proceed in conducting a web-based survey once the decision to conduct such a survey, either alone or in conjunction with other survey methods, has been made. Recruiting and Sampling Techniques for Transit Web-Based Surveys There are a variety of ways to obtain web-based survey respondents once the sample population is understood and considerations regarding coverage error have been addressed. The study objective is also critical in deciding not only who, but how, to sample. The following section describes two different types of studies; one for a wide geographical cover- age and the other for a more targeted sample. Wide Coverage Study Sometimes it is necessary for the transit researcher to understand how their entire geographic region feels about a transit-related research topic. This includes both riders and non-riders for all demographics across the entire transit area. For example, if the transit researcher is concerned with who uses transit and why, it may be necessary to randomly select households throughout the region using random digit dial or address-based sampling for a mail-out. Although there are a number of concerns about nonre- sponse error with random digit dialing and with mail sur- veys (1), these are still two of the most effective techniques to randomly sample the population of a large geographic area. One of the limitations of web surveys is that there is currently no way to generate a random list of e-mails for potential survey respondents in a particular geographic region. This limitation makes contacting random samples of wide areas difficult for web surveys. Nevertheless, random digit dial could still be the recruit- ment method: the transit researcher can contact potential respondents in the study area by phone, obtain their e-mail address, and send respondents an e-mail invitation with a link to the survey. The researcher can also give the respondent the option of taking the survey over the phone at that moment. Completing the survey at that moment over the phone may be convenient for the respondent, thus increasing the overall response rate for the study; however, phone completions may reduce the number of respondents taking the survey by means of the web. Offering a web completion option can also help increase response rates to CATI surveys: an interviewer can send an e-mail invitation with a survey link to respon- dents who are resistant to completing the survey on the tele- phone. This method has been used with limited success in at least one recent study. As part of a paper-based, mail-out/mail-back survey, a researcher can print a web address and unique password on each survey, providing the respondent with the option to take the survey by means of the web.

Although both mail and telephone survey methods are the most effective methods to ensure a random selection of respondents in a wide geographical area, both require the relatively expensive methods of contacting respondents by phone or mail. Once a respondent agrees to cooperate over the phone with a CATI operator, there is often little cost savings to having them take the survey on the web because, whereas a web response is less expensive than a CATI operator conducting the interview, there is no guarantee that the respon- dent will actually follow up and take the survey online. Fur- thermore, the greatest cost of a CATI survey is reaching a respondent and gaining their cooperation. Because this has occurred, it is logical to see the survey through to its conclu- sion. Offering a respondent a web survey option on a mail-out survey might increase response rates by giving respondents a more convenient method to take the survey; however, those respondents may have completed the questionnaire anyway using the mail-back method. Therefore, it can be said that respondents themselves may gain some benefit by using their preferred survey method; however, the actual response rate may or may not increase; in the meantime, the costs related to creating and administering the additional web-based survey instrument have still been incurred (1). Targeted Sampling for Riders A broad geographic sampling frame may not be necessary for many studies that a transit agency might be conducting. When this is the case, web-based sampling often becomes a strong survey option. Riders Most transit studies do not require a random sample of a large geographic population. For example, when transit agencies need to sample their ridership, they know how to find them: they are on board the vehicles, at the stations and terminals, and possibly in their customer database. There- fore, researchers conducting rider origin–destination and customer satisfaction surveys, for example, will be able to directly intercept riders using a paper-based, hand-out sur- vey or a personal interview on board transit vehicles or at transit stations and facilities. With a hand-out or interview survey, it is usually very easy to ask for an e-mail address on the handout instrument or, in the case of the interview, to directly ask for an e-mail address. Although onboard paper surveys are effective because the rider is “captive” during their transit trip and has the time to fill out a survey, offer- ing a web-based option or web-only survey can allow the transit researcher to conduct much more complicated surveys and to develop a customer database for future research needs as discussed later. That the rider will need to access the survey over the web later (and not right there on the transit vehicle) is not ideal, but as noted earlier, a self- administered web survey provides the ability for respon- dents to log in when it is convenient for them. 30 Customer Database What is ideal is that once an e-mail address is obtained from the sample population, compiling a customer/potential cus- tomer database with e-mails is a powerful incentive to con- ducting research with riders and non-riders alike. Creating this type of database is a particularly important tool for rider research, because many riders use the transit system over a span of years, and obtaining their e-mail address allows them to be easily contacted later for any research the transit agency might require. For example, 65% of NJ TRANSIT commuter rail riders have been riding the system for two or more years (2). Surveying this group, which has a large rider database, becomes a matter of creating the sample objective and send- ing out a batch of e-mails inviting respondents to participate. Although there is some additional fieldwork involved, because an existing database should be regularly updated to ensure new riders are being included and that the list stays current with changing demographics, conducting fieldwork to update and maintain a rider database is a much less oner- ous task than having to obtain a large sample for every study. To obtain such a database, a transit researcher starting from scratch with no customer list can send staff into the field and collect a very large sample of customer e-mail addresses. This can be done by asking for customers’ e-mail addresses using interviewers or a simple, onboard, paper-based card questionnaire (see Figure 18). The presence of an e-mail address is a strong indication of web penetration within the transit area. Once the researcher has a list of rider e-mail addresses, web-based studies can be readily conducted. As the list matures, it will need to be reg- ularly updated owing to respondents who opt out or indicate that they have stopped riding the transit service and to add new riders into the customer database. Finding new cus- tomers and other customers not on the list can be done with the same card and intercept methods as used to compile the original list, but targeted on specific types of riders needed to complete the sampling frame for the customer database (see chapter six for examples of two projects that collected cus- tomer lists using web surveys). Non-Rider Targeted Sampling Many surveys are project-specific, whereby certain targeted populations are needed to evaluate new service initiatives. For example, if a new light rail system is proposed, a mode choice study will be necessary to understand the ridership potential for such a system. For studies such as this, ran- domly intercepting respondents in the area of the proposed new service is an excellent way to sample. These respondents can either take the survey on the spot or later using the web (after providing an e-mail address to the researcher). When the targeted study population of a survey is non-riders of transit, web-based surveys can be very useful. Often one of the most important things for transit agencies to understand is

31 why people are not using transit and what the agency can do to entice non-riders to switch to transit. Although random sam- pling would be the best way to understand non-riders’ needs in a transit agency’s territory, convenience sampling can be very effective and can be done without incurring the high costs asso- ciated with random sampling by means of telephone and mail surveys. Large employers in the transit agency’s area of oper- ation can provide a good base for convenience sampling for a web-based survey on how to increase transit ridership (assum- ing employees have easy access to computers). Sampling of large employers in the study area can be sup- plemented with intercept surveys of potential respondents at public areas such as malls, department of motor vehicles offices, highway rest stops, and high-traffic pedestrian areas that are in locations relevant to the study. E-mail addresses can be obtained directly or through a simple hand-out/ hand-back instrument and added to a database of potential respondents who might be surveyed. MULTI-METHOD SURVEYS TO MITIGATE COVERAGE ERROR Earlier discussion of mitigating coverage error focused on understanding whether a significant coverage issue exists within a transit research sample. Furthermore, it has been discussed that for certain subpopulations, such as current non-riders who could be surveyed to understand what actions a transit provider could take to encourage them to ride, coverage bias may not be an issue. This being said, there are often some coverage issues and these warrant the use of a variety of survey methods to take advantage of the benefits of each different survey method type. An important method to mitigate coverage error is to develop studies that use multi-method sampling techniques. In other words, use a variety of survey methods to conduct a study and allow respondents to choose which method is most convenient, thereby increasing the study response. Many transit studies have been conducted using multi- method surveys. NJ TRANSIT’s Rail Customer Satisfaction ePanel study is discussed in chapter six. This was primarily a web-based survey; however, respondents who were inter- ested in the study but did not have web access were given a phone option. The respondent was called and, if reached, surveyed by an interviewer who used the web-based instru- ment as the CATI script. The interviewer therefore was reading from the exact survey that the respondent would have used had they logged onto the web and taken the survey themselves. A more typical multi-method survey in the transit context is seen when an onboard paper survey provides a web link; therefore, the respondent has the option to participate in that way. FIGURE 18 Sample handout card requesting name and e-mail address.

The survey conducted for this study discovered that every type of survey in the study (origin–destination, customer sat- isfaction, mode choice, planning, and many of the “others”) had incorporated some multi-method techniques; overall, 27% of all surveys described for this synthesis used multi- method techniques and some of these incorporated four or five different methods in one survey. Two-thirds of transit agency researchers who currently conduct web-based surveys are including multi-method tech- niques. As Table 7 shows, there is no one perfect survey method that easily captures all populations. Implementing a multi-method survey can introduce addi- tional expense to the overall cost of the survey, and it can introduce significant measurement errors, meaning that the same question may be answered differently because of the particular survey method being used (1). However, web- based surveys can be combined with other methods without introducing measurement error by programming a web- based survey that can be ported directly to laptop computers and set up at central sites in public places within a given study area. The research team has implemented this tech- nique in several recent projects (e.g., MTA–New York City Transit’s JFK Airport–Lower Manhattan 2005 Study, and the NY State Thruway Authority Westchester, Rockland, & Orange County Travel Study 2003) with success. With this arrangement, the survey can be administered by intercepting respondents in person or it can be taken by respondents directly over the web, whichever is most convenient for respondents to obtain the highest possible response rates. This strategy of programming a computer-based survey is employed frequently in many mode-choice studies because these surveys require complex structures to build customized future scenarios for respondents to choose from based on the respondents’ unique trips. For a mode choice study, anyone in selected public places within the study area of a potential new transit project may be a valid respondent to determine the viability of a new transit service. Respondents wishing to participate online may provide the interceptor with an e-mail address and be sent an e-mail invitation to take the survey on the web, or the respondent may be given a flyer with a web link (and preferably a unique password to ensure one survey per person) so that they can access the survey at their convenience. Again, the only difference between the computer-based, self-administered survey and the web-based 32 survey is that the web-based survey is transmitting the data by means of the Internet, whereas the laptop intercept survey in the field is reading and writing directly to the hard drive. Other examples of multi-method techniques include using a personal interview survey in combination with a web-based survey. A study to measure response to subway station reha- bilitation provides an example of this combination method. New York City subway riders who preferred not to engage in a personal interview at the subway station (because their train was coming or they needed to exit the station quickly to get where they were going) were asked for their e-mail address so they could be sent an e-mail with a password- embedded link to the survey containing the same interview questions. Well over 50% of those asked willingly provided their e-mail addresses directly to the interviewer. CONCLUSIONS This chapter described the research context in which web- based surveys are one of a number of survey methods. There are many ways web-based surveys can be incorporated into studies that benefit the research, often at low additional cost. Furthermore, as transit researchers become more familiar and comfortable with the tools, processes, and/or outside firms they can use to create and administer web-based sur- veys, more applications of web-based surveys will become apparent to the transit researcher. These applications will grow as web penetration rates grow. Web-based surveys are appropriate in the following situations: • Respondents have reasonable web penetration. Good examples of high web penetration situations are employer surveys and surveys of non-riders, where the incidence of web access may be higher among these special populations (i.e., students). • The survey has complexity that is best handled by a computer-based instrument. Many surveys require sig- nificant complexity to obtain useful information. A good example is stated preference mode choice sur- veys, where customized future transportation scenarios need to be constructed for each respondent. Although the survey itself is simple and straightforward for the respondent, there is significant behind-the-scenes pro- gramming used to resolve this complexity. The ability to survey respondents effectively using sophisticated methods allows the researcher to obtain the critical data he or she needs while making the survey experience simple and clear for the respondent. • Another example where survey complexity can be addressed through web-based surveys occurs when origin–destination geographical data needs to be col- lected, as mentioned previously in the New York MTA Bridges & Tunnels Origin–Destination Study 2004. TABLE 7 PERCENTAGES OF SURVEYS USING MULTI-METHOD TECHNIQUES Types of Surveys Using Multi- Method Techniques Percent Planning 46 Other 24 Origin–destination 19 Customer satisfaction 18 Mode choice 30

33 Geographical data are critical to transit research for a whole host of purposes, such as commuter sheds, sta- tion development, operations planning, and mode choice, to name but a few. Valid geographical data are difficult to collect and item nonresponse is a major issue for paper-based surveys and even CATI surveys where the interviewer is unfamiliar with the geography of the study area. Web-based surveys enable respondents to input geographical data that can be validated in real time and can be done in such a way as to mitigate break- off concerns as described previously. • Quick, “pulse-taking” surveys for a variety of purposes can be accomplished using web-based surveys. An example follows as a case study in chapter six in which TriMet asked respondents for feedback on its new inter- active map feature on its website. • Information may be needed for a specific purpose such as evaluating features of regional fare cards [San Francisco Bay Area Rapid Transit (BART) and MTA NYC]. E-mail invitations may be sent to known customers of the various agencies, who will likely respond because they recognize the sender of the invitation as being their own transit provider. • The survey continues over time. Web-based surveys are excellent for longitudinal studies (as discussed in detail in two case studies in chapter six), because web-based surveys significantly reduce the costs of contacting respondents multiple times, which is necessary for lon- gitudinal studies. Once a web-based survey is pro- grammed and designed, there is a very low marginal cost for obtaining additional surveys. This is especially true if the recruitment is conducted using an e-mail list of respondents. To conduct a new survey, the researcher simply has to send an invitation to the appropriate respondents at the appropriate time. This can be done using automated tools; therefore, obtaining a new wave of respondent data requires very little time and expense on the researcher’s part. Additional benefits of longitudinal studies include the ability to know what a respondent answered in their previous survey and to then ask them if anything has changed since they last took the survey. This function allows the researcher to “drill down” by noting changes from prior surveys and then asking respondents in real time the reason for the change. • Web-based surveys are an excellent option as part of a multi-method survey approach. As described earlier, web-based surveys are often very good paral- lel or supplemental survey instruments to other meth- ods being used directly in the field (e.g., paper-based surveys or field intercepting to a central site with computers).

Next: Chapter Five - Technology »
Web-Based Survey Techniques Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

TRB's Transit Cooperative Research Program (TCRP) Synthesis 69: Web-Based Survey Techniques explores the current state of the practice for web-based surveys. The report examines successful practice, reviews the technologies necessary to conduct web-based surveys, and includes several case studies and profiles of transit agency use of web-based surveys. The report also focuses on the strengths and limitations of all survey methods.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!