Web-Based Survey Techniques (2006)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

This chapter explains the basic technologies and technology issues researchers must address when implementing a web- based survey instrument. Transit researchers typically use one of the following three methods to implement and conduct web-based surveys: 1. Use a survey service and/or software—This method is inexpensive, but only allows predetermined question types and is therefore very limited in its flexibility. It is a viable choice for creating simple surveys, but ser- vice from the provider is minimal and one must be aware of hidden costs. 2. Create and conduct the entire survey in-house using information technology skills and resources within the researcher’s agency or organization—This requires technical expertise, but gives the researcher complete control. Depending on the skills of the researcher and the resources available, this option can range from being inexpensive to very costly. 3. Hire a consultant—This can be expensive, but pro- vides experience, expertise, and the ability to conduct complex and highly customized questionnaires. Other aspects of a survey, such as sampling plans and non- web survey instruments, recruiting, and reporting may also be addressed by a consultant. Responses to the synthesis survey indicated that most researchers contract out their web-based survey work to consultants, but that online survey tools and in-house devel- opment are often used as well (Figure 19). Researchers using a consultant for survey development also had those surveys hosted by the consultant. The same is true for those who developed surveys using online tools. Researchers electing to develop the survey in-house also hosted the sur- vey in-house. All three options might be used by the same transit researcher or agency for different reasons, depending on the needs of the study. For example, a survey-service question- naire from a provider could easily be created to supplement a simple paper-based instrument. For the annual cost of approximately $200 (plus other charges), a transit researcher can use such a provider to create simple web-based ques- tionnaires that are analogous to their paper questionnaires. They can then invite respondents to the survey by means of an e-mail list or a web link. 34 Using a survey service means that researchers must direct the survey creation themselves. This includes creating all questions, selecting how to display the questions, and deter- mining all the logic and validation rules. Researchers are also directly responsible for all respondent recruiting and cus- tomer support. There is likely no direct help for the researcher using such services; therefore, any questions or problems are usually answered by e-mail on the time frame of the service provider (not necessarily the time frame of the survey researcher). For many transit researchers, survey services are a very good solution to develop a survey at low cost and to learn first hand about web-based surveys and how the process works. However, researchers often find that online services and generic survey software do not meet their needs. For example, longitudinal surveys cannot be created that track one respondent over time using such tools. Nor can stated preference surveys for mode choice studies be produced effectively using less expensive online survey services, although there is much more expensive software that does allow for advanced online mode choice surveys to be created. Features such as online geocoding and linking transit sched- ules are typically not incorporated into these surveys. Advanced validation cannot be accomplished, as these tools are not capable of, for example, comparing a zip code with a data table of zip codes to confirm if a respondent’s answer is an existing zip code or not. In the case of more advanced needs and sophisticated sur- veys, a consultant is often hired to conduct the survey unless there is significant expertise in-house, although as men- tioned earlier there is software available that allows transit researchers to do more of this themselves. Although some advanced software is very good, it does not mean that the process is simple; therefore, a consultant is hired to help implement or guide a study using third-party software. Table 8 lists some of the strengths and limitations found with different survey providers. The following list addresses various technologies that are important to consider when conducting a web-based survey. • The web-based technology itself—is it server driven? Server-driven technology for web-based surveys is important, because it means that most of the logic and technology reside on the survey provider’s server as CHAPTER FIVE TECHNOLOGY

35 18% 27% 55% 0 20 40 60 80 100 Designed in-house using web page layout software Designed with online survey development tool Contracted out to consulting or web development firm Percent FIGURE 19 Web-based survey development methods. Approach Strengths Limitations Contract out to consulting or web development firm Provides experience and expertise Can develop complex, highly customized questionnaires Expensive Not many available consultants with high degree of expertise Ability to display any desired graphics/logos on survey pages Allows recruit by any method Can develop sampling plans Can develop non-web survey instruments to accompany web-based survey Can develop complex (e.g., multi- dimensional) weighting schemes as part of the study May provide advanced data validation; i.e., trip lengths in time and distance May provide online geocoding Provides advanced longitudinal survey capabilities, such as presenting respondents with the option to simply confirm previous answers from previous waves of a longitudinal survey May provide ability to search open-end answers in database in real time May provide ability to link transit schedules and maps of systems Hosts survey on its own servers Can develop password scheme to screen out multiple answers per respondent Can see results live Provides clean, validated dataset Will provide data in format require by agency TABLE 8 ADVANTAGES AND DISADVANTAGES OF VARIOUS WEB-BASED SURVEY DEVELOPMENT METHODS (continued on next page)

opposed to requiring this technology to reside on the respondent’s computer through a late-version browser with sophisticated technologies. The objective is for the survey provider to be responsible for as much technol- ogy as possible on their end, thereby enabling older computers and software to still easily run the survey and to benefit from its advanced features. • Recruiting—are you able to recruit using e-mail? Can you use your own e-mail system? Will you be sending reminders, or merging fields into the e-mail? • Security—does the survey use an authentication system so that each respondent can take the survey only once? 36 This is often very important for transit studies, as there are frequently proponents and opponents to many of the projects that a transit agency is attempting to imple- ment. A web-based survey that allows the same user to repeat the survey again and again can create a serious bias in the survey. This problem is easily taken care of if each respondent is provided a single, unique pass- word that allows him or her to take the survey only once. • Can the survey be made multi-method? As discussed in chapter four, multi-method surveys are often an excel- lent way to conduct a transit survey. Can the web-based Use online survey development tool Inexpensive, although costs can be hidden Requires predetermined question types *Some examples include: Easy to use for simple questionnaires Limited flexibility in questionnaires by templates used to create them Allows recruit by e-mail or web link Limited ability to display graphics/logos on survey pages Agency can “get their feet wet” and try out new technology May require software downloads Can see results live Must program survey oneself May provide managed list of opt-in survey respondents Must devise own data validation No need to involve information technology (IT) department Must create own logic scheme SurveyMonkey Zoomerang Infopoll SurveyCrafter SuperSurvey Perseus Hosts survey on its own servers Must create own survey screens Cannot create stated preference surveys Cannot longitudinally track respondents over time Cannot provide advanced data validation Develop/program/host in- house Dependent on in-house expertise Dependent on in-house expertise Complete control Significant work to produce complex surveys Can develop complex, highly customized questionnaires May require a high learning curve Can integrate directly to other in-house systems (such as agency website, database, etc.) Requires time and effort of IT department, which is often already busy with many other priorities Can develop sampling plans Can develop non-web survey instruments to accompany web-based survey Can develop complex (e.g. multi- dimensional) weighting schemes as part of the study May provide ability to search open-end answers in database in real time Hosts survey on its own servers Can develop password scheme to screen out multiple answers per respondent Can see results live Provides clean, validated dataset Will provide data in format require by agency *These include those known to the research team at the time the research was conducted. Any omissions were inadvertent. TCRP does not endorse specific products and services. TABLE 8 ADVANTAGES AND DISADVANTAGES OF VARIOUS WEB-BASED SURVEY DEVELOPMENT METHODS (continued)

37 survey be used on stand-alone computers without Inter- net access in a field environment for intercepting respondents? • Hosting—it is almost always easiest to host the data with the same provider as the entity developing and implementing the survey. Although it is certainly pos- sible for a consultant to implement a survey somewhere else (e.g., on a transit agency’s servers), this often adds significant costs without much gain. It is important that the hosting entity has been through a security audit and, if necessary, has been approved by the agency’s legal department. • Data—most survey service providers will provide data in export formats. It is important to ensure that providers give all data over to the client and that data such as how many respondents started the question- naire but did not finish it, the date of the survey, the sample size, the response rate, etc., are all included in the data set. • Real-time data access and reporting—to what extent are up-to-date results required to check the progress and distribute interim results? • Repeatability—once a transit agency has invested in developing a survey can it be reused or repeated? • Technical support—will respondents be able to call or e-mail with questions as they take the survey? • Data validation—does the software allow for error checking and answer inconsistencies? Internet technology has been a hot topic over the last decade, and often the very small details of technology are the focus when discussing any web-based technologies. When imple- menting, these small details are critical, and each web applica- tion requires making a large number of technical decisions. This chapter intentionally focuses on the broader considera- tions of technology in web-based surveys. It is important for the transit researcher to understand the benefits and costs of differ- ent broad technology decisions first. Transit researchers need to understand what they want out of their research plans and this understanding will drive their decisions about which of the three types of technology methods described earlier they should be pursuing. Many researchers might pursue them all. Other researchers might pursue them in varying degrees (e.g., a con- sultant helping them using third-party software).