Cover Image

Not for Sale

View/Hide Left Panel
Click for next page ( 59

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 58
58 Guidebook for Conducting Airport User Surveys It may then be necessary to take corrective action, either by coaching an interviewer who missed something during training or by retraining the entire group if the trainer failed to con- vey a point well enough. The survey planning team would be wise to budget for both of these occurrences, particularly if the questionnaires or procedures are complicated or difficult. It is also important either to train backup interviewers, who may or may not be deployed, or to provide for a second training session for new hires if people need to be replaced. As it is diffi- cult to find people who are willing to be trained without any assurance that they will have the opportunity to participate in the survey, the latter approach is more typical. However, a second training session can present scheduling difficulties because attrition of the trained interviewers can occur throughout the survey. Another problem arises with recruiting and training additional interviewers at short notice if they have to be issued with security badges, because this process can be quite time-consuming. Therefore, it may be better to recruit and train enough interview- ers at the start of the survey to allow for some attrition, and adjust the length of the survey period and the hours worked by each interviewer to manage the attrition that actually occurs. Finally, it is important to note that when mistakes happen, the decision about whether to retain and tabulate work that is imperfect is both a research and a policy decision. The research decision has to do with the necessity of having enough interviews to achieve the desired statistical preci- sion and power; the policy decision has to do with the importance of a given piece of information. It is also important to consider just how imperfect less than "perfect" work really is. 4.7 Pre-Tests and Pilot Tests Pre-tests of questionnaires and pilot tests of survey procedures provide an opportunity to identify any problems with question wording or other aspects of the questionnaire as well as the planned arrangements for performing the survey. Such tests allow corrections or adjustments to be made before the main data collection effort. 4.7.1 Pre-Test of Questionnaires The goal of a pre-test of the questionnaire is to make sure the survey questions can be asked easily by the interviewers, are clear to the respondents, and produce the desired information. Pre- tests also provide an opportunity to identify unanticipated responses or situations and to adjust the question response categories or the survey script as necessary. An initial pre-test may be performed on a convenient group of people who were not involved in developing the instrument, such as staff of the survey sponsor. However, while this may help improve the question wording and identify unanticipated responses that require changes to the question response categories, it is not a truly representative test of the questionnaire. It should be followed by a pre-test involving a sample of the intended respondents of the survey, per- formed in the same way as planned for the survey itself. This pre-test will increase the chances of identifying unanticipated responses or situations. The questionnaire designers should be involved in conducting the pre-test, because they will be able to identify unanticipated responses or any misunderstanding of the intent of a question better than someone who was not involved in the development of the questions. The results of the pre-test should be subject to careful analysis to identify any apparent difficul- ties with the question wording or survey flow, such as missing or incomplete answers, illogical responses, or incorrect skips. In particular, responses of "Other" to categorical questions, where the respondent has provided an explanation, should be examined to identify any misunderstanding of the categories or commonly occurring responses that should be added to the designated categories.

OCR for page 58
Survey Design 59 If electronic data collection devices are to be used in the survey, the programming needs to be thoroughly tested and debugged before the pre-test, or the validity of the pre-test may be com- promised by errors in the programming. A rigorous testing plan for the software should be included in the project schedule. This program testing, done by people other than the program- mers, will attempt to ensure that all possible response options and question branches are tested. This can be quite time-consuming and can be facilitated by preparing a set of response scripts that test all possible branches from each question. These test scripts ensure that certain response options are not overlooked, and can also be used to check that changes made to fix any problems have not affected other parts of the program. The survey schedule should allow enough time to adequately test the programming and make any needed revisions before the first pre-test. 4.7.2 Pilot Test of Survey Procedures The primary purpose of a pilot test of survey procedures is to identify any problems with the planned approach to conducting the survey and to refine the estimates of the number of field staff required. Because a pilot test typically involves performing the survey on a representative sample of potential respondents, it presents an opportunity to conduct a pre-test of the survey questionnaire, and the two tests are often combined. Other objectives of a survey pilot test include the following: Testing the survey sampling strategy. Validating the sampling plan. Identifying problems with survey logistics. Checking the data quality. Feedback from the pilot test should include both field observations by supervisory staff and issues reported by field staff in debriefing sessions, supplemented by an analysis of the survey responses collected. In the case of interview surveys, noting the start and end time on each response will enable this analysis to examine the average time to perform a survey as well as the interval between ending one interview and starting the next. This will provide information on the amount of time spent waiting for subjects to become available to be interviewed as well as the time required to move between locations. Field staff should also note refusals to participate and record the reason where given or apparent, such as a language barrier or cell phone conversation. The pilot test results should be compared to the expected number of responses from the sampling plan. This comparison may require collecting data on the potential number of respondents, such as the number of passengers on flights surveyed in airline gate lounges. The results should be exam- ined for data quality, including missing or incomplete information, apparently illogical or incon- sistent responses, or difficulty interpreting free-text write-in answers. Where addresses or other location data are collected, these should be examined to ensure that the location can be identified. While this analysis addresses many of the same issues as the survey pre-test, it focuses on the conduct of the survey rather than the design of the questionnaire. Results of the pilot test analy- sis can be used to identify issues requiring particular attention in training field staff for the data collection phase. 4.7.3 Scheduling of Pre-Tests and Pilot Tests Pre-tests and pilot tests should be performed far enough in advance of survey data collection that there is time to adequately analyze the results and resolve any issues, as well as perform any additional testing required. This process is likely to require several weeks to do well. Any changes to the questionnaire have to be finalized before the forms are printed for the data collection phase