The committee did not undertake a review of the methodology used by the NAOMS team to determine the recall period because it did not have access to data from these studies. Therefore, the committee cannot comment authoritatively on the choice of a 60-day recall period versus a recall period of a different length. However, analysis of the redacted survey data (discussed in Chapter 7) indicated several problems: (1) considerable rounding effects in reported numbers of hours and flight legs flown (see Figure 7.1 in Chapter 7) and (2) a high fraction of anomalous data for both Section A (number of hours/legs flown) and Section B (event counts) of the survey questionnaires. It is possible that these problems are not related to the recall period. Nonetheless, it is surprising that Battelle’s final report notes the existence of a “downward bias” without a systematic investigation of the size of the bias, how it varies across the different types of events, and so on.22 This information is critical to the validity of the survey results, especially given the rare nature of some of the events being surveyed. Finally, the committee agrees with the Battelle report that the effect of the bias would be smaller on trends than on actual rates, provided that the nature of the bias remains constant over time—hence the need to investigate the magnitude and nature of the potential biases.
The NAOMS team considered three different ways to conduct the survey: in-person interviews, self-administered questionnaires, and computer-assisted telephone interviews. Each type was weighed against several criteria, including cost, respondent satisfaction, response rate, and quality of the data.
The NAOMS team tested the different methods of conducting the survey in a field trial. Early in the field trial, the NAOMS team determined that in-person interviewing required too much time and cost, and it was dropped from consideration. The results of the trial also demonstrated clear differences between the CATI system and the self-administered questionnaires. While the CATI system took longer to complete and cost more, it had a higher response rate and fewer incomplete responses. In the end, the team opted to use the CATI system for the full implementation of the survey.23
The NAOMS team decided to use professionally trained interviewers rather than aviation-safety professionals to conduct the interview. There are advantages and disadvantages to both options. The advantage of using aviation-safety professionals lies in their ability to clarify the intent of the questions or to ask the respondents to verify answers that seem implausible. However, there is the possibility of interviewer bias, by which the interviewer may lead the respondent in the direction of expected responses, and it would be difficult to quantify the nature of the assistance and clarifications provided by the interviewer and whether the interviews led to reproducible answers. For this reason, most surveys use professionally trained interviewers who have no subject-matter knowledge in the area of investigation. On balance, if the survey instrument is well defined and the possible questions and ambiguities are anticipated and addressed, the use of professionally trained interviewers will lead to more statistically reliable results.
Finding: The decision by the NAOMS team to use professionally trained interviewers was reasonable. The use of the CATI method for the survey was also appropriate.