Following are examples of NAOMS survey questions with the problem discussed above. Question ER5 in the AC questionnaire reads:
How many times during the last (TIME PERIOD) did an inflight aircraft on which you were a crewmember experience smoke, fire, or fumes that originated in any of the following areas:
the engine or nacelle?
the flight deck?
the cargo hold?
elsewhere in the passenger compartment?
During the last (TIME PERIOD), how many times did an inflight aircraft on which you were a crewmember experience smoke, fire or fumes that originated other than in the engine or nacelle, flight deck, cargo hold, galley, or passenger compartment?
Where did the smoke, fire or fumes originate? SPECIFY.
This question cannot be answered accurately without analyzing post-flight data because pilots may or may not know or be able to tell the difference between bleed air fumes (oil-based), electrical fumes, or solid object fumes. For example, smoke or fumes detected by the pilot anywhere in the aircraft could have originated in the engine and spread throughout the aircraft as a function of pressurization bleed air extraction. In some situations, it is also possible that the perceived smoke or fumes could have come from outside the aircraft, particularly when the aircraft is on a taxiway adjacent to an active runway awaiting its turn to take off.
Other questions have similar potential problems. One question asks about uncommanded movements of control surfaces, but the pilot would not necessarily know what failure resulted in what appeared to be an uncommanded movement. Another question asks for how many degrees an aircraft rolled in a wake turbulence encounter, but without a post-flight analysis of the flight data recorder, a pilot would not know how much the aircraft had rolled. Another question asks for airspeed deviation during a wind shear event. During such sudden and unexpected encounters, pilots are typically more concerned with recovering the aircraft than with estimating the degree of roll or airspeed deviation. Other questions ask whether the aircraft came within 500 feet of another aircraft. Again, in such unexpected situations, the pilot is typically neither in a position nor trained to make an accurate estimate of the absolute distance. Still another question asks whether hazardous materials were packaged and loaded on the aircraft in compliance with the appropriate regulations, but those are not regulations that a pilot is required or even expected to be familiar with. Appendix F contains nine additional examples of questions that potentially ask pilots for information that they would not typically have access to.
Finding: Some of the questions in the NAOMS survey would not provide accurate and consistent measures of events because they asked about situations in which pilots would not typically have access to the information needed for an accurate response.
The literature on the design of survey questionnaires emphasizes the importance of clear and carefully worded questions to elicit reliable responses. In particular, it is important that respondents answer a question consistently even if the question is asked at different points in the survey and that, ideally, different respondents interpret the same question in the same way. This is especially important in a survey such as NAOMS that is intended to collect information on events that occurred, rather than to collect opinions and perceptions of respondents.
The NAOMS team field-tested the survey at several stages and apparently redesigned the questions to take into account some of the comments that it received. Nevertheless, the committee finds that several questions in the survey contain wording that pilots may have found difficult to interpret precisely and to answer consistently. These include (1) long questions with complex structure that would be difficult to understand in a computer-assisted telephone interview; (2) questions that appear to combine multiple, unrelated events; (3) questions about events