protections can be put in place by regulation so that the respondents can be protected at the same time that the data are serving useful purposes for policy making and research.
In summary, most large-scale surveys evolve over time, and their survey methodologies are refined on the basis of experience before they attain excellence. Often, the changes are gradual, but some surveys have undergone major design changes. Examples include the NCVS as well as the National Assessment of Educational Progress. Another feature of successful government surveys is that they typically have a research team and resources to support the investigation of issues or problems of particular import. They also have a core staff dedicated to the survey’s ongoing improvement and adaptation to change. Many large-scale survey programs also develop an organizational culture that fosters a professional approach to the development of survey methodology and produces staff in both technical and administrative areas that are very knowledgeable about a particular survey, its history, and its key issues.
Finding: Successful large-scale surveys typically require a substantial commitment of time and resources to develop, refine, and improve the survey methodology and to ensure that the survey provides useful and high-quality data.
When NAOMS was proposed, the available sources of data on aviation safety included the following: (1) accident and incident data from the NTSB and the FAA, (2) data from the FAA’s NASA-operated ASRS, (3) FAA’s Near Midair Collision Database, and (4) FAA’s Operational Error Detection Program. As noted in Section 2.1, the NTSB and FAA accident and incident databases include only incidents that meet certain thresholds and so do not include all potentially unsafe occurrences. The ASRS database consists largely of self-reports of incidents by pilots. Though it is large and rich in information, it is not a probability sample, so it is impossible to obtain statistically valid estimates from ASRS data. The same limitation holds for the Near Midair Collision Database. The Operational Error Detection Program data cover only aircraft operating in controlled airspace, so planes flying under visual flight rules, which include a high proportion of general aviation flights, would not be covered.
In recent years, the use of onboard data-acquisition systems to collect aircraft operations data is becoming common. Now, virtually all new commercial airliners and most high-end business jets are equipped with flight data recorders, which provide the basis for FOQA systems that provide detailed information about flight operations. These systems are not affected by the types of measurement errors that are present in surveys of pilots or other personnel. However, as noted in Section 2.1, FOQA data do not provide a complete picture of the entire airspace, as piston-engine and turboprop general aviation aircraft are not typically equipped with these data-collection systems. Even if it were possible to obtain FOQA data for the entire population of aircraft in the U.S. airspace, the resources involved in assembling the data from all the air carriers and in ensuring privacy and confidentiality so that the data could be shared among all the carriers, the government, and the public would be very high. The use of probability sampling techniques can be useful here, as one could collect and analyze a sample of the database that takes into account privacy and confidentiality considerations in order to obtain timely information at reasonable costs.
The Aviation Safety Information Analysis and Sharing System (discussed in Section 2.1) has made progress and shows promise in allowing the access to, analysis of, and integration of multiple large aviation safety data sets. As it continues to develop and as more data sets are added, it will become more comprehensive. However, even then it is unlikely to cover the entire aviation system, particularly general aviation and small commercial carriers operating in remote locations. In addition, as more databases are added, issues of privacy and confidentiality are likely to take on increasing importance before the data can be shared among all parties and with the public.
Sample surveys can be used to provide new or supplemental information about aviation safety, even in the presence of these other data-collection efforts. The scope and usefulness of NAOMS are explored in detail in the next two chapters, but generally speaking, NAOMS was an attempt to capture the experiences of the frontline personnel (pilots, flight attendants, air traffic controllers, and mechanics) regarding flight operations and aviation safety. In the committee’s view, such information could be potentially useful, particularly in those segments of