There was a problem loading page R1.
THE NATIONAL ACADEMIES PRESS
500 Fifth Street, N.W. Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This study is based on work supported by Contract NNH05CC16C between the National Academy of Sciences and the National Aeronautics and Space Administration. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author and do not necessarily reflect the view of the agency that provided support for the project.
International Standard Book Number-13: 978-0-309-14646-3
International Standard Book Number-10: 0-309-14646-1
Cover: Image courtesy of My Brainstorm Center, available at www.mybrainstormcenter.com. Design by Tim Warchocki.
Copies of this report are available free of charge from:
Aeronautics and Space Engineering Board
National Research Council
500 Fifth Street, N.W.
Washington, DC 20001
Additional copies of this report are available from the
National Academies Press,
500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu.
Copyright 2009 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council.
OTHER REPORTS OF THE AERONAUTICS AND SPACE ENGINEERING BOARD AND THE SPACE STUDIES BOARD
America’s Future in Space: Aligning the Civil Space Program with National Needs (Space Studies Board [SSB] with the Aeronautics and Space Engineering Board [ASEB], 2009)
Approaches to Future Space Cooperation and Competition in a Globalizing World: Summary of a Workshop (SSB with ASEB, 2009)
Assessment of Planetary Protection Requirements for Mars Sample Return Missions (SSB, 2009)
Fostering Visions for the Future: A Review of the NASA Institute for Advanced Concepts (ASEB, 2009)
Near-Earth Object Surveys and Hazard Mitigation Strategies: Interim Report (SSB with ASEB, 2009)
A Performance Assessment of NASA’s Heliophysics Program (SSB, 2009)
Radioisotope Power Systems: An Imperative for Maintaining U.S. Leadership in Space Exploration (SSB with ASEB, 2009)
Assessing the Research and Development Plan for the Next Generation Air Transportation System: Summary of a Workshop (ASEB, 2008)
A Constrained Space Exploration Technology Program: A Review of NASA’s Exploration Technology Development Program (ASEB, 2008)
Ensuring the Climate Record from the NPOESS and GOES-R Spacecraft: Elements of a Strategy to Recover Measurement Capabilities Lost in Program Restructuring (SSB, 2008)
Final Report of the Committee for the Review of Proposals to the 2008 Engineering Research and Commercialization Program of the Ohio Third Frontier Program (ASEB, 2008)
Final Report of the Committee to Review Proposals to the 2008 Ohio Research Scholars Program of the State of Ohio (ASEB, 2008)
Launching Science: Science Opportunities Provided by NASA’s Constellation System (SSB with ASEB, 2008)
Managing Space Radiation Risk in the New Era of Space Exploration (ASEB, 2008)
NASA Aeronautics Research: An Assessment (ASEB, 2008)
Opening New Frontiers in Space: Choices for the Next New Frontiers Announcement of Opportunity (SSB, 2008)
Review of NASA’s Exploration Technology Development Program: An Interim Report (ASEB, 2008)
Science Opportunities Enabled by NASA’s Constellation System: Interim Report (SSB with ASEB, 2008)
Severe Space Weather Events—Understanding Societal and Economic Impacts: A Workshop Report (SSB, 2008)
Space Science and the International Traffic in Arms Regulations: Summary of a Workshop (SSB, 2008)
United States Civil Space Policy: Summary of a Workshop (SSB with ASEB, 2008)
Wake Turbulence: An Obstacle to Increased Air Traffic Capacity (ASEB, 2008)
Limited copies of SSB reports are available free of charge from
Space Studies Board
National Research Council
The Keck Center of the National Academies
500 Fifth Street, N.W., Washington, DC 20001
COMMITTEE ON NASA’S NATIONAL AVIATION OPERATIONS MONITORING SERVICE (NAOMS) PROJECT: AN INDEPENDENT ASSESSMENT
VIJAYAN N. NAIR,
University of Michigan, Ann Arbor,
CLINTON V. OSTER, JR.,
Indiana University, Bloomington,
DAVID L. BANKS,
ROBERT M. BELL,
ANTHONY J. BRODERICK, Independent Consultant
National Transportation Safety Board (retired)
International Air Transport Association
IAIN M. JOHNSTONE,
Indiana University, Bloomington
ELIZABETH A. LYALL,
Research Integrations, Inc.
DONALD W. RICHARDSON,
Donrich Research, Inc.
THOMAS B. SHERIDAN,
Massachusetts Institute of Technology (emeritus)
ALFRED T. SPAIN,
JetBlue Airways Corporation (retired)
S. LYNNE STOKES,
Southern Methodist University
PAUL JACKSON, Study Director,
Aeronautics and Space Engineering Board
NEAL GLASSMAN, Senior Program Officer,
Board on Mathematical Sciences and Their Applications
ANDREA REBHOLZ, Program Associate,
Aeronautics and Space Engineering Board (from February 2009)
SARAH CAPOTE, Program Associate,
Aeronautics and Space Engineering Board (through November 2009)
LEWIS GROSWALD, Research Associate,
Space Studies Board
AERONAUTICS AND SPACE ENGINEERING BOARD
RAYMOND S. COLLADAY,
Lockheed Martin Astronautics (retired),
KYLE T. ALFRIEND,
Texas A&M University
AMY L. BUHRIG,
Center for Strategic and International Studies
University of Maryland
JOHN-PAUL B. CLARKE,
Georgia Institute of Technology
RAVI B. DEO,
Northrop Grumman Corporation (retired)
MICA R. ENDSLEY,
Massachusetts Institute of Technology
JOHN B. HAYHURST,
Boeing Company (retired)
Gulfstream Aerospace Corporation
RICHARD KOHRS, Independent Consultant
Air Force Research Laboratory
ELAINE S. ORAN,
Naval Research Laboratory
Case Western Reserve University
United Air Lines (retired)
RICHARD E. ROWBERG, Interim Director (from March 2, 2009)
MARCIA S. SMITH, Director (through March 1, 2009)
COMMITTEE ON APPLIED AND THEORETICAL STATISTICS
Jet Propulsion Laboratory
MICHAEL F. GOODCHILD,
University of California, Santa Barbara
University of Wisconsin-Madison
University of Chicago
SCOTT T. WEIDMAN, Director
NEAL GLASSMAN, Senior Program Officer
On October 22, 2007, a news article from the Associated Press (AP) brought a relatively obscure NASA project, the National Aviation Operations Monitoring Service (NAOMS)—a survey administered to pilots from April 2001 through December 2004—to the attention of the public as well as Congress.1 The article revealed that a letter from NASA had indicated that a Freedom of Information Act (FOIA) request by the AP for information related to the NAOMS project was being turned down. Specifically, the letter stated: “Release of the requested data, which are sensitive and safety-related, could materially affect the public confidence in, and the commercial welfare of, the air carriers and general aviation companies whose pilots participated in the survey.”2
Citing an unnamed source familiar with the results of the survey, the AP article also reported that the survey data showed that “the pilots reported at least twice as many bird strikes, near mid-air collisions, and runway incursions as other government monitoring systems show.”3
A few days before the AP news story broke, the Subcommittee on Investigations and Oversight of the U.S. House of Representatives’ Committee on Science and Technology sent a letter to NASA Administrator Michael Griffin requesting that a variety of materials pertaining to the NAOMS project be turned over “to help the subcommittee understand more clearly what information NASA collected in the three years that it surveyed pilots in the NAOMS project.”4
The Committee on Science and Technology held a hearing on NAOMS on October 31, 2007, at which Dr. Griffin expressed disagreement with the phrasing of the FOIA denial letter and firmly stated that NASA was not putting “commercial interests ahead of public safety.”5 According to Dr. Griffin, the FOIA request was denied because “the data likely contained confidential commercial information.”6 He further indicated that NASA would be releasing all NAOMS data that did not contain either confidential commercial information or information that
would compromise pilot identity. Redacted versions of the data were subsequently released to the public, with the first release occurring in December 2007.
Dr. Griffin also pointed out:
None of the research conducted in the NAOMS project, including the survey methodology, has been peer-reviewed to date. Accordingly, any product of the NAOMS project, including the survey methodology, the data, and any analysis of that data, should not be viewed or considered at this stage as having been validated.7
The above statement speaks to the central purpose of this assessment. In fact, just prior to the congressional committee hearing on October 31, 2007, NASA had asked the National Research Council (NRC) of the National Academies to review the NAOMS survey methodology.
To conduct the review requested by NASA, the NRC established the Committee on NASA’s National Aviation Operations Monitoring Service (NAOMS) Project: An Independent Assessment, composed of experts in the fields of aviation operations (including pilots), aviation safety, survey methodology, and statistics. Biographical information on the committee members is provided in Appendix A of this report.
The committee was asked to assess the NAOMS survey methodology and to analyze the publicly available survey data to determine their potential utility. Additionally, NASA requested that the committee provide recommendations on the most effective ways to use the NAOMS data. The committee’s full statement of task is as follows:
The Aeronautics and Space Engineering Board (ASEB), in conjunction with the Committee on Applied and Theoretical Statistics (CATS), will create an ad hoc study committee to make an independent assessment of NASA’s National Aviation Operations Monitoring Service (NAOMS) project. The NAOMS project used a survey methodology to anonymously collect data from commercial and general aviation pilots over several years regarding aviation safety-related events. The NAOMS project contracted with Battelle Memorial Institute to design the survey and collect the data. The study committee will assess the NAOMS survey methodology, and, to the extent possible, analyze the survey data. This assessment will be based upon information in the public domain including the following items that will be provided to the committee by NASA: (1) a final report provided by the prime contractor, Battelle, that was released to the public on December 31, 2007; (2) a November 13, 1998, NAOMS briefing to the NASA Aviation Safety Reporting System Advisory Subcommittee; and (3) the redacted set of survey responses that were released to the public on December 31, 2007. The study committee will also provide recommendations on the most effective ways to use the NAOMS data.
Specifically, as part of the assessment, the study committee shall:
Assess the process used by the contractor and described in the contractor report to determine how to acquire a statistically meaningful data set representative of a variety of factors that may affect (or impact) the safety of the national airspace system and that would enable one to track how these factors change over time.
Assess the advantages and disadvantages of using a survey method to collect such a statistically meaningful data set.
Assess the survey methodology used by the contractor and described in the contractor report to include:
An analysis of specific details of the survey methodology such as the recall period, collection approach, sampling approach, questionnaire design and the use of non-aviation experts as interviewers.
An analysis of method or methods used to validate the survey methodology.
An identification of the various sources of error (both random and systematic) due to the survey methodology, along with estimates of the magnitudes of those errors, including an analysis of the adequacy of the sample size.
Recommendations of how one might estimate appropriate error bars for the survey results.
Recommendations regarding any methods that might enable one to correct for errors introduced by the methodology.
Conduct an analysis of the project survey data provided by NASA to determine its potential utility. (Note: The survey data will be a redacted data set also released to the public. This data set will be redacted in a manner that
preserves the anonymity of the pilot respondents. Details of the redaction process will be provided.) This analysis may include an assessment of the data’s validity using other known sources of information.
Provide recommendations on the most effective ways to use the NAOMS data. Such recommendations can include the possibility of using the data in combination with other safety data.
Between June 2008 and March 2009, the committee held five meetings to receive briefings, review materials, and write this report. This schedule included a meeting at the NASA Ames Research Center, where a subcommittee met with some of the NASA researchers connected with the NAOMS project. During the course of the study, the committee received briefings from representatives of a variety of organizations on issues related to the NAOMS project. Appendix B lists these presenters.
This report presents the committee’s analyses and findings. Following the Summary, Chapter 1 provides background on NAOMS and a general description of the features of the NAOMS survey. Chapter 2 discusses how aviation safety is measured, what other sources of aviation safety data existed when NAOMS was developed, and what data exist today. Chapter 3 describes how sample surveys are conducted, examines some of the major surveys used in the government sector, and assesses the advantages and disadvantages of using a survey in the field of aviation safety. Chapter 4 assesses the NAOMS sampling design; it examines the impact of the specific features of the sample design and the impact of potential coverage biases on the accuracy of the estimates that are based on these data. Chapter 5 examines the structure of the questionnaire used in the project as well as the content and wording of the specific questions. Chapter 6 explores the limitations imposed by the redacted data set. Chapter 7 summarizes the committee’s analysis of the publicly available data set and provides its recommendation on the most effective ways to use the NAOMS data. In addition to Appendixes A and B mentioned above, Appendix C lists select acronyms used in the report, Appendix D discusses the principal segments of aviation in the United States, Appendix E gives additional examples of surveys by federal agencies, Appendix F lists the additional survey questions with problems discussed in Chapter 5, Appendix G reprints the full air carrier questionnaire, and Appendix H reprints the full general aviation questionnaire.
Acknowledgment of Reviewers
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the Report Review Committee of the National Research Council (NRC). The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report:
Peter J. Bickel, University of California, Berkeley,
R. John Hansman, Massachusetts Institute of Technology,
Raymond LaFrey, Lincoln Laboratory (retired),
Daniel R. Masys, Vanderbilt University School of Medicine,
Edmond Soliday, United Airlines (retired),
Roger Tourangeau, University of Maryland, and
Kirk Wolter, National Opinion Research Center.
Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its release. The review of this report was overseen by Lawrence D. Brown, University of Pennsylvania, and Adib Kanafani, University of California, Berkeley. Appointed by the NRC, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.