National Academies Press: OpenBook
« Previous: 3 Assessment of the Airspace Systems Program
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 71
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 72
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 73
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 74
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 75
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 76
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 77
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 78
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 79
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 80
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 81
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 82
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 83
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 84
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 85
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 86
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 87
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 88
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 89
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 90
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 91
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 92
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 93
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 94
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 95
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 96
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 97
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 98
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 99
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 100
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 101
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 102
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 103
Suggested Citation:"4 Assessment of the Aviation Safety Program." National Research Council. 2004. Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs. Washington, DC: The National Academies Press. doi: 10.17226/10861.
×
Page 104

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

l - 4 Assessment of the Aviation Safety Program BACKGROUND Program Information The Aviation Safety Program (AvSP) is one of three programs in the Aeronautics Technology Pro- grams of NASA's Aerospace Technology Enterprise. AvSP was created in 2000 as an outcome of a formal process initiated by NASA to develop a research in- vestment strategy in the area of aviation safety. The goal of the AvSP is to protect air travelers and the public. Its research and development strategy is to increase safety by three primary methods: Aviation system modeling. Identify and correct problems using aviation system-level data, Accident prevention. Identify interventions and develop technologies to eliminate recurring types of accidents, and · Accident mtigation. Reduce injury and decrease fatalities in survivable accidents. These methods are applied in the three major re- search and development components: iG. Finelli, NASA Langley, "NASA Aviation Safety Program Overview," presentation to panel, February 2003. 71 Vehicle Safety Technology, which includes Single Aircraft Accident Prevention, Accident Mitigation, and Synthetic Vision Systems, Weather Safety Technology, which includes Aircraft Icing and Weather Accident Preven- tion, and System Safety Technology, which includes Systemwide Accident Prevention, Search and Rescue,2 and Aviation System Monitoring and Modeling. A fourth research component, security research, will be added in FY04. The committee did not evaluate this component since no research and development work is currently under way. The AvSP also has an effort in Technical Integration, which is separate from the three research projects. Research and development for AvSP is performed at NASA Langley Research Center, NASA Glenn Re- search Center, NASA Ames Research Center, and NASA Dryden Flight Research Center, with the pro- gram headquarters at Langley. A program organization chart is shown in Figure 4-1. AvSP was funded at $156.2 million in FY03 under 2Search and Rescue is funded through AvSP but is implemented through the Office of Space Flight. Since all programmatic devel- opment and all technical research are performed under the Office of Space Flight, the Aviation Safety Panel did not review this work.

72 AN ASSESSMENT OF NASA 'S AERONAUTICS TECHNOLOGY PROGRAMS I _ l Technical Integration ~ Aviation Safety Program ~ · Effort // \~ System Safety Technology Project A: Vehicle Safety Technology Project Synthetic Vision Systems 1 ~ Commercial and Business Aircraft Single Accident Aircraft Mitigation Accident Prevention I ~ r Vet ~ Prevention Flight Critical System Design Propulsion System Safety Technologies Control Upset Prevention and Recover General Aviation Enabling Technologies , _ Weather Safety | Technology Project Aircraft Weather Icing Accident Prevention Design and Analysis Tools Aircraft Ice Protection Education and Training Aviation Weather Information Weather Information Comm. Turbulence Prediction and Warning Systems FIGURE 4-1 Aviation Safety Program organization chart. the full-cost accounting scheme.3 Vehicle Safety Tech- nology accounted for $83.9 million (54 percent of the AvSP total), Weather Safety Technology accounted for $31.6 million (20 percent of the total), and System Safety accounted for $40.7 million (26 percent of the total). NASA is in the process of transitioning to full- cost accounting from a net accounting scheme; previ- ously, NASA managers assessed their budgets by the amount of funding available to them for contracts, grants, and other types of procurements. Uncler the net accounting scheme, Vehicle Safety Technology is bud- geted at $19.8 million, Weather Safety Technology at $14.7 million, and System Safety Technology at $18.4 million. In this report, specific subprojects and tasks are discussed in net dollars only, as this was the only information provided to the committee. The net budget breakdowns by subproject are shown in Table 4-1. 3Full-cost accounting encompasses all costs, including research and program management; institutional infrastructure costs, such as research operations support; direct procurements; direct civil service workforce, benefits, arid Gavel; service pools; center gen- eral and adrninis~ative (G&A); and corporate G&A. _ Systemwide Aviation Accident System Prevention Monitoring and Modeling 1 1 ,- 1 Human Performance Models Maintenance Human Factors Crew Training Program Human Factors Data Analysis Tool Development Extramural Monitoring Modeling and Simulations Intramural Monitoring Like other NASA programs, each AvSP project has a 5-year lifespan. This does not imply that the program ceases to exist after 5 years, however. Project plans are reevaluated after each 5-year time period to phase in new projects that build upon previous research and de- velopment. Review Process The Aviation Safety Panel was formed in Decem- ber 2002 as one of three panels that would review NASA's Aeronautics Technology Program. The Avia- tion Safety Panel met for the first time on February 26- 28, 2003, in Washington, D.C. At this first meeting, the 10-person panel received technical briefings from the program and project managers in AvSP on the over- al1 program, specific projects, and individual tasks. After the first meeting, panel members participated in site visits to each of the relevant NASA facilities (NASA Langley, NASA Glenn, and NASA Ames). The purpose of the site visits was to obtain a deeper understanding of the research and development in the program, to speak directly with the principal investiga- tors for each project task, and to observe the products

ASSESSMENT OF THE AVIATION SAFETY PROGRAM TABLE 4-1 Net Budget for the Aviation Safety Program Budget (million $) ProjeetlSubproject Name FY03 FY04 Vehicle Safety Technology Single Aircraft Accident Prevention Accident Mitigation Synthetic Vision Systems Weather Safety Technology Aircraft Icing Weather Accident Prevention System Safety Teehnologya System-Wide Accident Prevention Aviation System Monitoring and Modeling 19.8 4.7 8.4 9.8 0.4 2.2 7.2 5.0 9.7 s.o 8.4 13.9 17.7 0.2 2.6 7.0 s.o 8.9 5.1 8.6 aSystem Safety includes Search and Rescue, which is not reviewed here. SOURCE: Information provided to the NRC panel by G. Bond, Aviation Safety Program Office, NASA Langley Research Center. and facilities firsthand. The site visits are listed in Ap- pendix C. Panel members visited on-site or spoke via teleconference with NASA personnel from every AvSP task. Panel members, who were experts in their fields, also reviewed technical reports and journal articles and followed up with individual principal investigators by means of teleconference calls and written questions. Before the first meeting, the NRC asked each prin- cipal investigator to complete a short questionnaire with 12 questions relating to the research and develop- ment goals, products, roadblocks, users, and technical outcomes. A blank questionnaire is shown in Appen- dix D. The completed questionnaires were distributed to the panel for review prior to the first panel meeting. Thus, the panel members were already somewhat fa- miliar with the programs and projects under review before they were briefed in person by the NASA re- searchers and program managers. The questionnaires proved to be a valuable tool for the panel in performing its program assessment. Upon completion of the three site visits, the panel met for a second time, again in Washington, D.C., on May 27-29, 2003, to come to consensus on findings and recommendations for the program. The panel dis- 73 cussed outstanding questions and issues of concern with program staff from NASA. It also developed crosscutting observations across the different projects and tasks within AvSP. The panel then provided its input to the Aeronautics Technology Programs parent committee in the form of working documents. Four of the ten panel members represented the panel on the committee. PORTFOLIO The committee evaluated the appropriateness of the AvSP research portfolio based on the amount of basic research versus user-driven research; the presence of gaps or incomplete areas of research; the balance be- tween high-risk, high-payoff research and more evolu- tionary work; and whether or not the portfolio ad- dresses real-world problems. The committee is concerned about the balance be- tween fundamental and product-driven research in the Aviation Safety Program. It observed a shift away from essential basic research over recent years. Such basic research is necessary for the development of future safety products that will enable the AvSP to reduce

74 ': 1 AN ASSESSMENT OF NASA 'S. AERONA UTlCS TECHNOLOGY PROGRAMS accident rates. In some instances, the committee ob- served ineffective work-arounds created out of neces- sity to divert resources from funded, low-payback projects to accomplish unfunded but critical basic re- search. Furthermore, with a few notable exceptions (such as the Aircraft Icing subproject and the Modeling and Simulations task in the Aviation System Monitor- ing and Modeling subproject), the committee felt that this problem was widespread within the program. The committee found examples of research that is essentially complete and ready for transition (such as Fault Tolerant Modular Architectures-, Personal Elec- tronic Device electromagnetic susceptibility, virtual and augmented reality for maintenance crews, and the Performance Data Analysis and Reporting System). The committee also found places where basic research was lacking for example, high-temperature materi- als for engines, weather display interfaces, turbulence warning systems, and human factors work in many ar- eas. The committee's findings and recommendations regarding specific instances where research is too prod- uct-driven or where additional basic research is needed are presented in the discussion of each task. Finding: Support for Basic Research. There has been a shift away from essential basic research in the Aviation Safety Program in recent years. Program Recommendation: Support for Basic Re- search. The Aviation Safety Program should rein- state a core competency program dedicated to basic research that is essentially unencumbered by short- term, highly specified goals. Without a strong basic research program, the more applied research even- tually suffers from a lack of good ideas and trained personnel. The criterion for starting or restarting such an activity within a Center is that a need must exist for knowledge that is not now available. The committee noted specific gaps in the portfolio at the subproject and task levels in subsequent sections. It found one significant program-wide omission in the research portfolio: rotorcraft. Finding: Rotorcraft. Rotorcraft safety can be im- proved with additional research in the areas of de- cision aids, synthetic vision, training, workload, and situational awareness. Program Recommendation: Rotorcraft. The Avia- tion Safety Program should reincorporate rotor- craft research into its program. The research should consider the most effective approaches for reducing the workload of rotorcraft pilots and improving their ability to conduct safe, low-speed, low-altitude rotorcraft operations in obstacle-rich environments and in adverse weather. PROGRAM PLAN The AvSP program plan emerged from a series of strategic planning sessions on aviation safety in 1997 known as the Aviation Safety Investment Strategy Team (ASIST). ASIST established a vision and priori- tized the research and development investment areas for the AvSP. The AvSP approach includes system modeling, accident mitigation, and accident prevention, with an emphasis on mitigating problems that contrib- ute most heavily to accident and fatality rates. The AvSP was established in 2000 with a 5-year program plan. Each individual task within the AvSP is structured to last 5 years. This 5-year programming cycle is more suitable for a product-oriented program. It is difficult, if not impossible, for NASA to maintain core compe- tencies with these 5-year programs. In addition, there do not appear to be sufficient off-ramps to transition research that has been completed before the 5-year time window closes. Finding: Use of Sunset Requirements. NASA func- tions on a 5-year schedule to the detriment of solid research. Program Recommendation: Use of Sunset Require- ments. The Aviation Safety Program should struc- ture its program based on the natural duration of each research effort and not compel conformity to a 5-year cycle for every task. NASA should eliminate arbitrary time constraints on program completion and schedule key milestones based on technology maturity, task complexity, and resource limitations. Research involving the human-machine interaction and causes of human error should be a major focus of any aviation safety research program. The AvSP con- tains a wide array of human factors research, from syn- thetic vision displays to aviation weather information requirements to tools for aircraft maintenance teams. In general, the committee found evidence of high qual-

ASSESSMENT OF THE AVIATION SAFETY PROGRAM , .j ity in all of NASA's human factors research; however, it also found that the work was not always well inte- grated into a cohesive program without overlaps. The committee approves of the efforts of the Aviation Safety Program Office in pulling together some of the disparate human factors tasks through cross-center meetings and through the Program Human Factors task of the Systemwide Accident Prevention subproject. However, the committee did not observe any improve- . . . . ment In the ~ntertask communication or any synergy from the human factors research within the program. Aviation accident data make clear that human er- ror is a much greater factor than hardware or software failure or environmental conditions. Ideally, every technology effort should be examined from a human factors perspective at an early design phase to antici- pate problems. However, the advice of human factors professionals, who must necessarily draw on the softer behavioral sciences, is often disregarded by the engi- neering designers, who view it as negative or irrelevant. NASA has traditionally supported research in human factors, and the human factors group at NASA Ames has truly been a national resource. Finding: Human Factors Research. In recent years the Aviation Safety Program's work in human fac- tors has been eroding; senior in-house research staff have left, and in order to get the work done, more human factors professionals have found themselves managing contractors, a task for which they often are not well qualified. Crosscutting efforts to inte- grate human factors have also suffered. Program Recommendation: Human Factors Re- search. Critical human factors expertise should be better supported in order to maintain critical mass, to foster basic research in this field, to identify gaps in our understanding of safety, and to be available to consult with various NASA projects. Program Recommendation: Early Analysis of Hu- man Factors. Project requirements should include requirements for human factors analysis early in the design phase. The committee found that the considerable layers of both line management and project management ob- scure the lines of accountability in AvSP. In at least one case, a person's subordinate in the research project hierarchy is his or her superior in the line staff hierar- 75 _ chy. The committee felt that subproject- and task-level plans, goals, metrics, and responsibility could not be clearly traced back to an overarching plan and vision for the AvSP. In other words, the planning appeared to be more bottom-up than top-down. Additionally, the committee heard from a number of technical civil ser- vants in the program that too much of their time was spent "doing management" (e.g., making PowerPoint slides) and not enough doing science and technology. In addition, it was not clear to the committee what methods and metrics NASA uses to evaluate objec- tively the status of its research projects against its own stated goals. The program effort in Technical Integra- tion (described in a subsequent section) would be a natural place for such an evaluation. Finding: Management Structure. The organiza- tional structure is unnecessarily complex, making it difficult to trace lines of responsibility. Subproject- and task-level plans, goals, metrics, and responsi- bility could not be clearly traced back to an overarching plan and vision for the Aviation Safety Program. Program Recommendation: Management Struc- ture. NASA should articulate a clear, long-range plan for the Aviation Safety Program and a hierar- chy of goals, and it should adopt a less complex management system that enables program account- ability and implementation to be clearly traced. The committee suggests that NASA reexamine its names for the AvSP activities (many terms sound like they overlap or are ambiguous) so that the goals of each major project are easily understood. This ambiguity is particularly evident in the Single Aircraft Accident Pre- . . venhon su project. TECHNICAL PERFORMANCE The committee asked a variety of questions to as- sess the technical quality of the work, to evaluate the facilities and personnel, to find evidence of system- level assessments, and to determine the balance be- tween experimental and theoretical work. The commit- tee also compared the quality of the AvSP work relative to that of other work in industry, academia, and gov- ernment, including international work. The committee found the technical quality of the AvSP to be very good. In some cases, particularly in

76 specific parts of the weather work, NASA personnel can be considered among the world leaders in their re- spective fields. The review committee found the facili- ties to be adequate for achieving the research goals; in some cases (such as the icing wind tunnel), the facili- ties can be considered true national assets. The committee identified several specific tasks and subtasks that have achieved an outstanding level of technical achievement: . i AN ASSESSMENT OF NASA 'S AERONA UTICS TECHNOLOGY PROGRAMS by NASA that are similar to or have considerable over- lap with products developed by industry. (Specific ex- amples will be discussed in the task-specific sections.) Finding: Redundancy with Industry. Some prod- ucts being developed by NASA are similar to or have considerable overlap with products already developed by industry. Program Recommendation: Benchmarking Against Industry. The Aviation Safety Program should com- pare (benchmark) its research projects against those of other research and development entities in government and industry to ensure that NASA's work is leading. If it is not, NASA should terminate the work. . Structures health management subtask of the Vehicle Health Management and Flight Criti- cal System Design task in the Single Aircraft Accident Prevention subproject, Mode confusion subtask of the Vehicle Health Management and Flight Critical System Design task in the Single Aircraft Accident Prevention subproject, Scale model development and testing work in the Single Aircraft Accident Prevention sub- proJect, · Design and Analysis Tools task in the Aircraft Icing subproject, Aircraft Ice Protection task in the Aircraft Icing subproject, and · Crew Training task in the Systemwide Acci- dent Prevention subproject. A number of outstanding products have been de- veloped, but many of these (an example being the Per- formance Data Analysis and Reporting System (PDARS) trajectory monitoring tool) are ready for handoff to industry. Much of the low-TRL research is excellent, but its relevance and potential usefulness seem not to have been made clear to potential users (a good example is human performance models). USER CONNECTIONS User connectivity was evaluated in two separate ways. First, the committee asked how well NASA per- sonnel reflect and leverage work being conducted else- where and how well NASA research results are ac- cepted and adopted by the outside community. Second, the committee asked how the research itself is con- ducted --- for example, if it uses an appropriate mix of internal and external personnel. In comparing the work of the Aviation Safety Pro- gram with other work in the community, the committee found several instances of products being developed Exploring the second aspect of user connectivity (how well the program uses expertise from the outside community), the committee found that the answer var- ied from task to task. In some cases (particularly the Vehicle Safety Technology project), the committee felt the project would benefit from additional involvement with the outside community. In particular, the commit- tee believes NASA's fundamental research projects would benefit from increased university participation. In other cases (especially in the System Safety Tech- nology project), the committee felt there were too few in-house personnel and that too much of the research was being conducted by contractors. This tends to weaken the core competencies of NASA. ASSESSMENT BY PROJECT Technical Integration Pro jest The AvSP has an effort in Technical Integration, which is designed to provide program assessments, develop systems-level implementation strategies, and integrate research and development efforts across pro- gram tasks, particularly in flight testing. The committee believes the concept behind the Technical Integration project is very important, provided it plays a significant role in deciding what research to undertake and when such research should be modified, transitioned to industry, or discontinued. The commit- tee understands that because the Technical Integration project began after the current 5-year plan had begun, it has been playing catch-up with regard to its status in the overall program. However, the committee had difficulty

ASSESSMENT OF THE A VIA TlON SAFETY PROGRAM , . ~ determining the effect of the Technical Integration ac- tivities on current planning. The Technical Integration project seemed to be running almost as an independent activity somewhat disconnected from project manage- ment. The committee also observed that subjective evaluations are being made, mostly by NASA project managers, and it believes that NASA should have more input from customers and industry and from lower-level managers, scientists, and engineers engaged directly in the various efforts. For example, the market penetration of AvSP products should be studied. The Technical Integration effort as currently con- stituted seems best suited for evaluating AvSP's near- term products. However, the committee is concerned about how Technical Integration will integrate project "stovepipes" into a workable whole. For example, there appears to be little integration of NASA Ames human factors activities with the synthetic vision work at NASA Langley. The committee also sees a need for anticipatory or prospective integration of the Human Performance Models task, the Monitoring and Simula- tion task, and the other monitoring tools efforts. Finding: Use of the Technical Integration Project. The Technical Integration effort does not play the role it needs to play in deciding what research to undertake, in performing cost-benefit analyses for projected and ongoing projects, and in deciding when such research should be modified, trans- itioned to industry, or discontinued. Recommendation: Use of Systems Engineering. NASA project managers should employ systems engineering approaches to ensure proper integra- tion of projects. Recommendation: Use of a Quality Assurance Pro- gram. NASA should institute a quality assurance activity, separate and independent from project management, the results of which should be re- ported directly to the Aviation Safety Program manager and to the Aerospace Technology Enter- prise associate administrator. As discussed in the assessment of the Airspace System Program, above, there appear to be significant overlaps in the various system modeling efforts within NASA, and it may be feasible to consolidate or inte- grate some projects. In particular, modeling research by the AvSP should be coordinated with Virtual Air- 77 space Simulation Technologies, which is part of ASP. NASA should also develop and implement a plan for evolving current models, simulations, and analysis tools into large-scale models. The committee applauds the Technical Integration support of the Commercial Aviation Safety Team and the Joint Implementation Measurement Data Analysis team. Vehicle Safety Technology Pro tech Background The Vehicle Safety Technology project is designed to strengthen aircraft against vehicle system and com- ponent failures, loss of control, loss of situational awareness, and postcrash and in-flight fires. The project focuses on applications for the aircraft itself. The majority of the research is conducted at NASA Langley, with a relatively small amount of work in pro- pulsion safety and fire prevention conducted at NASA Glenn. The Vehicle Safety Technology project was funded at $83.9 million (full-cost)/$19.8 million (net) in FY03 and is divided into three subprojects: Single Aircraft Accident Prevention, Accident Mitigation, and Synthetic Vision Systems. In net dollars, Single Air- craft Accident Prevention is funded at $10.4 million, Accident Mitigation at $2.2 million, and Synthetic Vi- sion at $7.2 million. Portfo/io The goals of the project are focused on the vehicle itself, in applications related to the flight deck, flight critical systems, propulsion, and airframe. The research focuses on loss-of-control prevention and recovery; flight critical systems; vehicle health monitoring; pro- pulsion systems safety; fire mitigation, detection, and prevention; and improving low-visibility conditions by providing a synthetic picture of the outside world. This is an ambitious project with many diverse goals, applications, and areas of research expertise. The folding of such diversity into a single project and the integration of the results of each research effort present a considerable challenge. As with all AvSP programs, the projects within the Vehicle Safety Technology project have a 5-year life span. The termination point for the Vehicle Safety Technology Project tasks is scheduled to be 2005, although many of the projects will probably be continued in some form into the next phase of the AvSP.

78 The committee found the researchers stretched in many directions in the Vehicle Safety Project and believed it was unlikely that every subtask could achieve its stated goals to an appropriate level of detail by the project ter- mination point in 2005. Further, the fact that the names of many research tasks seemed to be similar suggested that some tasks could be combined. Overall, the committee believes there is an appro- priate balance of low-TRL work with more applica- tion-driven research in the Vehicle Safety Technology project. Across the AvSP as a whole, the committee has some concerns about the increasing trend toward product-driven research and development, so it was pleased to see several fundamental, low-TRL tasks within Vehicle Safety Technology, such as the design work in the Control Upset Prevention and Recovery (CUPR) task. The committee urges a continued em- phasis on this basic research in the next phase of the Aviation Safety Program. At the same time, the com- mittee notes that several tasks for example, some of the fire prevention work and fault-tolerant integrated modular architectures have already attained a high level of technology readiness and should be transitioned to industry. The committee is sensitive to the fact that by fo- cusing on fewer concepts, the project eliminates other worthy research ideas. However, despite recommend- ing that the Vehicle Safety Technology project focus on fewer tasks in greater detail, the committee also found a significant omission in the array of activities in this project namely, rotorcraft. The committee be- lieves that NASA could have a significant impact on rotorcraft safety by including the topic in this project. The committee believes that NASA's decision to ter- minate rotorcraft work is a mistake, as there are a num- ber of real-world problems in rotorcraft safety that ap- parently are not being addressed outside NASA. Program Plan The committee believes that NASA will make sig- nificant impacts if it can mature the technologies in the Vehicle Safety Technology project. However, the com- mittee judges the program plan for technology matura- tion to be overoptimistic. Finding: Portfolio Breadth. Despite the encourag- ing progress reported. to date, the time remaining is insufficient to achieve the goals set forth in the pro- gram plan. The breadth of the work in Vehicle AN ASSESSMENT OF NASA 'S. AERONA UTlCS TECHNOLOGY PROGRAMS Safety Technology is coming at the expense of tech- nical depth. Recommendation: Portfolio Breadth. The Aviation Safety Program should narrow the scope of activi- ties in the Vehicle Safety Technology project to in- crease the depth of research activities and focus them in fewer, more specific, higher-priority areas. A few tasks within the Vehicle Safety Project have already reached a high TRL, and the committee noted that there were no appropriate off-ramps or transitions for those tasks that have reached or will reach comple- tion before the 2005 project end date. Specific instances are noted in the commentary on the individual tasks, below. Technica/ Performance The committee found the individual researchers to be bright, aware of the relevant literature, and able to answer both theoretical and application-related ques- tions. The facilities are state of the art and appropriate for carrying out the project. The committee found evidence of several notewor- thy research tasks within this project that have a high level of technical achievement, such as the structures health management subtask. Several other tasks per- haps should be transitioned because they have already completed their research objectives, such as fault-tol- erant integrated modular architectures and some of the fire mitigation work. In no case did the committee rec- ommend research termination for lack of quality. The committee is concerned about the functional integration of the many diverse activities talking place across the different NASA research centers. NASA should develop software ant! hardware interface specifi- cations that connect the various subsystems early on to aid in the integration and definition of the scope and plans for program research. These specifications can be spiraled into more detail and refined accordingly as the program evaluations progress. They form the basis for integrating the work taking place between the NASA centers and NASA contractors. These interfaces should include interactions between all the vehicle subsystems, including the controls arid display tasks. Finding: Interim Integration Milestones. There ap- pears to be a lack of interim task-level milestones to track the progress of integration activities.

ASSESSMENT OF THE AVIATION SAFETY PROGRAM Recommenclation: Interim Integration Milestones. NASA should integrate the information that sys- tems evolving from individual tasks such as Vehicle Health Management and Flight Critical System Design and Control Upset Prevention and Recovery can provide to the flight-deck crew. Recommendation: Interim Integration Milestones. NASA should develop software and hardware in- terface specification documents that address the various subsystems early on to aid in the integra- tion and definition of the scope and plans for pro- gram research. Recommendation: Interim Integration Milestones. NASA should incorporate interim test and evalua- tion milestones for its flight simulation facilities to measure the impact of its design integration on on- · ~ · ·. ~ going crew performance act~v~es. User Connections In general, the committee found that the research- ers are collaborating with the appropriate outside agents; by and large, there is the right degree of in- volvement with industry, and the connectivity to the research community is impressive. In a few cases, es- pecially in areas with low-TRL work, the NASA re- search could be augmented with university research. Specific instances are noted below. It appears that uni- versity involvement is relatively minor, notably in for- mal methods of software verification and validation and in some of the propulsion safety technologies. Assessment by Subyroject Single Aircraft Accident Prevention Subproject The Single Aircraft Accident Prevention (SAAP) subproject is designed specifically to develop and implement technologies that enhance aircraft airwor- thiness and resiliency against loss of control while in flight. Again, the work focuses on onboard technolo- gies for the individual vehicle. The subproject contains three tasks: Vehicle Health Management and Flight Critical System Design (VHM and FCSD), Propulsion System Safety Technologies, and Control Upset Pre- vention and Recovery (CUPR). The net budget for SAAP is $10.4 million in FY03, with $4.8 million for 79 VHM and FCSD, $4.1 million for Propulsion Safety Technologies, and $1.5 million for CUPR. NASA's effort to expand and improve industry knowledge of the aerodynamic performance envelopes of transport-category aircraft appears to be on target for reducing the incidence of loss-of-control accidents. This subproject promises to improve the fidelity of flight simulators used as tools for improving pilot performance in manual recovery from extreme attitudes. The research could lead to better avoidance of such conditions as well as to systems that effect automatic recovery. By their nature, many of the modeling and analysis efforts do not have a well-defined end point, and there is always room for improvement. The lack of a clear completion point for some of the SAAP work was nev- ertheless troubling, and the committee believes that NASA should develop ways to "declare victory" and make clear the degree to which the effort has succeeded and the amount of research still needed to achieve suc- cess. For example, the modeling of follower aircraft interaction with wake vortices from lead aircraft is in its infancy because of the complexity of the problem, but it should continue to be pursued in future years. On the other hand, the work in fault-tolerant integrated modular architectures is at a high TRL and ready for transition to industry. Finding: Wake Vortex. While wake vortex interac- tions have an obvious impact on capacity, there are equally important safety considerations, and the AvSP is not sufficiently involved in the wake turbu- lence effort. Recommendation: Wake Vortex. NASA should in- clude wake turbulence interaction models in its Control Upset Prevention and Recovery dynamics modeling and simulation technologies work. Cur- rent models used in airline training simulators are quite crude and provide insufficient fidelity for ef- fective pilot training purposes. The committee was pleased to observe the excite- ment in using the 1/20 scale model 757 for both flight and wind tunnel tests of control upset and other tasks. Such tests could integrate and coordinate the diverse activities in SAAP. The collaboration with other relevant parties (the FAA, DoD, and industry) appears to be excellent in this subproject.

so Vehicle Health Management and Flight Critical System Design Task The goal of this task is to research technologies to reduce loss-of-control accidents and system or component failures on the vehicle itself. Even within this single task, there is a large array of activities, from structures health evaluation to software integ- rity evaluation. Research is conducted in the follow- ing areas: Structures health management Flight systems health management Verification of neural networks Mode confusion Software safety Requirements modeling · Recoverable computing Modular avionics · Electromagnetic susceptibility of avionics · Neutron particle effects on flight critical systems · Validation methods The TRL also varies widely across the work in this task: Some of the software work is at a relatively low TRL. while some of the structures health monitoring work is near product development stage. The net fund- ing for the Vehicle Health Management and Flight Critical System Design work is $4.8 million for FY03 and is scheduled to be $5.1 million for FY04. The committee found a number of activities in this task worthy of commendation. It was particularly im- pressed by the specific research activity in two areas: structures health management, particularly the fiber- optic strain systems (FOSS), and pilot confusion over automation control-display modes. The structures health management activity is an area that NASA should showcase in the program. It has made significant progress in a relatively short amount of time, and NASA has truly catalyzed breakthroughs in this area. In general, the cost-benefit analysis work done in this area is impressive, and it is clear that NASA knows what it would take to install and Held its systems. The committee found the task to be well thought out in terms of the interaction between corro- sion and other properties of aging materials and the measurement and diagnosis of structural faults. The FOSS work at NASA Langley is interesting with an appropriate blend of fundamental and user-driven re- AN ASSESSMENT OF NASA 'S AERONA UTICS TECHNOLOGY PROGRAMS search. The potential payoff in this area is very high. The mode confusion work also has a very high poten- tial payoff, and the work being performed in this area is novel and of high quality. The committee also encourages NASA's contin- ued involvement in the verification of flight systems, particularly as software becomes more complex and new issues must be addressed, bringing corresponding increases in cost and development time. In addition, NASA should continue to foster the introduction of object-orientecl (OO) programming into the flight criti- cal software area. Flight critical software is software onboard an air vehicle that is used to control the vehicle and whose fail- ure would lead directly to the Toss of that vehicle. Be- cause of the cost of recertification, this is an area in which commercial companies are slow to invest, even though all recognize the eventual payoff. The payoff of GO techniques, while not directly related to safety, comes from reuse, savings in cost and time, and increased effi- ciencies in verification and validation activities. It would be useful if NASA could determine or demonstrate ways to reduce the risks and costs of re- certifying software, and its activity in OO program- ming with the FAA is a good step in that direction. As the committee noted in its subproject discus- sion, this task could benefit from fewer tasks. There is such a broad range of activities within this subproject that the committee found it unlikely they all can be brought to meaningful closure, with an appropriate TRL, by the task's end in 2005. Specifically, the com- mittee believes NASA should reorganize that portion of the SAAP that combines VHM (including the model- based diagnostics of the propulsion arm) and the detec- tion, identification, reconfiguration, and recovery part of CUPR in a single anomaly detection, identification, and reconfigurationlrecovery structure. This would eliminate the appearance of redundant research efforts and further enable functional integration. Finagling: Portfolio Breadth. The Single Aircraft Accident Prevention activities are linked by their common goal (reducing system or component fail- ures on the aircraft) but not necessarily by common expertise or research methodologies. Similar activi- ties appear to be taking place in multiple subtasks. Recommendation: Portfolio Breadth. NASA should restructure or descope this task.

l' , ASSESSMENT OF THE AVIATION SAFETY PROGRAM In several specific areas the committee has doubts about the utility of NASA's continued involvement. While the committee understands NASA's desire to offer a complete solution to the Flight Critical System problem, the committee is not convinced that NASA should be working in fault-tolerant integrated modular architectures. Commercial companies are in this busi- ness and competing heavily with one another. The TRL of this technology is well above 6, and NASA is not needed to foster innovation. The second area that the committee questions is in personal electronic device electromagnetic susceptibility. This work. seems more appropriate for industry (i.e., airlines and airframers). The committee understands that part of this work is sponsored by an airline but believes that the effort should have low priority in the NASA research plan. Finding: Modular Architectures and Personal Elec- tronic Devices. The work in fault-tolerant inte- grated modular architectures and personal elec- tronic device electromagnetic susceptibility is at a high TRL and more appropriate for industry. Recommendation: Modular Architectures and Per- sonal Electronic Devices. NASA should terminate its involvement in modular architecture develop- ment and electromagnetic interference activities in order to concentrate resources in other less-re- searched areas of the program. Propulsion System Safety Technologies Task The purpose of the Propulsion System Safety Tech- nologies task is to reduce propulsion system failures as a factor in civil aircraft accidents through the predic- tion, detection, and testing of propulsion system mal- functions and failures. The propulsion system safety team works in system health monitoring, crack-resis- tant blades and disks, and engine containment. This effort is conducted at NASA Glenn and has a net bud- get of $4.1 million per year in FY03 and FY04. The committee found the researchers to be knowl- edgeable and familiar with the relevant work in the outside community. In general, the task was well orga- nized and had a more focused goal and approach than the other tasks in SAAP. The committee found two ar- eas worthy of notice: model-based diagnostics and en- gine sensor technology, particularly the eddy current sensors. Also, the committee found that NASA has played a key role in integrating the various aspects of 81 crack-detection technologies sensors, algorithms, and testing resources. NASA's involvement in model-based diagnostics shows promise for onboard diagnostics and is a worth- while investment, but it could benefit from integration with related subtasks in SAAP. Finding: Integration of Related Activities. The model-based diagnostics subtask is not well inte- grated with related activities in Single Aircraft Ac- cident Prevention. Recommendation: Integration of Related Activities. NASA should integrate model-based diagnostics with the vehicle health monitoring activities in the Vehicle Health Management and Flight Critical System Design task when it plans the future of these tasks. NASA efforts in embedded technologies with eddy current sensors offer good promise in the early detec- tion of faults. Engine companies are also working on these technologies, however. Finding: Eddy Current Sensors. Some of the eddy current sensor work may be redundant with the work by industry. Recommendation: Eddy Current Sensors. NASA should perform additional experimental work and operational testing on these resilient sensors and other sensors under development by the engine companies only if it is leading and not following the · ~ engine compames. In general' the work in engine disk crack detection and engine materials research is well integrated and following good experimental practices. The commit- tee believes this work could be enhanced with addi- tional research at high temperatures. Finding: High-Temperature Engine Materials. NASA lacks some basic research activities in alter- native high-temperature engine materials. Recommendation: High-Temperature Engine Ma- terials. NASA should also foster progress into other engine materials and heat-treating technology. This work might benefit from additional university in- volvement.

82 AN ASSESSMENT OF NASA 'S AERONA UTICS TECHNOLOGY PROGRAMS The CUPR task works to reduce accidents due to vehicle loss of control. This task focuses on three ac- tivities: modeling and simulation, to characterize air- craft dynamics under upset conditions; system tech- nologies, to develop onboard prevention and recovery methodologies; and validation, to evaluate the tech- nologies developed and transition them to the commer- cial sector. This task is operating with a net budget of $1.5 million in FY03 and $1.0 million in FY04. Control Upset Prevention and Recovery Task flying VMC. Procedures might be significantly im- proved (in particular, departure delays might be re- duced) if, rather than waiting until the required 2 min- utes has elapsed before the next IMC departure can take place, the pilot were able to proceed as soon as the wake has dissipated. There are two outstanding re- search issues here: assessing the strength of the wake and developing an understanding of tolerable wake strengths for each type of aircraft. The current NASA research focuses on upset recovery. To apply CUPR work to wake vortex encounters, NASA should per- form additional research to better understand the onset of loss of control, especially when it results from a wake encounter. As in other tasks within this subproJect, a number of diverse areas are being worked on. A wide range of technologies is required to solve problems related to The committee believes that the research being performed by NASA in this area is well justified. There are different regimes for loss of vehicular control, and scientists and engineers have been unable to map these regimes thoroughly as yet. The aeronautics databases usually do not adequately describe the attitude, angle of attack, and sideslip envelopes encountered during loss of control. Furthermore, there is a significant dif- ference between extrapolated data and experimental data. Without experimental data, valid classes of upset responses cannot be developed. The emphasis of this research program is address- ing core NASA issues and is on target. Based on the progress made to date, the committee believes that NASA should ensure continuation of the task beyond the current 5-year plan. The committee found the work on the free-flying airplane for subscale environmental research (FASER), a small remote control vehicle, to be promising. Work on a simple aircraft such as FASER can lead to discov- eries that will apply universally to all aircraft and is a way for NASA to gain meaningful insights into control upset and recovery. The committee applauds the work in extending the aerodynamic database of airline training simulators to include poststall recovery training. The end result will be a validated process for developing and validating large angular and angular-rate mathematical models for upset training. This is a prime example of NASA draw- ing successfully on the expertise of its research cen- ters' personnel in a particular area. The committee believes the CUPR research, par- ticularly in recovery and upset dynamics, may con~ib- ute to the research being performed at the FAA and NASA (including the ASP WakeVAS project) on the wake vortex spacing problem. Currently, conservative in-trail and lateral spacing buffers are used to prevent wake encounters during IMC approaches. However, pilots are allowed to develop their own spacing when _ loss of control, and it is difficult to produce an effective product if the research components are isolated from one another. The committee mentioned earlier that it believes the fault detection, isolation, and recovery work in CUPR should be combined into a general cat- egory of anomaly detection, identification, and reconfiguration/recovery. The committee believes the work involved in scale-model testing serves to inte- grate the diverse components involved in the CUPR tasks, and NASA should increase its efforts in such . . . . . 1ntegrahon achvltles. Finding: Involving Academia. The committee is aware of the partnerships with institutions such as the State University of New York, the University of Minnesota, the University of California at Berke- ley, and the Georgia Institute of Technology but finds that university involvement is generally not a well-integrated part of the program. Recommendation: Involving Academiae NASA should increase its partnerships with academia in the Control Upset Prevention and Recovery task. Because of the low TRL of the CUPR activities, academia could make meaningful contributions in this area. Accident Mitigation Subproject The Accident Mitigation subproject is composed of two tasks: Systems Approaches to Crashworthiness and Fire Prevention. Because the crashworthiness task reached completion at the end of 2002, it is not part of

ASSESSMENT OF THE AVIATION SAFETY PROGRAM the NRC review. The Fire Mitigation effort is funded at $2.2 million net for FY03 and $2.6 million net for FY04. The committee notes that NASA has announced plans to close the Impact Dynamics Research Facility at NASA Langley. This facility is the sole U.S. facility for crash test research pertaining to aeronautical struc- tures. While the budgetary constraints that may have contributed to this closure are understandable, the loss of this facility is tantamount to the end of full-scale, experimental research on crash-resistant and survivable aircraft structures in the United States. This state of affairs is of concern to the committee. Fire Prevention Task The goal of the Fire Prevention task is to develop technologies to reduce fatalities from in-flight fires, postcrash fires after a survivable crash impact, and fuel tank explosions. The Fire Prevention task is composed of three subtasks: safe fuels, fire detection sensors, and fuel inerting. The committee found the researchers to have a high level of technical competence in all areas and to have a good understanding of the problems they are addressing. The safe fuels objective of reducing the likelihood of fuel tank explosions is one of the transportation safety improvements most wanted by the National Transportation Safety Board. The main thrust of the research is to create a jet fuel with a higher flash point, using a variety of approaches. This work requires close ties with all of the relevant players: the FAA, DoD, Boeing, refineries, and the Environmental Protection Agency. Finding: Safe Fuels. The safe fuels subtask is quite mature and essentially at a maintenance level while awaiting a phase where all the players are ready to tackle the enormous job of introducing a new, safe aviation fuel. Recommendation: Safe Fuels. When NASA reprior- itizes its activities for future program phases, it should reinvestigate the need for near-term activity in the safe fuels area. The sensors subtask focuses on the use of chemical species detection sensors to augment smoke sensors, particularly in aircraft cargo bays. This subtask has made good progress, and the path to fielding a viable . . . . 83 system seems fairly straightforward assuming that there are no major problems with scheduled field tests and that packaging and installation issues can be resolved quickly. NASA and its partners appear to be taking advantage of the advances in microelectronics and pro- cessing to arrive at an effective technology for fire de- tection. This work is also quite mature. NASA's fuel inerting subtask is focused on pro- viding efficient onboard air separation techniques that provide the nitrogen-rich gas needed to inert the fuel in the tanks. It may be possible that the separated oxygen from the process can satisfy requirements for the onboard oxygen also. The technology relies on effi- cient membranes for the separation. Finding: Fuel Inerting. In December 2002, the FAA performed representative tests on the ground with an apparently less sophisticated inerting system and obtained encouraging results. Both the FAA and NASA are working with industry to Drovide the inerting capability. ,,' ~ Recommendation: Fuel Inerting Cost-Benefit An- alysis. NASA should perform a cost-benefit analysis involving the FAA, NASA, and industry to deter- mine if further basic research in fuel inerting is warranted at this time. Recommendation: Long-Term Research in Fuel Inerting. The fuel inerting task should be separated into a near-term, product-oriented activity and a longer-term, research-oriented activity involving in- dustry to produce more efficient air separation tech- niques. Synthetic Vision Systems Subproject The Synthetic Vision Systems (SVS) has as its goal "to eliminate low-visibility-induced incidents and acci- dents."4 SVS utilizes a terrain database, the Global Po- sitioning System (GPS), and altitude sensing to give the pilot a computer graphic of the out-the-windscreen view of the ground with key instrument data superposed. The objectives of the SVS subproject address sev- eral of the most critical aircraft safety concerns: con- trolled flight into terrain, runway incursions, and low- 4D. Baize, NASA-Langley, "Synthetic Vision Systems sub- project introduction," presentation to the panel in February 2003.

84 visibility-induced approach and landing mishaps. The subproject is divided into three tasks: Commercial and Business Aircraft, General Aviation, and Enabling Technologies. The entire subproject has a net funding level of $7.2 million in FY03 and $7.0 million in FY04. The SVS work stimulated a very spirited discus- sion among the panel and committee members, and they still have many unresolved questions about the viability and future of synthetic vision systems, par- ticularly with regard to their operational implementa- tion. The committee agreed that the technology is ex- citing and holds promise for both safety and efficiency but was uncertain about its market value. Overall, how- ever, the committee believes there are substantive re- search issues yet to be addressed in SVS, particularly in display human factors, pilot training, and the integ- rity and reliability of the terrain database system. The committee believes the display interface re- mains one of the most substantial research issues. It also believes that head-mounted display technology may enable 360-degree viewing and that NASA should thoroughly evaluate such methods of display. In addi- tion, although experienced instrument-rated pilots may be able to revert to conventional means for aircraft con- trol in the case of SVS system degradation or failure, those with limited conventional instrument flying ex- perience may be more susceptible to disorientation and loss of control. NASA should conduct studies to deter- mine if additional training will be needed. The com- m~ttee believes that basic research in these human fac- tors considerations should be continued within NASA. If these questions can be answered, the committee sees SVS as a promising too} in the long run. . . . - AN ASSESSMENT OF NASA 'S AERONA UTICS TECHNOLOGY PROGRAMS Finding: Synthetic Vision Systems Long-Term Re- search. NASA is overemphasizing the synthetic vi- sion product at the expense of the long-term re- search questions in synthetic vision technologies. Recommendation: Synthetic Vision Systems Long- Term Research. The Synthetic Vision Systems activity should be separated into two parts: a short-term prod- uct for handoff to and market evaluation by industry and air carriers in competition with the Enhanced Ground Proximity Warning System and long-tenn development work emphasizing human interface re- quirements, new applications to runway incursions, and other applications currently at low TRL. The committee found that SVS researchers are aware of similar work going on at DoD and are col- laborating with the Air Force at the Wright-Patterson Air Force Base. The committee encourages continued close partnering with the defense community, as SVS work has obvious relevance to that community. The committee recognizes that there are additional challenges associated with database integrity and field of view for rotorcraft, but the number of controlled flight into terrain accidents experienced by rotorcraft warrants the additional research effort. Finding: Synthetic Vision Systems and Rotorcraft. Synthetic vision could significantly enhance rotor- craft safety. Recommendation: Synthetic Vision Systems and Rotorcraft. NASA should expand its synthetic vi- sion work into the rotorcraft area. The potential safety improvements that could be obtained with synthetic vision would warrant its extension to ro- torcraft. Most of the above comments regarding SVS hold Due for the subproject as a whole; a few additional task-spe- cific comments are provided in the following paragraphs. Commercial and Business Aircraft Task The committee found the simulations associated with the Eagle-Vail Airport in Colorado to be impressive. Finding: Synthetic Vision Systems Cost-Benefit. The incremental value provided by Synthetic Vision Systems for commercial and business aircraft is unclear given the capability of many existing flight management systems on these aircraft and given the proficiency of their pilots under instrument flying conditions. Recommendation: Synthetic Vision Systems Cost- Benefit. NASA should perform, with industry, a cost-benefit analysis for the technologies necessary to improve the existing enhanced ground proximity warning system capability and should reprioritize the activities in this area accordingly. Finding: Synthetic Vision Systems Field of View. There is no research into field-of-view requirements for either head-down or head-mounted Synthetic Vision Systems displays.

ASSESSMENT OF THE AVIATION SAFETY PROGRAM Recommendation: Synthetic Vision Systems Field of View. NASA should establish field-of-view re- quirements for head-down synthetic vision displays. This would include addressing the efficacy of syn- thetic vision in large excursion maneuvers. NASA should thoroughly investigate head-mounted dis- play technology that may improve field-of-regard operation by the pilot. General Aviation Task Again, the committee found the SVS- systems as demonstrated to be impressive. The system potentially gives a general aviation pilot with minimal instrument flying skills the opportunity to operate under a broader spectrum of flight conditions. Concerns about cost ef- fectiveness and market value are particularly prevalent in the general aviation area. The committee has some concerns about both the operational concept (how the system will be used and the pilots trained) and the ex- pected final system costs. A review of earlier cost- benefit assessments and predictions is warranted, as they may affect NASA's future research priorities. The committee suggests that NASA develop a rigorous method for establishing field-of-view requirements for head-down SVS displays and that it draw upon the wide array of literature in optimal display characteristics when developing SVS displays. Finding: Synthetic Vision Systems and General Aviation. The biggest benefit of synthetic vision in the near term is in the general aviation arena. Recommendation: Synthetic Vision Systems and General Aviation. In future continuations of Syn- thetic Vision Systems, NASA should solidify the human-aircraft interface requirements in general aviation through tests and evaluations. NASA should work with DoD to utilize DoD's current and soon-to-be-available synthetic vision display tech- nologies for these evaluations. The committee had several other more specific sug- gestions for applications of SVS to general aviation. In particular, the committee encourages NASA to re- search the efficacy of all SVS presentations in the pre- vention and recovery of upset maneuvers. Peripheral cues can be vital to the pilot in coping with upsets; these cues are essentially absent in current SVS appli- cations. Second, the committee suggests that NASA 85 seize Project Capstone as an excellent opportunity to obtain empirical data that could help optimize the way pilots perform when they use the technology. NASA and FAA project managers should collaborate closely in the design and execution of this valuable phase of the research. The committee encourages NASA to de- velop a list of research questions to incorporate into the evaluations. Enabling Technologies Task For proprietary reasons, the committee was unable to assess the use of high-resolution WXR2100 radar to detect airborne and surface aircraft. Reports of such a use are encouraging, but without substantive informa- tion, the committee doubts that weather radar can be applied to the detection of other aircraft, both in the air and on the ground, at a nonprohibitive cost. Finding: Synthetic Vision Systems and Weather Radar. The use of WXR2100 to detect aircraft is unlikely to be successful. Recommendation: Synthetic Vision Systems and Weather Radar. NASA should thoroughly evaluate the results of the 2003 flight test involving WXR2100 radar to determine its efficacy in detect- ing surface obstacles as part of runway incursion research. In addition, NASA should review this approach in light of current FAA activities to miti- gate runway incursion and determine whether it would be useful and cost effective. Currently, SVS is not a high-priority technology for runway incursion prevention owing to high costs and a lack of infrastructure. If better ground surveil- lance becomes available, some form of wide-angle viewing becomes practical, and SVS costs are reduced, SVS may have great promise for reducing runway in- cursions. This is especially important as pressures mount to crowd more and more aircraft into major air- ports with closer and closer separations. Weather Safely Technology Pro tech Background The goal of the Weather Safety Technology project is to reduce the frequency and severity of weather-re- lated accidents and injuries. This project was funded at

86 . . i, · . , a net value of $14.7 million in FY03 and is composed of two subprojects: Aircraft Icing and Weather Acci- dent Prevention. The Aircraft Icing subproject is funded at $5.0 million (34 percent of the project bud- get) and the Weather Accident Prevention subproject is funded at $9.7 million (66 percent of the budget). The Weather Safety Technology project is spread across several NASA research centers. Icing work is performed at NASA Glenn, turbulence work is split between NASA Dryden and NASA Langley, commu- nications work is performed at Glenn, and the aviation weather information work is conducted at Langley. The project management is based at Glenn. Po~fo/io The project objective is to develop and foster tech- nologies that will significantly reduce aviation acci- dents and incidents due to atmospheric conditions, in- cluding convection, limited visibility, turbulence, and icing. The project encompasses a broad set of activi- ties, including sensors, decision aides, design and analysis tools, data dissemination, and training aids. The activities range from fundamental research, such as in the Aircraft Icing subproject, to applied re- search, such as in the Aviation Weather Information (AWIN) task, where work is approaching TRL 6. Many, but not all, of the activities involve strong inter- action with the FAA, other national laboratories, and industry. The Weather Safety Technology project, with the exception of the Aircraft Icing subproject, is weighted toward applied research more than basic research, but the selected projects respond to well-recognized weather safety risks. Finding: Terminal Area Operations. The weather safety portfolio focuses solely on en route operations and not on terminal area operations, where many weather-related accidents occur. Recommendation: Terminal Area Operations. NASA should include weather research in both re- gional and terminal area operations in its future work plans for Weather Safety Technology. Program Plan This project has a variety of subprojects, with plan- ning that is designed to address the individual goals AN ASSESSMENT OF NASA 'S AERONAUTICS TECHNOLOGY PROGRAMS and objectives. For instance, the Aircraft Icing sub- project is an important sustaining activity that provides a mix of fundamental research, design and analysis tools, and educational products. In contrast, the Avia- tion Weather Information task has already completed some of its objectives, such as developing graphical weather displays in the cockpit and fostering their cle- velopment by industry, and is now turning to integrat- ing information from a variety of diverse sources. The committee found the overall project planning to be good, with goals and objectives appropriately designed to address aviation weather hazards. There is, however, a serious concern that the planning of some subprojects led to a large number of participants, re- quiring considerable coordination efforts for NASA managers. This has diminished the internal technical base at the various NASA centers. Finding: Research Outsourcing. The Weather Safety Technology project is outsourcing too much of its research effort. Recommendation: Research Outsourcing. NASA should determine whether the current practice of using a large number of contract participants is di- minishing the technical skills of government staff, who are being called on to manage contracts, and should modify it as necessary. Finding: Project Exit Criteria. There are no clear exit criteria for many of the projects. Recommendation: Project Exit Criteria. NASA should develop clear exit criteria for each of its projects. The committee recognizes that it is difficult to as- sess the impact of weather safety research because there is a lag between the development of new technologies and their implementation in significant enough num- bers to obtain useful field data. In addition, the existing data-collecting mechanisms at NASA and FAA (lo not formally obtain weather incident information, which is vital to understanding the exposures to various risks and to better guide safety programs such as the AvSP. Finding: Evaluating Research Impact. It is unclear how the weather safety projects and products will measure their impact on aviation safety.

ASSESSMENT OF THE AVIATION SAFEl,Y PROGRAM Recommendation: Evaluating Research Impact. NASA should work with the FAA and industry to develop a risk assessment methodology and associ- ated field indicators that together would document how safety is enhanced as the result of a weather technology product, even with limited deployments of the product. Technica/ Performance The committee was very impressed with the over- all technical performance of the Weather Safety Tech- nology project. The Aircraft Icing subproject is a na- tionally recognized facility with top-notch staff, is well managed, and has excellent collaborations with other federal government organizations and industry. Its educational outreach work is highly valued by general . . aviation. The AWIN and Weather Information Communica- tions (WINCOMM) tasks have also been successful in dramatically increasing the availability of graphical weather in the cockpit during en route flight by very high frequency (VHF) and satellite communications methods. If the tropospheric airborne meteorological data reporting (TAMDAR) subtask to provide meteo- rological data from aircraft to the ground weather in- frastructure is successfully completed and deployed, it will be a great benefit to aviation weather forecasting and, in turn, safety. The committee is, however, very concerned that calibrated TAMDAR data have not yet been made available to the meteorological community and that the business case was based on turboprop com- muter aircraft, which are rapidly being replaced by re- gional jets. This is discussed in more detail in the AWIN section. The Turbulence Prediction and Warning System (TPAWS) task had a major success when it demon- strated an in-flight 95-second warning of turbulence using forward-Iooking microwave Doppler radar. However, this work needs to continue in order to meet the industry request for longer warning times. Further, it needs to be more connected to the research spon- sored by the FAA; recent meetings suggest that the is- sue is being addressed. User Connections Most of the subprojects within the Weather Safety Technologies project are well connected to the avia- tion community and to industry. The NASA Glenn 87 icing facility has staff that are highly respected by in- dustry and academia and works very closely with the National Center for Atmospheric Research and other organizations. The AWIN and WINCOMM tasks have also worked closely with pilot groups and industry. The turbulence activity has worked closely with a commer- cial airborne weather radar manufacturer but has not collaborated with the FAA turbulence product team, a matter that has been recently addressed (but not solved) by both parties. Assessment by Subyroject Aircraft Icing Subproject The focus of the Aircraft Icing subproject is to en- able the elimination of icing as a cause of aircraft acci- dents. This is accomplished through a combination of (1) basic research in icing physics, (2) more applied, focused work in tool design and technologies for icing detection, anti-icing, and deicing, and (3) the develop- ment of icing educational materials for pilots, opera- tors, and engineers. The Aircraft Icing subproject is funded at $5.0 million in FY03 and FY04. It is com- posed of three tasks: Design and Analysis Tools, Air- craft Ice Protection, and Education and Training. The committee was very impressed with the icing program, its facilities, and staff. The icing test facility is clearly a unique national asset that is vital to the air transportation community, both civil and military. The effort is well managed, and the key staff are experi- enced professionals, knowledgeable in both the theo- retical and practical aspects of icing phenomena and their effect on aircraft aerodynamics. They are also well versed in the FAA certification process. The icing ac- tivity is characterized by high quality and productivity. The committee was pleased to observe a strong coupling with the National Transportation Safety Board (NTSB), the FAA, industry, and the icing effort at the National Center for Atmospheric Research (NCAR). Several dozen test flights are conducted each year us- ing NASA Glenn's instrumented aircraft in collabora- tion with NCAR remote-sensing experiments. This co- operative effort is extremely important and should be maintained. Finding: Icing. The Aircraft Icing subproject rep- resents the best technical work in the Aviation Safety Program and is an important national asset.

88 Recommendation: Icing. NASA should ensure the Aircraft Icing subproject continues as a well-bal- anced research effort to understand and mitigate aircraft icing. If possible, work should be increased in areas such as anti-icing fluids and the assessment of holdover times. Design and Analysis Tools Task The Design and Analysis Tools task develops prod- ucts to assist in the design, certification, and qualifica- tion of aircraft and aircraft components. This task con- ducts research in six areas: super-cooled large droplet (SLD) engineering tools, LEWICE, SmaggIce, LEWICE3D, experimental methods and databases, and flight testing. This task has developed and implemented a number of tools in ice growth prediction, ice shape prediction, thermal ice protection system performance, and the aerodynamics of aircraft with ice. It is funded at $2.9 million in FY03 and FY04. The work of supporting the FAA in its efforts to modify the criteria for icing certification based on a better understanding of SLD is very important and has already yielded a significant amount of new informa- tion. This should be continued vigorously. Development of the LEWICE computer code has produced a program that is ready for production and will be subject to evolutionary improvements. How- ever, the LEWICE3D program is still in the develop- ment stage and needs continued strong support. This is a very important too] for aircraft designers and re- searchers who work with three-dimensional bodies. For example, a wing, which usually uses LEWICE for analysis, has unique three-dimensional flow and drop- let trajectories that can only be handled accurately by LEWICE3D. The work on experimental methods and flight test- ing has been of a very high quality and should be con- tinued vigorously to provide real-world results that augment and validate theoretical investigations. There is still much to be learned about the effects of icing on aircraft aerodynamics. Aircraft Ice Protection Task The Aircraft Ice Protection task has two objectives: to develop remote sensing technologies to detect and measure icing conditions and to develop methods to assess the performance of an aircraft under icing con- AN ASSESSMENT OF NASA 'S AERONAUTICS TECHNOLOGY PROGRAMS ditions ("smart icing" systems). This task is funded at $1.7 million per year in FY03 and FY04. The FAA-sponsored NCAR program to develop methods to forecast icing conditions more accurately and the NASA effort to sense and document actual icing conditions have met with considerable success. This is one of the most critical areas for improving the safety of general aviation airplanes with respect to ic- ing. The vast majority of general aviation airplanes have little, if any, ice protection equipment installed. Their best defense is the ability to know where icing conditions exist, and then to avoid those conditions. This collaborative work has the end goal of incorporat- in~ successful methods and technologies into the offi- cial weather forecasting system. It should be vigor- ously supported until that end goal is met. Smart icing systems, which take the critical deci- sion making out of the pilot's hands and place them in the system, are still in the early stages of development. More work will be required before the results reach a practical stage, but this is an important area to pursue further. Education and Training Task The Education and Training task seeks to create and disseminate a suite of training materials on in-flight icing. The task has developed and distributed five train- ing guides thus far (three video-based and two com- puter-based) and expects to develop more guides, as well as plan curricula for pilot training. Clearly, this is a highly applied effort on the part of NASA and is a different type of work and work product than the rest of the program under evaluation. Education and Train- ing is funded at $400,000 in both FY03 and FY04. The committee, in reviewing icing training aids, found the materials to be well made, with good infor- mation on how to plan flights in icing conditions, how to use weather reports, and how to identify escape routes. They contain some science and have interesting video clips of aircraft in icing conditions. However, the committee found the recovery procedures to be somewhat weak on piloting techniques: The aids should emphasize more fully how a pilot can tell the difference between wing stall and tail stall based on aircraft response, and they could benefit from addi- tional information on pilot perceptions and initial cues of problems. The committee suggests that the educa- tion materials should incorporate recovery methods and

ASSESSMENT OF THE AVIATION SAFETY PROGRAM also discuss asymmetric icing (such as occurred in the Roselawn accident and the Detroit accident). The efforts to improve pilot education have con- tributed significantly to the enhancement of aviation safety. The Aircraft Owners and Pilots Association (AOPA) Air Safety Foundation (ASP) is pleased with the work done in this area. That being noted, the com- mittee believes that the dissemination of the icing train- ing materials seems much too limited; perhaps with assistance from the FAA and broader efforts on the part of industry, they can be more widely distributed. Finding: Icing Education. The good work in icing education notwithstanding, several recent ice-re- lated aircraft accidents indicate that pilots are still ignoring warnings of potential icing conditions. Recommendation: Icing Education. The Aviation Safety Program should evaluate how to have a greater impact on the education and training of pi- lots and how to focus research on aids for pilot deci- sion making in icing conditions. A greater effort should be made to distribute such training aids. Weather Accident Prevention Subproject The Weather Accident Prevention (WxAP) sub- project goal is to develop technologies that will reduce weather-related accident causal factors by 50 percent and turbulence-related injuries by 50 percent by the year 2007. To accomplish that goal, it has three objec- tives: develop means to provide better weather infor- mation to the cockpit during en route flight, develop supporting communications, and develop turbulence avoidance technologies. WxAP is funded at $9.7 mil- lion in FY03 and $8.9 million in FY04. It has three tasks: Aviation WeatherInformation (AWIN), Weather Information Communications (WINCOMM), and the Turbulence Prediction and Warning Systems (TPAWS). The committee finds the WxAP goal to be ambi- tious but appropriate, for three reasons. First, weather continues to be involved with a significant number of aviation incidents and accidents. Second, national in- vestments in Next Generation Weather Radar (NEXRAD) Doppler weather radars, Terminal Doppler Weather Radars, the National Convective Weather Forecast, and the Integrated Terminal Weather System are expected to significantly improve the timely dis- 89 semination of weather and weather hazards to aircraft cockpits. Third, airborne weather radar with advanced signal processing could improve turbulence sensing significantly. The committee finds that the goals and planning were appropriate as originally formulated. However, the committee noted a certain inflexibility in the plan that prevents the investigation of topics not originally definecl, such as pilot penetration of storms in the ter- minal or transitional areas (not an en route issue, but clearly a weather safety matter). Aviation Weather Information Task The AWIN task is designed to provide the pilot with accurate, intuitive, informative, and timely weather-related information that will help to reduce the number of accidents and incidents due to weather. The task conducts research in airborne hazard awareness, satellite aviation weather products, tropospheric air- borne meteorological data reporting, and interface and display technologies. The task has a budget of $4.1 million in FY03 and $3.7 million in FY04. The AWIN task goal is to reduce en route acci- dents attributed to a lack of weather information by 25 to 50 percent. This is an important goal, since the an- nual AOPA ASP Nall reports indicate that adverse weather continues to be a major safety concern for air transportation, especially for low-end general aviation. For transportation aircraft, turbulence encounters are . · · · · 1 - causlng serious passenger Injuries and are a mayor safety issue with flight attendants. The 2002 ASP re- port states "weather is usually the culprit in cruise acci- dents, and was the cause in 15.2 percent of all pilot- related fatal accidents in 2001."5 However, the Nall report also indicates "attempted VFR flight into IMC continues to be the most deadly weather-related acci- dent cause, with 84 percent involving fatalities."6 This suggests that the AWIN program should consider ex- panding its scope to improve weather situational aware- ness during the terminal and approach phases of flight. The committee finds that the plan to achieve the stated objective is well formulated and reflects a com- petent understanding of the issues and developments needed to reduce the en route weather risks. The air- borne weather data collection, fusion, and decision aide JAOPA Air Safety Foundation. 2002. 2002 Nall Report, p. 4. 6Ibid., p. 7.

go AN ASSESSMENT OF NASA 'S AERONAUTICS TECHNOLOGY PROGRAMS approaches are particularly important. Also important are the collaborations with the FAA, NCAR, the Na- tional Oceanic and Atmospheric Administration (NOAA), and academia to learn how to incorporate current and future satellite observations into aviation forecasts. Future weather satellites can be expected to provide high-rate, high-resolution, multispectral imag- ery cued by numerical model forecasts to areas of bad weather, so it is appropriate that NASA participate in their development. An effort is planned for 2007 using the Geostationary Imaging Fourier Transform Spec- trometer (GIFTS) satellite to determine the benefit of altitude-resolved temperature, moisture, and wind, as well as other advanced weather satellite technology insertion. If this effort goes forward, it will be a note- worthy and potentially very valuable element in the program and NASA should actively participate. It is not clear whether the objective goal of reduc- ing accidents by 25 to 50 percent will be achieved, and the evidence confirming the success of the effort is unlikely to be available for some years aher the AWIN task has been completed. However, as mentioned above, the committee recommends that NASA develop some field indicators to help understand the potential benefits of this initiative, perhaps with the help of in- dustry groups such as the Air Line Pilots Association (ALPA), the Air Transport Association (ATA), and AOPA. The committee observes that this program involves many other organizations and is concerned that the ef- fort to coordinate such a complex activity is preventing NASA staff from becoming fully involved in the tech- nical aspects. The AWIN effort to stimulate commercial meth- ods to transfer graphical weather images to the cockpit appears to have been successful, as measured by the variety of commercial methods now on the shelf. The program is now focusing on the integration of various onboard weather data to help the flight crew avoid weather hazards, and that effort is also appropriate and needed. Finding: Aviation Weather Information Integration. The operational deployment of the Integrated Termi- nal Weather System provides an opportunity to inte- grate existing Anation Weather Information products with Integrated Terminal Weather System graphical six-level weather images to provide an improved awareness of weather when operating in the Inte- grated Terminal Weather System coverage areas. Recommendation: Aviation Weather Information Integration. NASA should conduct an assessment of the benefits of integrating existing onboard avia- tion weather information with uplinked convection and wind data from the Integrated Terminal Weather System. Should this appear beneficial, part task or full task human factors studies on pilot decision making and follow-on field demonstrations should be conducted. Aircraft penetration of convective weather in the terminal area is an unresolved issue. Basic studies were conducted in the 1960s at the National Severe Storms Laboratory by J.T. Lee,7 providing advice to airline crews on avoiding areas with heavy precipitation. More recently, however, a NASA-sponsored study of aircraft operations in the Dallas-Fort Worth area indi- cated that airline crews are penetrating areas of high precipitation during arrivals.8 This work raised several unaddressed key issues regarding pilot penetration of convective weather. Finding: Pilot Penetration. It is unclear whether tra- ditional six-level precipitation is the appropriate indicator of turbulence and why pilots choose to penetrate those levels against their training and air- line operating procedures. Recommendation: Pilot Penetration. The Aviation Weather Information task should work with the FAA, airlines, and pilot communities to address the issue of pilots penetrating convective weather, in- cluding the adequacy of existing onboard weather radars for portraying flight hazards, the adequacy of current pilot training in their use, and related pilot decision making. 7J.T. Lee. 1962. A Summary of Field Operations and Data Col- lection by the National Severe Storms Project in Spring 1961. Norman, Okla.: National Severe Storms Laboratory; J. T. Lee, L.D. Sanders, and D. T. Williams. 1964. Field Operations of the Na- tional Severe Storms Project in Spring 1963. Norman, Okla.: Na- tional Severe Storms Laboratory. Dale Rhoda and Margo Pawlak. 1999. An Assessment of l~hun- derstorm Penetrations and Deviations by Commercial Aircraft in the Terminal Area. Project Report NASA-A/2. Lexington, Mass.: Massachusetts Institute of Technology Lincoln Laboratory.

ASSESSMENT OF THE AVIATION SAFETY PROGRAM In the last 10 years, the FAA Aviation Weather Research Program has developed new automated fore- casts that predict the future location of convective sys- tems. For example, the National Convective Weather Forecast is being used operationally by FAA traffic flow managers. In development are similar tools that will show current and future areas of turbulence and icing. Since these tools may be very beneficial to flight crews, it is important that appropriate research be con- ducted to determine how they can efficiently be pro- vided to the cockpit and used safely by pilots. Many of these products are available today in the Aviation Digi- tal Data Service system and could be used by AWIN as in-flight display products. Finding: Weather Display Interfaces. There are a number of outstanding research questions in the area of weather display interfaces. Recommendation: Weather Display Forecasts. NASA should include weather research to deter- mine the methods and value of providing evolving I- to 2-hour automated weather forecasts of convec- tion, turbulence, and icing to the flight crews of vari- ous classes of aircraft. Such information would maximize avoidance of unsafe conditions in termi- nal, transition, and en route airspace. Finding: Weather Display Consistency. There is a need for consistency in weather information dis- plays in the aviation weather community. It is par- ticularly important that the information being pre- sented to the flight crew be consistent with that seen by FAA air traffic controllers and the airline opera- tions centers. Recommendation: Weather Display Guidelines. The Aviation Weather Information program should help the FAA and industry to develop best practices guidelines for the integration and display of uplinked and onboard weather information. The value of uplinked weather to general aviation aircraft is convincing for both safety and utility pur- poses, especially for less costly piston aircraft. How- ever, the business case for provisioning transport- category aircraft with graphical uplinked weather prod- ucts is not compelling. Unless the cost-benefit case can be made, what seems to be of great value may come to naught. 91 Finding: Uplinked Weather. NASA's justification for research in uplinked weather to transport air- craft Is unconvincing. Recommendation: Uplinked Weather. When evalu- ating the potential benefits and costs of new tech- nologies, especially those for transport aircraft, NASA should ensure that the views of corporate operations officers and financial decision makers, as well as those of the pilot community, are heard to ensure that the return on investment is sound. Lower atmosphere data such as humidity and tem- perature are becoming increasingly important for sup- porting forecasts of convective growth, ceilings, and visibility. The Tropospheric Airborne Meteorological Data Reporting (TAMDAR) system is designed to meet that need, and the committee supports the use of such a system. Finding: Tropospheric Airborne Meteorological Data Reporting (TAMDAR) Data Availability. Data from the developmental TAMDAR are still not available to others who will use the information to support advanced meteorological forecasts. It ap- pears that there are ongoing sensor problems, which may account for the delayed access to output data. It is vital that the data measurement methods be well understood if the instrument is to be useful for aviation weather forecasting. Recommendation: Tropospheric Airborne Meteo- rological Data Reporting (TAMDAR) Data Avail- ability. Information about TAMDAR and its data should be made available to appropriate aviation weather scientists as soon as possible to enable veri- fication and validation. The TAMDAR test flights must be done in such a way as to allow a side-by-side comparison of the TAMDAR turbulence eddy dissipation rate (EDR) data and data produced when the turbulence EDR algorithm is implemented on the large commercial fleet. A way must be found to ascertain comparability. In addition, there needs to be close collaboration with the FAA Aviation Weather Research Team, which will be a key recipient of the data. Finding: Tropospheric Airborne Meteorological Data Reporting (TAMDAR) Data Comparison. The

92 , AN ASSESSMENT OF NASA 'S AERONA UTICS TECHNOLOGY PROGRAMS TAMDAR project is somewhat disconnected from FAA and industry activities. Recommendation: Tropospheric Airborne Meteo- rological Data Reporting (TAMDAR) Data Com- parison. The TAMDAR flight tests planned for 2004 should not commence until the results of the Twin Otter and WP3-D flight tests show compliance with the stated requirements. Also, the TAMDAR project should ensure that the needs of the FAA Aviation Weather Research Convective Product Team are carefully understood, in particular their strong need for humidity data to support tactical storm growth and decay forecasts. Finding: Tropospheric Airborne Meteorological Data Reporting (TAMDAR) Data Value. The TAMDAR project assumed the sensors would be carried by turboprop commuter aircraft that oper- ate at the lower altitudes. With the large-scale re- placement of turboprop aircraft by turbine aircraft that operate at higher altitudes, the value of TAMDAR data may be significantly lessened. Recommendation: Tropospheric Airborne Meteo- rological Data Reporting (TAMDAR) Data Value. The business case for TAMDAR should account for the ongoing replacement of turboprop aircraft by regional jets. Weather Information Communications Task The WINCOMM task seeks to develop advanced communications technologies to assist in the clissemi- nation of weather information to the cockpit. This ef- fort is funded at $2.8 million in FY03 and FY04. The research into ways of making current weather information accessible to pilots has been quite success- ful. Several representatives from industry report that their progress in developing air-ground data links has been significantly helped by the WINCOMM program. As a result, a number of unique media for accessing and transmitting weather data into the cockpit have been successfully demonstrated. This program appears to be reaching maturity, and the committee was uncer- tain what additional value NASA could provide in this area. NASA intends to continue the AWIN task, focus- ing on the development of increased uplink bandwidth . . . In anticipation of the need for more uplinked weather. NASA also intends to establish an aviation uplink in- frast~ucture that it believes will not be available from industry owing to the low volume of aviation traffic. Finding: Uplinked Weather Needs. There does not appear to be a convincing need for significant addi- tional uplinkecl weather. Until that case is made, the development of yet another aviation data link is not compelling. Recommendation: Uplinked Weather Needs. NASA should develop a prioritized list of weather infor- mation communications requirements and adjust the goals of the weather communications research accordingly. Turbulence Prediction and Warning Systems Task The goal of the TPAWS task is "to develop tech- nology to reduce turbulence-related injuries for the traveling public and aircrew."9 The task is funded at $2.8 million in FY03 and $2.4 million in FY04. Recent NASA-funded tests demonstrated a 95-sec- ond turbulence warning based on spectral analysis (sec- ond moment) of an airborne weather radar. The results, which were confirmed by NASA flight testing, are a significant accomplishment that recently received a NASA award and a sign of healthy collaboration be- tween NASA researchers and a manufacturer. The work is impressive and may be offered as an optional feature on airborne weather radars in the future, since its probability of detection (20.2 g) is 81 percent, a value that is consistent with the CAST goal of 80 per- cent detection probability of moderate or severe turbu- lence.~° However, the system was also found to have a nuisance (false alarm) alert rate of 10.53 percent. A1- though this is a metric about which CAST is silent, the acceptability of a turbulence warning system such as TPAWS will depend strongly on airline and pilot ac- ceptance of false alert performance as well. The com- 9R. Bogue, NASA-Langley, "Turbulence prediction and warn- ing systems," questionnaire completed in January 2003 (see Ap- pendix D). i°L. Cornman, NCAR, "Turbulence prediction and warning sys- tem, FY02 Flight Test Data, Weather Accident Prevention Annual Project Review," presentation to Massachusetts Institute of Tech- nology Lincoln Laboratory on November 20-21, 2002.

ASSESSMENT OF THE AVIATION SAFETY PROGRAM mittee is also concerned that the cost of this forward- looking radar may be too great for implementation by the civil transport industry during difficult times, espe- cially since the use of such a system is not a high CAST priority. The transfer of successful research results to the airborne microwave weather radar industry appears to be very good, although the committee was concerned that the connectivity between NASA turbulence re- search and the vendors was not being maintained as well as in the past, a situation that may be related to the difficult economic times for civil aviation. The level of TPAWS collaboration with other agencies and industry appears inconsistent at best. Some industries are associated with the effort, and here the collaboration is good; however, the record of col- laboration with the outside community of scientists and engineers is often unsatisfactory. There appears to be a sense of secrecy in the effort, which should not be the case with a healthy program. The committee was in- formed by NASA that there are tentative plans to de- velop a singular plan in collaboration with the FAA in this area, and the committee strongly encourages NASA to establish and strengthen such collaborations. Separately, NASA has been funding work on in situ (onboard) turbulence detection, using accelerom- eter data to compute gust load, with the plan to clatalink the information to other aircraft as a turbulence warn- ing. There has been some concern at the FAA and on the part of other scientists working in the airborne tur- bulence arena that this approach would seriously con- flict with the FAA approach, which uses eddy clissipa- tion rate. It is critical that NASA management establish and maintain a consistent approach to in situ turbu- lence detection and reporting to avoid technical con- flicts with FAA in the future. The current advanced warning from light detection and ranging (lidar) is 45 seconds, far less than the 95 seconds of the radar system. There appears to be little value to the airlines and little evidence of reliability, safety, or cost effectiveness. UnIess a much more pow- erful (and more expensive) lidar could be developed, there is no operational concept to justify the lidar work. Finding: Lidar. The potential for lidar as a sensor for turbulence detection is very limited. Recommendation: Lidar. The lidar initiative should be terminated. 93 Finding: Turbulence Warning. The Commercial Aviation Safety Team stated that turbulence detec- tion was a lower priority issue but important never- theless and that a 95-second warning would be in- adequate. A study has been done by industry to establish a practical warning time. Recommendation: Turbulence Warning. NASA should initiate a research activity in advanced air- borne-based turbulence detection to significantly improve the airborne turbulence warning time to satisfy the needs of the airline industry. System Safely Technology Pro lent Background The System Safety Technology project is designed to reduce the frequency and severity of aviation acci- dents and incidents through proactive management of safety risks using a systemwide approach. The project is composed of two subprojects: Systemwide Accident Prevention (SWAP) and Aviation System Monitoring and Modeling (ASMM). The two subprojects focus on mitigating risks associated with human error and hu- man performance and continuously monitoring the na- tional airspace to identify and analyze safety trends and precursor events. System Safety Technology is funded at $18.4 million net in FY03 and $17.7 million in FY04. Approximately 27 percent ($5.0 million) is allocated to SWAP and 46 percent ($8.4 million) to ASMM. The remaining funds support the Search and Rescue effort, which was not assessed in this review. SWAP and ASMM are based at NASA Ames. Portfolio NASA has been a mainstay of aviation system safety research for many years. It has carried the vision for what is possible and must continue to do so if the United States is to maintain the safest possible flying environment for the American traveling public. There are marry safety programs in NASA research initiatives Tithe Search and Rescue work is funded by the AvSP, but Goddard Space Flight Center is responsible for management of the subproject. As the AvSP does not have programmatic responsibili- ties in Search and Rescue, the panel did not evaluate this effort.

94 AN ASSESSMENT OF NASA 'S AERONAUTICS TECHNOLOGY PROGRAMS that have served the aviation community well for years and there are sure to be many others that show great promise for future applications. Many of the programs are user driven, both internally and externally. Unfor- tunately, the expectation that projects can be completed in 5 years appears to constrain NASA's ability to ar- ticulate a consistent and clear vision for Tong-term core research. It also seems to impact the ability to create well-defined goals that lead to an integrated research program. The goal of providing a general knowledge foundation, as in the case of human performance mod- eling, should be part of a core research program. Bal- ancing the System Safety Technology suite of applied research activities with more basic research would help sustain essential core competencies within the associ- ated groups. A specific long-range goal such as creating a fully integrated virtual National Air Space model by 2020 or 2050 for modeling total system safety and efficiency would be helpful in focusing and balancing research projects. Research initiatives should support such a central long-range goal, and the AvSP should work closely with the VAMS effort in the ASP program to achieve this. Requirements could then be more easily developed, including problem statements, standards, and test procedures. These should be established in a way that encourages innovation while maintaining fo- cus and accountability. Adclitionally, a continuous sys- tems analysis approach would be constructive in iden- tifying research priorities and allocating resources to projects with the greatest impact on safety. Program Plan Many of the safety tasks have articulated very desir- able outcomes, but plans to achieve these outcomes were often unclear or lacked measurable milestones. For ex- ample, a number of outcomes are in the form of a per- centage reduction in accidents or in fatalities. There ap- pears to be no method in place in the research program for evaluating such outcomes or for assessing progress. This gap appeared to be driven by a disconnect between the resources or time required to accomplish the target outcomes and the availability of assets and time. The committee acknowledges that some of this work is low- TRL and difficult to relate directly to measurable changes in accident mitigation. Nevertheless, the com- m~ttee believes that NASA should develop interim m~le- stones and metrics for internally evaluating the success of the System Safety Technology project relative to in- tended project deliverables. This should be done in con- junction with the Technical Integration effort. The process of research project selection, planning, resourcing, programming, and accountability within the matrix management scheme was complex and dif- ficult for the committee to understand. Some program- matic decisions appear sensible from a safety perspec- tive but do not seem to relate to an overall research plan. Technica/ Performance The committee was impressed with the technical capabilities of the NASA Ames staff associates! with the System Safety Technology project. NASA Ames has an excellent reputation for "basic applied" research. The committee encourages NASA to uphold this strong reputation by sustaining basic research programs at Ames, where scientific publication is a core value. The committee is concerned that the balance of in- house and contractor personnel is becoming heavily weighted toward outsourcing to an extent that could compromise the ability to maintain core competencies. Additionally, heavy outsourcing forces scientific per- sonnel to focus on management oversight rather than on building internal scientific activities. This discour- ages young researchers from joining the NASA team or even remaining with NASA. Basic research seems constrained in a number of areas owing to either lack of access to data or lack of resources to process available data. Good examples of this are the highly respected Aviation Safety Reporting System product and the Maintenance Human Factors task. As long as there are barriers to accessing data, basic research could languish. User Connections The committee was impressed with the establish- ment of an integrated FAA/NASA Aviation Safety R&D Plan, an Aviation Safety Working Group, and an Aviation Safety Program Executive Council, all to en- sure greater coordination of FAA/NASA research. Ef- fective use of these groups will be vital to establishing post-2004 safety research goals. In some areas NASA seems to be pursuing tech- nologies or tools that have reached maturity or are complementary to items already available in industry or other government agencies. This is true, for example, with Performance Data Analysis and Reporting Sys-

ASSESSMENT OF THE AVIATION SAFETY PROGRAM tern (PDARS), Aviation Performance Measuring Sys- tem (APMS), and the virtual reality maintenance work. It is critical that regular, open, candid product benchmarking and communications occur among NASA, FAA, industry, and other research entities in order to avoid duplication, to ensure that valuable and limited resources are effectively and efficiently allo- cated, and to sustain world-class research standards and products. Assessment by Subproject System wide Accident Prevention Subproject SWAP is the AvSP subproject devoted to human factors and its relationship to error mitigation. The fo- cus of the research is primarily in error modeling, train- ing procedures, and maintenance procedures. SWAP is also responsible for identifying crosscutting issues in human factors that relate to the AvSP as a whole or to other subprojects and tasks under the AvSP purview. SWAP is broken into four tasks: Human Performance Models, Maintenance Human Factors, Crew Training, and Program Human Factors. SWAP is funded at a net value of $5.0 million in FY03 and $5.1 million in FY04. Human Performance Models Task The Human Performance Models task utilizes cog- nition and perception models to detect and analyze hu- man error and to develop tools for system design. The task works primarily with five human performance models: Air Man-Machine Integration Design and Analysis System (AirMIDAS), ACT-R/PM, A-SA, D- OMAR, and IMPRINT/ACT-R. Each model uses a dif- ferent cognitive approach and each has a different ap- plication to sources of pilot error. The Human Performance Models task has also developed a track- ing system, the Crew Activity Tracking System, to pre- dict operator behavior and to interpret operator actions. The Human Performance Models task of SWAP is funded at a net value of $1.5 million in FY03. The activities in this area are appropriately weighted toward fundamental research. The goal is clearly to create state-of-the-art modeling techniques. While resources seem adequate for the stated goals, the 5-year program constraint appears to limit the long- term potential of this core research area. Error analysis appears to focus on error as devia- tion from nominal procedure rather than considering 95 which deviations are dangerous and which are merely alternative but still acceptable ways to accomplish the task. These alternative methods may in some cases be better than the nominal (e.g., under off-normal condi- tions). Expanding the scope of work to include accept- ability analysis may broaden the potential application of this effort. While application of NASA human performance modeling research to other efforts at NASA, such as synthetic vision research, would seem promising, there is little to show as yet. There appears to be no connec- tion with human performance modeling at other gov- ernment agencies. Finding: Collaboration with Other Agencies. There appears to be no substantive interface with human performance modeling at other government agen- cies such as the Air Force Laboratory's Human Per- formance Modeling Integration Program, the De- partment of Defense, or government laboratories such as the Human Emulation Laboratory at Sandia National Laboratories. NASA is not part of the Hu- man Performance Modeling Special Interest Area.~2 Recommendation: Collaboration with Other Agen- cies. NASA should conduct collaborative research with both the Defense Advanced Research Projects Agency and the DoD to leverage techniques devel- oped by these other agencies for piloting, decision making, estimating human error in automated sys- tems, and vigilance. There is a well-documented, short-term plan with reasonable milestones, but the long-range vision and plan for this initiative lack definitive goals and metrics. Development of a method for comparison across mod- els is encouraged, since current metrics vary from model to model. Finding: Human Factors Outreach. Much of the Human Performance Models work was done by human factors engineers for human factors engi- neers. There is too little outreach to NASA engineers in other disciplines who should be future users of these models. Additionally, program deliverables and their purposes were not clearly articulated. 12See <http://www.msiac.dmso.mil/hpm/default.asp>.

96 Recommendation: Human Factors Outreach. The NASA Human Performance Models group should work with the managers of all internal aviation re- search programs to identify each manager's need for human performance models and to support the testing of emerging models against human-in-the- loop simulation as well as flight demonstration. Recommendation: Human Factors Outreach. NASA human factors programs should publish a book or CD on the state of human performance modeling to communicate what can realistically be done in this type of modeling and to measure progress in this research area. Recommendation: Human Factors Outreach. NASA should create, document, and apply more clearly defined off-ramps for high-TRL Human Performance Models. NASA Ames has maintained an excellent reputa- tion for sponsoring and convening human performance modelers for several decades. It is essential to strive for continual high quality since human lives are af- fected by the accuracy of the safety estimates derived from these models. The committee applauds the par- ticipation of academia in the NASA Ames aviation safety work but strongly urges outreach to the govern- ment agencies listed in the finding on collaboration, above. In addition, this group has only been able to apply models to a limited number of real-world prob- lems, such as taxiing errors. The committee feels that these models can be tested and improved by applying them to additional real-world problems. NASA is developing tools in this area for others to use. However, actual and potential users, both manag- ers and researchers, should be more clearly iclentified so their input can be solicited when research and appli- cations are being identified and prioritized. Maintenance Human Factors Task The Maintenance Human Factors task is designed to develop "guidelines, recommendations, and tools directly to maintenance personnel and managers"~3 through a combination of research in understanding human error in maintenance and developing mainte- i3B. Karlki, NASA-Ames, "Maintenance Human Factors," ques- tionnaire completed in January 2003 (see Appendix D). AN ASSESSMENT OF NaSA 'S AERONAUTICS TECHNOLOGY PROGRAMS nance tools and aids to enhance safety. The mainte- nance program focuses on risk analysis, resource man- agement, advanced displays, and human error baselines. The effort is funded at $1.1 million net for FY03 and FY04. The importance of human error in the maintenance of aircraft was underscored recently by the US Air- ways Express Air Midwest Flight 5481 accident. The National Transportation Safety Board concluded in May 2003 that the probable cause of the accident, in combination with several other factors, was improp- erly set elevator control cables a maintenance over- sight. In this case, maintenance personnel skipped criti- cal steps outlined in the maintenance manual because they felt the steps were superfluous. This maintenance human factors initiative is criti- cal to reducing maintenance errors as well as to pre- venting injuries to personnel and damage to equipment. Industry applauds the effort. There are many facets to this program, but the resources seem limited relative to the need. This research group has made significant con- tributions in raising maintenance human factors aware- ness within the aviation community. However, this type of research is still in its infancy and just beginning to receive enough attention to identify data sources from which to generate statistically sound trend information. Finding: Maintenance Data Collection. Sources from which to collect data have been identified, but barriers to collection and processing seem to be slowing productive research. Recommendation: Maintenance Data Collection. NASA should develop a clear plan to include inspec- tion data and information from maintenance tech- nician training in its research data set. There is a coherent short-term plan for each of the projects, but the long-range strategic goals seem to be disjointed. The process used for selection of the par- ticular research topics was unclear to the committee. All are potentially useful tools at some level but lack the anchor of a long-term research mission. Specifi- cally, the committee is uncertain how the virtual reality and augmented reality work differs from or comple- ments what industry uses already and how such work will be applied to real-world maintenance error mitiga- tion. There also does not appear to be a systems analy- sis approach to setting priorities for the research effort.

ASSESSMENT OF THE AVIATION SAFETY PROGRAM Finding: Goals for Maintenance Research. The Main- tenance Human Factors task is an excellent activity but seems to lack clearly defined long-range goals. Recommendation: Goals for Maintenance Research. NASA's Maintenance Human Factors task should set clear and quantitative long-range goals and test its research against these goals annually. This is an area for long-term research and should be an area for de- veloping enduring core competencies. Finding: Virtual/Augmented Reality. The project to create virtual and augmented reality tools for main- tenance technicians seems to be operating without a clear understanding of what is available today in automation for maintenance technicians and the realities of an all-weather, real-world airline main- tenance operation. Recommendation: Virtual/Augmented Reality. NASA should formally assess the enhanced displays for maintenance research work, including what is currently in use by the airline industry, to deter- mine a more focused and practical approach to vir- tual and augmented reality tools for maintenance. The external community of maintenance human factors researchers was described as small, and there were said to be very close connections between agen- cies and academia. However, the allocation of roles and responsibilities among FAA, NASA, the Navy Safety Center, the Air Force Safety Center, and other research entities was not clear. Finding: Outreach to Community. There are a few omissions in the links to the outside community in this task. Recommendation: Outreach to Community. NASA should establish links to the Air Force Safety Cen- ter as well as airframe and power plant training in- stitutions. NASA should perform an active outreach to the maintenance technician unions for program planning, research vetting, and research participa- tion. NASA should collaborate with the Professional Aviation Maintenance Association on aviation maintenance research and with maintenance tech- nician schools such as the Stratford School to col- lect data and provide research results to enhance safety training. 97 C~ · ~ 7 rew 1 raining 1 ash; The goal of this task is to develop training tech- niques and tools to help pilots avoid making errors that lead to accidents and to manage in-flight problems in situations brought about by external circumstances such as weather or system failures. The effort is funded at $1.9 million net for both FY03 and FY04. The Crew Training task of System Safety Technol- ogy has served the commercial aviation training com- munity for many years, producing excellent research work that could occur nowhere else. The current scope of activities is excellent; however, without a long-term core research plan, the projects seem disjointed. Sim~- larly, the individual subtasks in Crew Training are well planned but do not amount to a core training research program. Training research is inherently a long-term activity. Given the inability to go beyond the 5-year horizon for NASA program planning, the researchers in this task have tried to build longer-term research into the current 5-year plan; for example, they have devel- oped an anchor procedure for solving issues relating to flaps and auxiliary power units. In general, the committee found this research ef- fort to be productive and of high quality, with several activities in this task judged to be outstanding. The re- search in distributed team performance is clearly state of the art and is vital in developing flight as well as maintenance training programs. This group has also developed a number of high-quality training tools that have been distributed to the aviation community, par- ticularly the tool known as "How to Train Automa- tion." It is clear that core competencies within this group must be preserved. There is significant interaction and trust between this group and the aviation community operations and training personnel as well as labor unions. The ALPA training council had a meeting at NASA Ames in March 2003. Boeing will be at NASA Ames to review its internal research and development with NASA. These links keep NASA honest and enhance transition to industry. Finding: Outreach to Community. The Crew Train- ing task's already excellent user connection could be enhanced by greater interaction with entities outside the NASA aviation community, including high-level training decision makers at the officer level of major airlines and general aviation companies. Users could

98 be more involved than they currently seem to be in setting goals for NASA training research. Finding: Use of Milestones and Feedback. Clear metrics for understanding the impacts of NASA- developed training materials were not apparent. Recommendation: Use of Milestones and Feedback. NASA should institute a crew training quality as- surance program complete with feedback tools that measure adherence to goals and objectives, exit cri- teria, and status in regard to similar research being performed throughout the world. Or '' - ' . . .;' . ., ~ . . , ~ AN ASSESSMENT OF NASA 'S AERONAUTICS TECHNOLOGY PROGRAMS The committee identified several areas in which the training work could be expanded to have additional impact within the aviation community. In particular, there are needs and opportunities for research on main- tenance training that could be addressed in addition to flight crew training. Rotary wing crew training could also benefit from the research expertise gained through this task. The committee felt that some of the projects, such as research on the effects of low blood sugar on safety, would fit better in other venues like FAA's Civil Aero- space Medical Institute, which already has a long- stancling program in this area. Additionally, some of the research results (such as "How to Train Automa- tion") might better be disseminated through the FAA to avoid potential or perceived conflicts between regu- lator expectations and a respected research body such as NASA. Program Human Factors Task The goal of the Program Human Factors task is to identify crosscutting issues in human factors within the AvSP and to make specific human factors recommen- dations to other projects within the program. The effort is funded at $500,000 for both FY03 and FY04, in net dollars. The cockpit integration of the various and dispar- ate tasks of the aviation safety technologies is impor- tant and should be continuously and thoroughly ad- dressed. The Program Human Factors task at NASA Ames is designed to cut across multiple subprojects, in- cluding Synthetic Vision Systems, Weather Accident Prevention, and Single Aircraft Accident Prevention. Each of these subprojects is to perform its own internal human factors research. However, it appears that many key researchers in human factors are affiliated with Ames, making this an appropriate group to evaluate the overall safety program from a human factors perspec- tive. The group has completed a crosscutting look at is- sues arising as a function of humans interacting with synthetic vision. The study revealed that off-nominal procedures were weak; the technology was built, but procedures were poorly developed. This was the only work looking at full integration of synthetic vision with other existing and emerging technologies. This is a critical, real-world issue being addressed by no one else. The committee noted that the objectives of this task seem to have diminished over time, with corre- sponding reductions in allocated resources. Coupled with the 5-year life expectancy of research projects, the end result is that the plan to carry out the program seemed somewhat fragmented. The committee noted that there is only a single in-house researcher; all oth- ers come from outside contractors and academia. This threatens the future of the human factors core compe- tencies at Ames that are so essential to long-term re- search. Finding: Acceptance of Program Human Factors. The other projects within the Aviation Safety Pro- gram may be unresponsive to the recommendations of the Program Human Factors task. Recommendation: Acceptance of Program Human Factors. NASA management should foster greater accountability for the findings of the Program Hu- man Factors research and findings to ensure coop- eration within NASA so that human factors issues identified in Synthetic Vision Systems, Single Air- craft Accident Prevention, and Accident Mitigation are well considered by and integrated into all ap- propriate projects. The program is somewhat disconnected from the users. As with most of the Ames programs, the poten- tial users are quite broadly defined and interaction with users is not sufficiently documented. Human factors engineering has to be assertive to make clear its rel- evance, and thus the committee encourages coopera- tion and outreach with both industry and NASA Lan- gley and broad dissemination of research results. NASA should benchmark against similar external work, such as military projects like those at the De- fense Advanced Research Projects Agency, Big Pic-

"at ' '; ; 1 ASSESSMENT OF THE AVIATION SAFETY PROGRAM sure, and Quiet Knight and in forums such as the FAA and the Society of Automotive Engineers and leverage the results of that work. NASA researchers should present results of their work at the Institute of Electri- cal and Electronics Engineers ant! the American Insti- tute of Aeronautics and Astronautics to improve out- reach to potential users. Finding: Human Factors for Commercial Carriers. The research in this area only considers commer- · ~ · ~ cla1 alr carriers. Recommendation: Human Factors for General Aviation and Rotorcraft. NASA's Program Human Factors research should add general aviation and rotorcraft to its work on human factors, as it could have great impact in these areas. Aviation System Monitoring and Modeling Subproject The ASMM subproject develops technologies to view aviation safety from a systemwide perspective, develops metrics for the safety of the NAS, and pre- dicts systemwide effects of changes to the NAS. The subproject is composed of four tasks: Data Analysis Tool Development, Extramural Monitoring, Modeling and Simulations, and Intramural Monitoring. ASMM has $8.4 million in net funding for FY03 and $8.6 mil- lion in FY04. Data Analysis Tool Development Task The Data Analysis Tool Development task ana- lyzes both digital and textual data. This work tends to be low-TRL. The task develops concepts that are then instantiated in some of the AS MM modeling efforts. This task emphasizes tool design and development over modeling. Currently, the task focuses on two major ar- eas: digital data analysis tools and textual data analysis tools. The first set of tools, a system known as the Profiler, takes digital data from a system like the Avia- tion Performance Measuring System to generate and evaluate flight signatures. From these signatures, the researchers produce a list of atypical flights, identify the atypical parameters, and summarize the results. The second set of tools, known as PLADS (which stands for the steps in the preprocessing: Phrase ID, Leave, Augment, Delete, Substitute) and the Automatic Lan- guage Analysis Navigator (ALAN), preprocesses and 99 processes the kind of text data that would be found in the Aviation System Reporting System. The goal is to identify atypical situations without any a priori infor- mation merely by sifting through the flight data. This task is funded at $1.7 million in FY03 and FY04. The committee was impressed with the work of the contractors and their knowledge of analytical methods. However, it was concerned that the expertise for devel- oping this system is contractor-based and is not part of the NASA Ames knowledge base. The committee was generally impressed with the Profiler work and its abil- ity to identify atypical parameters from signatures. The committee also found the statistical methods used to be sound. In the text area, NASA does not seem to have le- veraged existing software in use by the Securities and Exchange Commission, the Defense Advanced Re- search Projects Agency, and the intelligence commu- nity. Data mining in the textual domain is a widely stud- ied problem, and the committee suggests that the researchers build on existing methodologies. In addi- tion, the text data research work should be dissemi- nated and benchmarked at major text search venues such as the Text Retrieval Evaluation Conference spon- sored by the National Institute of Standards and Tech- nology. The committee was encouraged to see collabora- tion with Office National cl'Etudes et de Recherches Aerospatiales, the French research agency. Such col- laboration should be extended further to other foreign agencies to assure quality benchmarking, including the Japan Aerospace Exploration Agency, the Depart- amento do Aviaco Civil (Brazil), the National Aero- space Laboratory (Netherlands), the State Research Institute of Aviation Systems (Russia), and the Defence Science and Technology Laboratory (United King- dom). Finding: Use of Milestones. The intended path to technology maturation for these data mining tools was not clear. In particular, it was unclear how data mining research was divided among the low-TRL too! development work in this task, the work on data mining applications taking place in the Extramural Monitoring task, and the work on Aviation Perfor- mance Measuring System analysis in the Intramu- ral Monitoring task. Recommendation: Use of Milestones. NASA should define clear goals and objectives, exit criteria, and a

100 set of milestones for technology transfer or for the next level of development. Extramural Monitoring Task . . . ~ . - ,. . I. . - ,. . ~ Am ASSESSMENT OF NASA 'S AERONA UTICS TECHNOLOGY PROGRAMS The Extramural Monitoring task strives to create a database of information to serve as the repository of aviation safety events and trends and the basis for avia- tion safety decision making. In particular, the task works with two databases: the National Aviation Op- erations Monitoring Service (NAOMS) and Aviation Safety Reporting System (ASRS), with most of the task investment in NAOMS. The overall funding for Extra- mural Monitoring is $2.0 million. NAOMS consists of a longitudinal survey of air- craft operators, gathering information about safety-re- lated experiences of pilots, cabin crews, and mainte- nance operators for both general aviation and air carriers. NAOMS is a random survey in which staff proactively question active operators in a telephone call. It provides statistically reliable results about the frequency of occurrence of safety-related incidents. In contrast, the ASRS is a joint FAA-NASA re- porting system that asks for the voluntary participation of operators who have experienced a safety-related problem. ASRS is funded by the FAA, although NASA administers the program. To encourage submissions to ASRS, NASA makes sure that the reporter remains anonymous. The FAA had agreed that an ASRS report cannot be used as evi- dence to substantiate an alleged violation in an enforce- ment action.~4 Only a small portion ($250,000 of $2.0 million) of the Extramural Monitoring budget supports ASRS-re- lated activities. That portion of the budget addresses data mining techniques applied to the ASRS database. The NAOMS approach is built on research and implementation of national surveys such as those of the Bureau of Labor Statistics. The NAOMS sampling methods have been grounded in sound interview poll- ing science; however, the interviews are conducted by professional pollsters, not aviation experts. The com- m~ttee has some concern about the level of accuracy attained by pollsters who have no expertise in the area in which they are conducting the telephone interview. The committee is also concerned about potential Remark Blazy, 1999. "We all know about ASRS, but what's an ASRP?" FAAviation News Magazine, October. redundancy between NAOMS data and data available from the air carriers or through the ASRS database. The NAOMS project seems to be developing a meth- odology to establish trends in aviation safety perfor- mance that are already available through other sources within the industry and government. For example, NAOMS appears to duplicate what many airlines are already doing both voluntarily and in FAA-mandated programs to track trends for example, in engine shutdowns. The NAOMS program may become more useful when applied to the general aviation commu- nity, however. NASA's decision to collect its own primary data in this case should rest on the type of research NASA wants to perform and whether that research can be supported by information obtained from the airlines. At this point, the committee does not see a compelling argument for independent data collection. Greater interaction with the Air Transport Association and the airlines might help to clarify the usefulness of this effort. The ASRS program has been around for many years. It is highly trusted by the pilot community and is growing in acceptance by the maintenance technician group. Because the program provides lim- ited immunity from certificate action by the regula- tor for errors by pilots, mechanics, and dispatchers (not willful acts), some tasks within the regulatory community resent the program, while others within the research community disparage its value because the inputs are voluntary. In truth, the threat of a cer- tificate action strongly encourages the submission of an ASRS. Unfortunately, the ASRS program is currently resourced to input less than 25 percent of the reports received into the database. Direct follow- up for additional information from the reporting par- ties can rarely be accomplished. Significantly greater volumes of data are anticipated from emerging Air- plane Safety Action Partnership (ASAP) programs, with no anticipated increase in research resources. This could create a serious shortfall in data available to researchers. While the committee is aware that the funding for the database collection work is pro- videcl by the FAA, not NASA, NASA is still respon- sible for maintaining the ASRS program. The com- mittee finds the defined ASRS activity for NASA to be much larger than its resource allocation; one or the other requires modification. It is important that when gaps in the ASRS data occur, phone calls should be made to fill in what is missing. The lack of resources to handle ASRS in a

ASSESSMENT OF THE AVIATION SAFETY PROGRAM statistically sound manner is a significant issue in un- derstanding the safety trends in the NAS. There are many opportunities to accomplish more research with the data available through the ASRS sys- tem. It was not clear if there were plans in this research task to optimize the joint use of ASRS and NAOMS. Finding: Aviation Safety Reporting System. Regret- tably, the Aviation Safety Reporting System data- base is only inputting about 25 percent of the sub- mitted reports. Interviews to follow up on Aviation Safety Reporting System submissions are very lim- ited owing to the lack of resources. The industry believes that the Aviation Safety Reporting System database has been underutilized for some time. The National Aviation Operations Monitoring Service is consuming the majority of the resources in this project area. Recommendation: Aviation Safety Reporting Sys- tem. NASA should combine the National Aviation Operations Monitoring Service methodology and resources with the Aviation Safety Reporting Sys- tem program data to identify aviation safety trends. Modeling and Simulations Task The Modeling and Simulations task seeks to incor- porate human performance models into an analysis of systemwide operations to identify safety-related char- acteristics and predict system response to safety inter- ventions. This program is not responsible for model development, but it incorporates models from other research efforts (such as the AirMIDAS mode! devel- oped in the SWAP program) into a larger, systems- level approach. The task is funded at $1.5 million in FY03 and $1.6 million in FY04. The committee applauds NASA's efforts to inte- grate the various performance models with models of the aircraft and air traffic control systems. This is bold and difficult work and is the kind of research in which NASA should be engaged. The TRL is low, but that is a quality of long-range research that can only be ac- complished by NASA. The weaknesses of the program seem to be a lack of interconnectivity and integration of tools as well as a limited ability to include issues such as clear air turbulence effects on traffic conflict and quality of performance. There is also no collabora- tion between the program and other programs that model environmental safety and noise. 101 The Reconfigurable Flight Simulator and Object Based Event Scenario Trees modeling programs are not tied to NAS models built using the FAA Consolidated Operations and Delay Analysis System and Aviation System Performance Metrics. Nor was there a tie to the Total Airspace and Airport Modeler, which has been validated by Eurocontrol, or the Traffic Organization and Perturbation Analyzer model (developed by the National Aerospace Laboratory in the Netherlands), which has been used to estimate the safety-capacity relationship that may be affected by airports at high operational workloads. Finding: Outreach to the Modeling Community. The modeling programs in this area have excellent potential but appear to lack coordination with other similar modeling programs. Recommendation: Outreach to the Modeling Com- munity. This task should benchmark its perfor- mance against other modeling implementation ef- forts and consolidate programs where possible to achieve a master system performance, capacity, and safety model. Intramural Monitoring Task Intramural Monitoring refers to internal quality assurance and safety functions within each air carrier and air traffic management organization. The Intramu- ral Monitoring products are the Aviation Performance Measuring System (APMS) and the Performance Data Analysis and Reporting System (PDARS). The APMS project is designed as a tool for analyzing aircraft flight data. APMS provides envelope data for each flight pa- rameter in typical flights, provides information about atypical flights, and provides descriptive statistics on phase-of-flight performance. PDARS is designed to collect, process, and analyze air traffic management data. It generates daily reports, shares data among fa- cilities, supports exploratory and causal analysis, and archives data for developing baselines. Its major strength is in the seamless integration of data from multiple sources. The overall task emphasis is on safety risk management. The task received $3.18 million in FY03 and expects $3.25 million in FY04. Some committee members worried that APMS and PDARS were not novel. The committee believes com- peting and sometimes superior systems are already used by airlines.

102 l .. . . .. ... AL ASSESSMENT OF NaSA 'S AERONAUTICS TECHNOLOGY PROGRAMS Overall, the APMS program has been in the refine- ment stage for several years. A target and milestones for technology transfer or the next level of develop- ment were not clear. Benchmarking of APMS against similar programs in other government arenas and academia seems to be lacking. The APMS tools to mine anomalous data are en- tirely appropriate and useful for airline flight opera- tional quality assurance (FOQA) programs. However, there are significant barriers—among them litigation issues to centralizing a general FOQA database at a government agency in the near term. This creates bar~- ers to close interaction with the industry. To make this array of activities more complete, emphasis and resources in this program need to shift further to integrating APMS and other complementary, commercially available FOQA software into an inte- grated operational efficiency and risk model. Finding: Aviation Performance Modeling System. The APMS software is mature in its development and is ready for the off-ramp to the marketplace. Recommendation: Aviation Performance Modeling System. NASA should redirect the APMS resources to pursue integrated data risk model research. The weather overlay work is a clear example of the kind of research that needs to be emphasized. Finding: Performance Data Analysis and Report- ing System. As a safety analysis tool, PDARS was well designed and is being utilized extensively by air traffic control management. PDARS is useful for airspace design, but it is at a fairly high TRL and is ready to be turned over to industry. The committee identified only one remaining gap in the research activity- data source integration. Recommendation: Performance Data Analysis and Reporting System. NASA PDARS resources should be used to integrate PDARS data with traffic and weather information to feed NASA's modeling and simulation activities. In addition, methods to inte- grate the Flight Operations Quality Assurance (FOQA) program and Airlines Safety Action Part- nership and Aviation Safety Reporting System in- formation into the higher level models should be developed.

Appendixes 1 . .

Next: Appendix A: Committee and Panel Member Biographies »
Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs Get This Book
×
Buy Paperback | $48.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The National Research Council (NRC) of the National Academies was asked by NASA and the Office of Management and Budget to perform an assessment of NASA's Aerospace Technology Enterprise. The first such review, which began in early 2002, examined Pioneering Revolutionary Technology (now known as Mission and Science Measurement Technology). The assessment presented here, of the Aeronautics Technology Programs, began in early 2003 and is the second in the review series.

The Aeronautics Technology Programs has three components: the Vehicle Systems Program, the Airspace Systems Program, and the Aviation Safety Program. To conduct this review, the NRC established three panels, one for each of the component programs. The NRC also established a parent committee, consisting of the chairman and a subset of members from each panel. The committee and panels comprised a cross-section of experts from industry, academia, and government and included senior-level managers and researchers in the aeronautics field. Biographical information on the committee and panel members is found in Appendix A.

Review of NASA's Aerospace Technology Enterprise: An Assessment of NASA's Aeronautics Technology Programs contains the committee's assessment of the Aeronautics Technology Programs. Chapter 1 presents a top-level assessment, and Chapters 2 through 4 provide the assessments of the Vehicle Systems Program, the Airspace Systems Program, and the Aviation Safety Program, respectively.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!