Click for next page ( 72


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 71
l - 4 Assessment of the Aviation Safety Program BACKGROUND Program Information The Aviation Safety Program (AvSP) is one of three programs in the Aeronautics Technology Pro- grams of NASA's Aerospace Technology Enterprise. AvSP was created in 2000 as an outcome of a formal process initiated by NASA to develop a research in- vestment strategy in the area of aviation safety. The goal of the AvSP is to protect air travelers and the public. Its research and development strategy is to increase safety by three primary methods: Aviation system modeling. Identify and correct problems using aviation system-level data, Accident prevention. Identify interventions and develop technologies to eliminate recurring types of accidents, and Accident mtigation. Reduce injury and decrease fatalities in survivable accidents. These methods are applied in the three major re- search and development components: iG. Finelli, NASA Langley, "NASA Aviation Safety Program Overview," presentation to panel, February 2003. 71 Vehicle Safety Technology, which includes Single Aircraft Accident Prevention, Accident Mitigation, and Synthetic Vision Systems, Weather Safety Technology, which includes Aircraft Icing and Weather Accident Preven- tion, and System Safety Technology, which includes Systemwide Accident Prevention, Search and Rescue,2 and Aviation System Monitoring and Modeling. A fourth research component, security research, will be added in FY04. The committee did not evaluate this component since no research and development work is currently under way. The AvSP also has an effort in Technical Integration, which is separate from the three research projects. Research and development for AvSP is performed at NASA Langley Research Center, NASA Glenn Re- search Center, NASA Ames Research Center, and NASA Dryden Flight Research Center, with the pro- gram headquarters at Langley. A program organization chart is shown in Figure 4-1. AvSP was funded at $156.2 million in FY03 under 2Search and Rescue is funded through AvSP but is implemented through the Office of Space Flight. Since all programmatic devel- opment and all technical research are performed under the Office of Space Flight, the Aviation Safety Panel did not review this work.

OCR for page 71
72 AN ASSESSMENT OF NASA 'S AERONAUTICS TECHNOLOGY PROGRAMS I _ l Technical Integration ~ Aviation Safety Program ~ Effort // \~ System Safety Technology Project A: Vehicle Safety Technology Project Synthetic Vision Systems 1 ~ Commercial and Business Aircraft Single Accident Aircraft Mitigation Accident Prevention I ~ r Vet ~ Prevention Flight Critical System Design Propulsion System Safety Technologies Control Upset Prevention and Recover General Aviation Enabling Technologies , _ Weather Safety | Technology Project Aircraft Weather Icing Accident Prevention Design and Analysis Tools Aircraft Ice Protection Education and Training Aviation Weather Information Weather Information Comm. Turbulence Prediction and Warning Systems FIGURE 4-1 Aviation Safety Program organization chart. the full-cost accounting scheme.3 Vehicle Safety Tech- nology accounted for $83.9 million (54 percent of the AvSP total), Weather Safety Technology accounted for $31.6 million (20 percent of the total), and System Safety accounted for $40.7 million (26 percent of the total). NASA is in the process of transitioning to full- cost accounting from a net accounting scheme; previ- ously, NASA managers assessed their budgets by the amount of funding available to them for contracts, grants, and other types of procurements. Uncler the net accounting scheme, Vehicle Safety Technology is bud- geted at $19.8 million, Weather Safety Technology at $14.7 million, and System Safety Technology at $18.4 million. In this report, specific subprojects and tasks are discussed in net dollars only, as this was the only information provided to the committee. The net budget breakdowns by subproject are shown in Table 4-1. 3Full-cost accounting encompasses all costs, including research and program management; institutional infrastructure costs, such as research operations support; direct procurements; direct civil service workforce, benefits, arid Gavel; service pools; center gen- eral and adrninis~ative (G&A); and corporate G&A. _ Systemwide Aviation Accident System Prevention Monitoring and Modeling 1 1 ,- 1 Human Performance Models Maintenance Human Factors Crew Training Program Human Factors Data Analysis Tool Development Extramural Monitoring Modeling and Simulations Intramural Monitoring Like other NASA programs, each AvSP project has a 5-year lifespan. This does not imply that the program ceases to exist after 5 years, however. Project plans are reevaluated after each 5-year time period to phase in new projects that build upon previous research and de- velopment. Review Process The Aviation Safety Panel was formed in Decem- ber 2002 as one of three panels that would review NASA's Aeronautics Technology Program. The Avia- tion Safety Panel met for the first time on February 26- 28, 2003, in Washington, D.C. At this first meeting, the 10-person panel received technical briefings from the program and project managers in AvSP on the over- al1 program, specific projects, and individual tasks. After the first meeting, panel members participated in site visits to each of the relevant NASA facilities (NASA Langley, NASA Glenn, and NASA Ames). The purpose of the site visits was to obtain a deeper understanding of the research and development in the program, to speak directly with the principal investiga- tors for each project task, and to observe the products

OCR for page 71
ASSESSMENT OF THE AVIATION SAFETY PROGRAM TABLE 4-1 Net Budget for the Aviation Safety Program Budget (million $) ProjeetlSubproject Name FY03 FY04 Vehicle Safety Technology Single Aircraft Accident Prevention Accident Mitigation Synthetic Vision Systems Weather Safety Technology Aircraft Icing Weather Accident Prevention System Safety Teehnologya System-Wide Accident Prevention Aviation System Monitoring and Modeling 19.8 4.7 8.4 9.8 0.4 2.2 7.2 5.0 9.7 s.o 8.4 13.9 17.7 0.2 2.6 7.0 s.o 8.9 5.1 8.6 aSystem Safety includes Search and Rescue, which is not reviewed here. SOURCE: Information provided to the NRC panel by G. Bond, Aviation Safety Program Office, NASA Langley Research Center. and facilities firsthand. The site visits are listed in Ap- pendix C. Panel members visited on-site or spoke via teleconference with NASA personnel from every AvSP task. Panel members, who were experts in their fields, also reviewed technical reports and journal articles and followed up with individual principal investigators by means of teleconference calls and written questions. Before the first meeting, the NRC asked each prin- cipal investigator to complete a short questionnaire with 12 questions relating to the research and develop- ment goals, products, roadblocks, users, and technical outcomes. A blank questionnaire is shown in Appen- dix D. The completed questionnaires were distributed to the panel for review prior to the first panel meeting. Thus, the panel members were already somewhat fa- miliar with the programs and projects under review before they were briefed in person by the NASA re- searchers and program managers. The questionnaires proved to be a valuable tool for the panel in performing its program assessment. Upon completion of the three site visits, the panel met for a second time, again in Washington, D.C., on May 27-29, 2003, to come to consensus on findings and recommendations for the program. The panel dis- 73 cussed outstanding questions and issues of concern with program staff from NASA. It also developed crosscutting observations across the different projects and tasks within AvSP. The panel then provided its input to the Aeronautics Technology Programs parent committee in the form of working documents. Four of the ten panel members represented the panel on the committee. PORTFOLIO The committee evaluated the appropriateness of the AvSP research portfolio based on the amount of basic research versus user-driven research; the presence of gaps or incomplete areas of research; the balance be- tween high-risk, high-payoff research and more evolu- tionary work; and whether or not the portfolio ad- dresses real-world problems. The committee is concerned about the balance be- tween fundamental and product-driven research in the Aviation Safety Program. It observed a shift away from essential basic research over recent years. Such basic research is necessary for the development of future safety products that will enable the AvSP to reduce

OCR for page 71
74 ': 1 AN ASSESSMENT OF NASA 'S. AERONA UTlCS TECHNOLOGY PROGRAMS accident rates. In some instances, the committee ob- served ineffective work-arounds created out of neces- sity to divert resources from funded, low-payback projects to accomplish unfunded but critical basic re- search. Furthermore, with a few notable exceptions (such as the Aircraft Icing subproject and the Modeling and Simulations task in the Aviation System Monitor- ing and Modeling subproject), the committee felt that this problem was widespread within the program. The committee found examples of research that is essentially complete and ready for transition (such as Fault Tolerant Modular Architectures-, Personal Elec- tronic Device electromagnetic susceptibility, virtual and augmented reality for maintenance crews, and the Performance Data Analysis and Reporting System). The committee also found places where basic research was lacking for example, high-temperature materi- als for engines, weather display interfaces, turbulence warning systems, and human factors work in many ar- eas. The committee's findings and recommendations regarding specific instances where research is too prod- uct-driven or where additional basic research is needed are presented in the discussion of each task. Finding: Support for Basic Research. There has been a shift away from essential basic research in the Aviation Safety Program in recent years. Program Recommendation: Support for Basic Re- search. The Aviation Safety Program should rein- state a core competency program dedicated to basic research that is essentially unencumbered by short- term, highly specified goals. Without a strong basic research program, the more applied research even- tually suffers from a lack of good ideas and trained personnel. The criterion for starting or restarting such an activity within a Center is that a need must exist for knowledge that is not now available. The committee noted specific gaps in the portfolio at the subproject and task levels in subsequent sections. It found one significant program-wide omission in the research portfolio: rotorcraft. Finding: Rotorcraft. Rotorcraft safety can be im- proved with additional research in the areas of de- cision aids, synthetic vision, training, workload, and situational awareness. Program Recommendation: Rotorcraft. The Avia- tion Safety Program should reincorporate rotor- craft research into its program. The research should consider the most effective approaches for reducing the workload of rotorcraft pilots and improving their ability to conduct safe, low-speed, low-altitude rotorcraft operations in obstacle-rich environments and in adverse weather. PROGRAM PLAN The AvSP program plan emerged from a series of strategic planning sessions on aviation safety in 1997 known as the Aviation Safety Investment Strategy Team (ASIST). ASIST established a vision and priori- tized the research and development investment areas for the AvSP. The AvSP approach includes system modeling, accident mitigation, and accident prevention, with an emphasis on mitigating problems that contrib- ute most heavily to accident and fatality rates. The AvSP was established in 2000 with a 5-year program plan. Each individual task within the AvSP is structured to last 5 years. This 5-year programming cycle is more suitable for a product-oriented program. It is difficult, if not impossible, for NASA to maintain core compe- tencies with these 5-year programs. In addition, there do not appear to be sufficient off-ramps to transition research that has been completed before the 5-year time window closes. Finding: Use of Sunset Requirements. NASA func- tions on a 5-year schedule to the detriment of solid research. Program Recommendation: Use of Sunset Require- ments. The Aviation Safety Program should struc- ture its program based on the natural duration of each research effort and not compel conformity to a 5-year cycle for every task. NASA should eliminate arbitrary time constraints on program completion and schedule key milestones based on technology maturity, task complexity, and resource limitations. Research involving the human-machine interaction and causes of human error should be a major focus of any aviation safety research program. The AvSP con- tains a wide array of human factors research, from syn- thetic vision displays to aviation weather information requirements to tools for aircraft maintenance teams. In general, the committee found evidence of high qual-

OCR for page 71
ASSESSMENT OF THE AVIATION SAFETY PROGRAM , .j ity in all of NASA's human factors research; however, it also found that the work was not always well inte- grated into a cohesive program without overlaps. The committee approves of the efforts of the Aviation Safety Program Office in pulling together some of the disparate human factors tasks through cross-center meetings and through the Program Human Factors task of the Systemwide Accident Prevention subproject. However, the committee did not observe any improve- . . . . ment In the ~ntertask communication or any synergy from the human factors research within the program. Aviation accident data make clear that human er- ror is a much greater factor than hardware or software failure or environmental conditions. Ideally, every technology effort should be examined from a human factors perspective at an early design phase to antici- pate problems. However, the advice of human factors professionals, who must necessarily draw on the softer behavioral sciences, is often disregarded by the engi- neering designers, who view it as negative or irrelevant. NASA has traditionally supported research in human factors, and the human factors group at NASA Ames has truly been a national resource. Finding: Human Factors Research. In recent years the Aviation Safety Program's work in human fac- tors has been eroding; senior in-house research staff have left, and in order to get the work done, more human factors professionals have found themselves managing contractors, a task for which they often are not well qualified. Crosscutting efforts to inte- grate human factors have also suffered. Program Recommendation: Human Factors Re- search. Critical human factors expertise should be better supported in order to maintain critical mass, to foster basic research in this field, to identify gaps in our understanding of safety, and to be available to consult with various NASA projects. Program Recommendation: Early Analysis of Hu- man Factors. Project requirements should include requirements for human factors analysis early in the design phase. The committee found that the considerable layers of both line management and project management ob- scure the lines of accountability in AvSP. In at least one case, a person's subordinate in the research project hierarchy is his or her superior in the line staff hierar- 75 _ chy. The committee felt that subproject- and task-level plans, goals, metrics, and responsibility could not be clearly traced back to an overarching plan and vision for the AvSP. In other words, the planning appeared to be more bottom-up than top-down. Additionally, the committee heard from a number of technical civil ser- vants in the program that too much of their time was spent "doing management" (e.g., making PowerPoint slides) and not enough doing science and technology. In addition, it was not clear to the committee what methods and metrics NASA uses to evaluate objec- tively the status of its research projects against its own stated goals. The program effort in Technical Integra- tion (described in a subsequent section) would be a natural place for such an evaluation. Finding: Management Structure. The organiza- tional structure is unnecessarily complex, making it difficult to trace lines of responsibility. Subproject- and task-level plans, goals, metrics, and responsi- bility could not be clearly traced back to an overarching plan and vision for the Aviation Safety Program. Program Recommendation: Management Struc- ture. NASA should articulate a clear, long-range plan for the Aviation Safety Program and a hierar- chy of goals, and it should adopt a less complex management system that enables program account- ability and implementation to be clearly traced. The committee suggests that NASA reexamine its names for the AvSP activities (many terms sound like they overlap or are ambiguous) so that the goals of each major project are easily understood. This ambiguity is particularly evident in the Single Aircraft Accident Pre- . . venhon su project. TECHNICAL PERFORMANCE The committee asked a variety of questions to as- sess the technical quality of the work, to evaluate the facilities and personnel, to find evidence of system- level assessments, and to determine the balance be- tween experimental and theoretical work. The commit- tee also compared the quality of the AvSP work relative to that of other work in industry, academia, and gov- ernment, including international work. The committee found the technical quality of the AvSP to be very good. In some cases, particularly in

OCR for page 71
76 specific parts of the weather work, NASA personnel can be considered among the world leaders in their re- spective fields. The review committee found the facili- ties to be adequate for achieving the research goals; in some cases (such as the icing wind tunnel), the facili- ties can be considered true national assets. The committee identified several specific tasks and subtasks that have achieved an outstanding level of technical achievement: . i AN ASSESSMENT OF NASA 'S AERONA UTICS TECHNOLOGY PROGRAMS by NASA that are similar to or have considerable over- lap with products developed by industry. (Specific ex- amples will be discussed in the task-specific sections.) Finding: Redundancy with Industry. Some prod- ucts being developed by NASA are similar to or have considerable overlap with products already developed by industry. Program Recommendation: Benchmarking Against Industry. The Aviation Safety Program should com- pare (benchmark) its research projects against those of other research and development entities in government and industry to ensure that NASA's work is leading. If it is not, NASA should terminate the work. . Structures health management subtask of the Vehicle Health Management and Flight Criti- cal System Design task in the Single Aircraft Accident Prevention subproject, Mode confusion subtask of the Vehicle Health Management and Flight Critical System Design task in the Single Aircraft Accident Prevention subproject, Scale model development and testing work in the Single Aircraft Accident Prevention sub- proJect, Design and Analysis Tools task in the Aircraft Icing subproject, Aircraft Ice Protection task in the Aircraft Icing subproject, and Crew Training task in the Systemwide Acci- dent Prevention subproject. A number of outstanding products have been de- veloped, but many of these (an example being the Per- formance Data Analysis and Reporting System (PDARS) trajectory monitoring tool) are ready for handoff to industry. Much of the low-TRL research is excellent, but its relevance and potential usefulness seem not to have been made clear to potential users (a good example is human performance models). USER CONNECTIONS User connectivity was evaluated in two separate ways. First, the committee asked how well NASA per- sonnel reflect and leverage work being conducted else- where and how well NASA research results are ac- cepted and adopted by the outside community. Second, the committee asked how the research itself is con- ducted --- for example, if it uses an appropriate mix of internal and external personnel. In comparing the work of the Aviation Safety Pro- gram with other work in the community, the committee found several instances of products being developed Exploring the second aspect of user connectivity (how well the program uses expertise from the outside community), the committee found that the answer var- ied from task to task. In some cases (particularly the Vehicle Safety Technology project), the committee felt the project would benefit from additional involvement with the outside community. In particular, the commit- tee believes NASA's fundamental research projects would benefit from increased university participation. In other cases (especially in the System Safety Tech- nology project), the committee felt there were too few in-house personnel and that too much of the research was being conducted by contractors. This tends to weaken the core competencies of NASA. ASSESSMENT BY PROJECT Technical Integration Pro jest The AvSP has an effort in Technical Integration, which is designed to provide program assessments, develop systems-level implementation strategies, and integrate research and development efforts across pro- gram tasks, particularly in flight testing. The committee believes the concept behind the Technical Integration project is very important, provided it plays a significant role in deciding what research to undertake and when such research should be modified, transitioned to industry, or discontinued. The commit- tee understands that because the Technical Integration project began after the current 5-year plan had begun, it has been playing catch-up with regard to its status in the overall program. However, the committee had difficulty

OCR for page 71
ASSESSMENT OF THE A VIA TlON SAFETY PROGRAM , . ~ determining the effect of the Technical Integration ac- tivities on current planning. The Technical Integration project seemed to be running almost as an independent activity somewhat disconnected from project manage- ment. The committee also observed that subjective evaluations are being made, mostly by NASA project managers, and it believes that NASA should have more input from customers and industry and from lower-level managers, scientists, and engineers engaged directly in the various efforts. For example, the market penetration of AvSP products should be studied. The Technical Integration effort as currently con- stituted seems best suited for evaluating AvSP's near- term products. However, the committee is concerned about how Technical Integration will integrate project "stovepipes" into a workable whole. For example, there appears to be little integration of NASA Ames human factors activities with the synthetic vision work at NASA Langley. The committee also sees a need for anticipatory or prospective integration of the Human Performance Models task, the Monitoring and Simula- tion task, and the other monitoring tools efforts. Finding: Use of the Technical Integration Project. The Technical Integration effort does not play the role it needs to play in deciding what research to undertake, in performing cost-benefit analyses for projected and ongoing projects, and in deciding when such research should be modified, trans- itioned to industry, or discontinued. Recommendation: Use of Systems Engineering. NASA project managers should employ systems engineering approaches to ensure proper integra- tion of projects. Recommendation: Use of a Quality Assurance Pro- gram. NASA should institute a quality assurance activity, separate and independent from project management, the results of which should be re- ported directly to the Aviation Safety Program manager and to the Aerospace Technology Enter- prise associate administrator. As discussed in the assessment of the Airspace System Program, above, there appear to be significant overlaps in the various system modeling efforts within NASA, and it may be feasible to consolidate or inte- grate some projects. In particular, modeling research by the AvSP should be coordinated with Virtual Air- 77 space Simulation Technologies, which is part of ASP. NASA should also develop and implement a plan for evolving current models, simulations, and analysis tools into large-scale models. The committee applauds the Technical Integration support of the Commercial Aviation Safety Team and the Joint Implementation Measurement Data Analysis team. Vehicle Safety Technology Pro tech Background The Vehicle Safety Technology project is designed to strengthen aircraft against vehicle system and com- ponent failures, loss of control, loss of situational awareness, and postcrash and in-flight fires. The project focuses on applications for the aircraft itself. The majority of the research is conducted at NASA Langley, with a relatively small amount of work in pro- pulsion safety and fire prevention conducted at NASA Glenn. The Vehicle Safety Technology project was funded at $83.9 million (full-cost)/$19.8 million (net) in FY03 and is divided into three subprojects: Single Aircraft Accident Prevention, Accident Mitigation, and Synthetic Vision Systems. In net dollars, Single Air- craft Accident Prevention is funded at $10.4 million, Accident Mitigation at $2.2 million, and Synthetic Vi- sion at $7.2 million. Portfo/io The goals of the project are focused on the vehicle itself, in applications related to the flight deck, flight critical systems, propulsion, and airframe. The research focuses on loss-of-control prevention and recovery; flight critical systems; vehicle health monitoring; pro- pulsion systems safety; fire mitigation, detection, and prevention; and improving low-visibility conditions by providing a synthetic picture of the outside world. This is an ambitious project with many diverse goals, applications, and areas of research expertise. The folding of such diversity into a single project and the integration of the results of each research effort present a considerable challenge. As with all AvSP programs, the projects within the Vehicle Safety Technology project have a 5-year life span. The termination point for the Vehicle Safety Technology Project tasks is scheduled to be 2005, although many of the projects will probably be continued in some form into the next phase of the AvSP.

OCR for page 71
78 The committee found the researchers stretched in many directions in the Vehicle Safety Project and believed it was unlikely that every subtask could achieve its stated goals to an appropriate level of detail by the project ter- mination point in 2005. Further, the fact that the names of many research tasks seemed to be similar suggested that some tasks could be combined. Overall, the committee believes there is an appro- priate balance of low-TRL work with more applica- tion-driven research in the Vehicle Safety Technology project. Across the AvSP as a whole, the committee has some concerns about the increasing trend toward product-driven research and development, so it was pleased to see several fundamental, low-TRL tasks within Vehicle Safety Technology, such as the design work in the Control Upset Prevention and Recovery (CUPR) task. The committee urges a continued em- phasis on this basic research in the next phase of the Aviation Safety Program. At the same time, the com- mittee notes that several tasks for example, some of the fire prevention work and fault-tolerant integrated modular architectures have already attained a high level of technology readiness and should be transitioned to industry. The committee is sensitive to the fact that by fo- cusing on fewer concepts, the project eliminates other worthy research ideas. However, despite recommend- ing that the Vehicle Safety Technology project focus on fewer tasks in greater detail, the committee also found a significant omission in the array of activities in this project namely, rotorcraft. The committee be- lieves that NASA could have a significant impact on rotorcraft safety by including the topic in this project. The committee believes that NASA's decision to ter- minate rotorcraft work is a mistake, as there are a num- ber of real-world problems in rotorcraft safety that ap- parently are not being addressed outside NASA. Program Plan The committee believes that NASA will make sig- nificant impacts if it can mature the technologies in the Vehicle Safety Technology project. However, the com- mittee judges the program plan for technology matura- tion to be overoptimistic. Finding: Portfolio Breadth. Despite the encourag- ing progress reported. to date, the time remaining is insufficient to achieve the goals set forth in the pro- gram plan. The breadth of the work in Vehicle AN ASSESSMENT OF NASA 'S. AERONA UTlCS TECHNOLOGY PROGRAMS Safety Technology is coming at the expense of tech- nical depth. Recommendation: Portfolio Breadth. The Aviation Safety Program should narrow the scope of activi- ties in the Vehicle Safety Technology project to in- crease the depth of research activities and focus them in fewer, more specific, higher-priority areas. A few tasks within the Vehicle Safety Project have already reached a high TRL, and the committee noted that there were no appropriate off-ramps or transitions for those tasks that have reached or will reach comple- tion before the 2005 project end date. Specific instances are noted in the commentary on the individual tasks, below. Technica/ Performance The committee found the individual researchers to be bright, aware of the relevant literature, and able to answer both theoretical and application-related ques- tions. The facilities are state of the art and appropriate for carrying out the project. The committee found evidence of several notewor- thy research tasks within this project that have a high level of technical achievement, such as the structures health management subtask. Several other tasks per- haps should be transitioned because they have already completed their research objectives, such as fault-tol- erant integrated modular architectures and some of the fire mitigation work. In no case did the committee rec- ommend research termination for lack of quality. The committee is concerned about the functional integration of the many diverse activities talking place across the different NASA research centers. NASA should develop software ant! hardware interface specifi- cations that connect the various subsystems early on to aid in the integration and definition of the scope and plans for program research. These specifications can be spiraled into more detail and refined accordingly as the program evaluations progress. They form the basis for integrating the work taking place between the NASA centers and NASA contractors. These interfaces should include interactions between all the vehicle subsystems, including the controls arid display tasks. Finding: Interim Integration Milestones. There ap- pears to be a lack of interim task-level milestones to track the progress of integration activities.

OCR for page 71
ASSESSMENT OF THE AVIATION SAFETY PROGRAM Recommenclation: Interim Integration Milestones. NASA should integrate the information that sys- tems evolving from individual tasks such as Vehicle Health Management and Flight Critical System Design and Control Upset Prevention and Recovery can provide to the flight-deck crew. Recommendation: Interim Integration Milestones. NASA should develop software and hardware in- terface specification documents that address the various subsystems early on to aid in the integra- tion and definition of the scope and plans for pro- gram research. Recommendation: Interim Integration Milestones. NASA should incorporate interim test and evalua- tion milestones for its flight simulation facilities to measure the impact of its design integration on on- ~ . ~ going crew performance act~v~es. User Connections In general, the committee found that the research- ers are collaborating with the appropriate outside agents; by and large, there is the right degree of in- volvement with industry, and the connectivity to the research community is impressive. In a few cases, es- pecially in areas with low-TRL work, the NASA re- search could be augmented with university research. Specific instances are noted below. It appears that uni- versity involvement is relatively minor, notably in for- mal methods of software verification and validation and in some of the propulsion safety technologies. Assessment by Subyroject Single Aircraft Accident Prevention Subproject The Single Aircraft Accident Prevention (SAAP) subproject is designed specifically to develop and implement technologies that enhance aircraft airwor- thiness and resiliency against loss of control while in flight. Again, the work focuses on onboard technolo- gies for the individual vehicle. The subproject contains three tasks: Vehicle Health Management and Flight Critical System Design (VHM and FCSD), Propulsion System Safety Technologies, and Control Upset Pre- vention and Recovery (CUPR). The net budget for SAAP is $10.4 million in FY03, with $4.8 million for 79 VHM and FCSD, $4.1 million for Propulsion Safety Technologies, and $1.5 million for CUPR. NASA's effort to expand and improve industry knowledge of the aerodynamic performance envelopes of transport-category aircraft appears to be on target for reducing the incidence of loss-of-control accidents. This subproject promises to improve the fidelity of flight simulators used as tools for improving pilot performance in manual recovery from extreme attitudes. The research could lead to better avoidance of such conditions as well as to systems that effect automatic recovery. By their nature, many of the modeling and analysis efforts do not have a well-defined end point, and there is always room for improvement. The lack of a clear completion point for some of the SAAP work was nev- ertheless troubling, and the committee believes that NASA should develop ways to "declare victory" and make clear the degree to which the effort has succeeded and the amount of research still needed to achieve suc- cess. For example, the modeling of follower aircraft interaction with wake vortices from lead aircraft is in its infancy because of the complexity of the problem, but it should continue to be pursued in future years. On the other hand, the work in fault-tolerant integrated modular architectures is at a high TRL and ready for transition to industry. Finding: Wake Vortex. While wake vortex interac- tions have an obvious impact on capacity, there are equally important safety considerations, and the AvSP is not sufficiently involved in the wake turbu- lence effort. Recommendation: Wake Vortex. NASA should in- clude wake turbulence interaction models in its Control Upset Prevention and Recovery dynamics modeling and simulation technologies work. Cur- rent models used in airline training simulators are quite crude and provide insufficient fidelity for ef- fective pilot training purposes. The committee was pleased to observe the excite- ment in using the 1/20 scale model 757 for both flight and wind tunnel tests of control upset and other tasks. Such tests could integrate and coordinate the diverse activities in SAAP. The collaboration with other relevant parties (the FAA, DoD, and industry) appears to be excellent in this subproject.

OCR for page 71
so Vehicle Health Management and Flight Critical System Design Task The goal of this task is to research technologies to reduce loss-of-control accidents and system or component failures on the vehicle itself. Even within this single task, there is a large array of activities, from structures health evaluation to software integ- rity evaluation. Research is conducted in the follow- ing areas: Structures health management Flight systems health management Verification of neural networks Mode confusion Software safety Requirements modeling Recoverable computing Modular avionics Electromagnetic susceptibility of avionics Neutron particle effects on flight critical systems Validation methods The TRL also varies widely across the work in this task: Some of the software work is at a relatively low TRL. while some of the structures health monitoring work is near product development stage. The net fund- ing for the Vehicle Health Management and Flight Critical System Design work is $4.8 million for FY03 and is scheduled to be $5.1 million for FY04. The committee found a number of activities in this task worthy of commendation. It was particularly im- pressed by the specific research activity in two areas: structures health management, particularly the fiber- optic strain systems (FOSS), and pilot confusion over automation control-display modes. The structures health management activity is an area that NASA should showcase in the program. It has made significant progress in a relatively short amount of time, and NASA has truly catalyzed breakthroughs in this area. In general, the cost-benefit analysis work done in this area is impressive, and it is clear that NASA knows what it would take to install and Held its systems. The committee found the task to be well thought out in terms of the interaction between corro- sion and other properties of aging materials and the measurement and diagnosis of structural faults. The FOSS work at NASA Langley is interesting with an appropriate blend of fundamental and user-driven re- AN ASSESSMENT OF NASA 'S AERONA UTICS TECHNOLOGY PROGRAMS search. The potential payoff in this area is very high. The mode confusion work also has a very high poten- tial payoff, and the work being performed in this area is novel and of high quality. The committee also encourages NASA's contin- ued involvement in the verification of flight systems, particularly as software becomes more complex and new issues must be addressed, bringing corresponding increases in cost and development time. In addition, NASA should continue to foster the introduction of object-orientecl (OO) programming into the flight criti- cal software area. Flight critical software is software onboard an air vehicle that is used to control the vehicle and whose fail- ure would lead directly to the Toss of that vehicle. Be- cause of the cost of recertification, this is an area in which commercial companies are slow to invest, even though all recognize the eventual payoff. The payoff of GO techniques, while not directly related to safety, comes from reuse, savings in cost and time, and increased effi- ciencies in verification and validation activities. It would be useful if NASA could determine or demonstrate ways to reduce the risks and costs of re- certifying software, and its activity in OO program- ming with the FAA is a good step in that direction. As the committee noted in its subproject discus- sion, this task could benefit from fewer tasks. There is such a broad range of activities within this subproject that the committee found it unlikely they all can be brought to meaningful closure, with an appropriate TRL, by the task's end in 2005. Specifically, the com- mittee believes NASA should reorganize that portion of the SAAP that combines VHM (including the model- based diagnostics of the propulsion arm) and the detec- tion, identification, reconfiguration, and recovery part of CUPR in a single anomaly detection, identification, and reconfigurationlrecovery structure. This would eliminate the appearance of redundant research efforts and further enable functional integration. Finagling: Portfolio Breadth. The Single Aircraft Accident Prevention activities are linked by their common goal (reducing system or component fail- ures on the aircraft) but not necessarily by common expertise or research methodologies. Similar activi- ties appear to be taking place in multiple subtasks. Recommendation: Portfolio Breadth. NASA should restructure or descope this task.

OCR for page 71
l' , ASSESSMENT OF THE AVIATION SAFETY PROGRAM In several specific areas the committee has doubts about the utility of NASA's continued involvement. While the committee understands NASA's desire to offer a complete solution to the Flight Critical System problem, the committee is not convinced that NASA should be working in fault-tolerant integrated modular architectures. Commercial companies are in this busi- ness and competing heavily with one another. The TRL of this technology is well above 6, and NASA is not needed to foster innovation. The second area that the committee questions is in personal electronic device electromagnetic susceptibility. This work. seems more appropriate for industry (i.e., airlines and airframers). The committee understands that part of this work is sponsored by an airline but believes that the effort should have low priority in the NASA research plan. Finding: Modular Architectures and Personal Elec- tronic Devices. The work in fault-tolerant inte- grated modular architectures and personal elec- tronic device electromagnetic susceptibility is at a high TRL and more appropriate for industry. Recommendation: Modular Architectures and Per- sonal Electronic Devices. NASA should terminate its involvement in modular architecture develop- ment and electromagnetic interference activities in order to concentrate resources in other less-re- searched areas of the program. Propulsion System Safety Technologies Task The purpose of the Propulsion System Safety Tech- nologies task is to reduce propulsion system failures as a factor in civil aircraft accidents through the predic- tion, detection, and testing of propulsion system mal- functions and failures. The propulsion system safety team works in system health monitoring, crack-resis- tant blades and disks, and engine containment. This effort is conducted at NASA Glenn and has a net bud- get of $4.1 million per year in FY03 and FY04. The committee found the researchers to be knowl- edgeable and familiar with the relevant work in the outside community. In general, the task was well orga- nized and had a more focused goal and approach than the other tasks in SAAP. The committee found two ar- eas worthy of notice: model-based diagnostics and en- gine sensor technology, particularly the eddy current sensors. Also, the committee found that NASA has played a key role in integrating the various aspects of 81 crack-detection technologies sensors, algorithms, and testing resources. NASA's involvement in model-based diagnostics shows promise for onboard diagnostics and is a worth- while investment, but it could benefit from integration with related subtasks in SAAP. Finding: Integration of Related Activities. The model-based diagnostics subtask is not well inte- grated with related activities in Single Aircraft Ac- cident Prevention. Recommendation: Integration of Related Activities. NASA should integrate model-based diagnostics with the vehicle health monitoring activities in the Vehicle Health Management and Flight Critical System Design task when it plans the future of these tasks. NASA efforts in embedded technologies with eddy current sensors offer good promise in the early detec- tion of faults. Engine companies are also working on these technologies, however. Finding: Eddy Current Sensors. Some of the eddy current sensor work may be redundant with the work by industry. Recommendation: Eddy Current Sensors. NASA should perform additional experimental work and operational testing on these resilient sensors and other sensors under development by the engine companies only if it is leading and not following the ~ engine compames. In general' the work in engine disk crack detection and engine materials research is well integrated and following good experimental practices. The commit- tee believes this work could be enhanced with addi- tional research at high temperatures. Finding: High-Temperature Engine Materials. NASA lacks some basic research activities in alter- native high-temperature engine materials. Recommendation: High-Temperature Engine Ma- terials. NASA should also foster progress into other engine materials and heat-treating technology. This work might benefit from additional university in- volvement.

OCR for page 71
94 AN ASSESSMENT OF NASA 'S AERONAUTICS TECHNOLOGY PROGRAMS that have served the aviation community well for years and there are sure to be many others that show great promise for future applications. Many of the programs are user driven, both internally and externally. Unfor- tunately, the expectation that projects can be completed in 5 years appears to constrain NASA's ability to ar- ticulate a consistent and clear vision for Tong-term core research. It also seems to impact the ability to create well-defined goals that lead to an integrated research program. The goal of providing a general knowledge foundation, as in the case of human performance mod- eling, should be part of a core research program. Bal- ancing the System Safety Technology suite of applied research activities with more basic research would help sustain essential core competencies within the associ- ated groups. A specific long-range goal such as creating a fully integrated virtual National Air Space model by 2020 or 2050 for modeling total system safety and efficiency would be helpful in focusing and balancing research projects. Research initiatives should support such a central long-range goal, and the AvSP should work closely with the VAMS effort in the ASP program to achieve this. Requirements could then be more easily developed, including problem statements, standards, and test procedures. These should be established in a way that encourages innovation while maintaining fo- cus and accountability. Adclitionally, a continuous sys- tems analysis approach would be constructive in iden- tifying research priorities and allocating resources to projects with the greatest impact on safety. Program Plan Many of the safety tasks have articulated very desir- able outcomes, but plans to achieve these outcomes were often unclear or lacked measurable milestones. For ex- ample, a number of outcomes are in the form of a per- centage reduction in accidents or in fatalities. There ap- pears to be no method in place in the research program for evaluating such outcomes or for assessing progress. This gap appeared to be driven by a disconnect between the resources or time required to accomplish the target outcomes and the availability of assets and time. The committee acknowledges that some of this work is low- TRL and difficult to relate directly to measurable changes in accident mitigation. Nevertheless, the com- m~ttee believes that NASA should develop interim m~le- stones and metrics for internally evaluating the success of the System Safety Technology project relative to in- tended project deliverables. This should be done in con- junction with the Technical Integration effort. The process of research project selection, planning, resourcing, programming, and accountability within the matrix management scheme was complex and dif- ficult for the committee to understand. Some program- matic decisions appear sensible from a safety perspec- tive but do not seem to relate to an overall research plan. Technica/ Performance The committee was impressed with the technical capabilities of the NASA Ames staff associates! with the System Safety Technology project. NASA Ames has an excellent reputation for "basic applied" research. The committee encourages NASA to uphold this strong reputation by sustaining basic research programs at Ames, where scientific publication is a core value. The committee is concerned that the balance of in- house and contractor personnel is becoming heavily weighted toward outsourcing to an extent that could compromise the ability to maintain core competencies. Additionally, heavy outsourcing forces scientific per- sonnel to focus on management oversight rather than on building internal scientific activities. This discour- ages young researchers from joining the NASA team or even remaining with NASA. Basic research seems constrained in a number of areas owing to either lack of access to data or lack of resources to process available data. Good examples of this are the highly respected Aviation Safety Reporting System product and the Maintenance Human Factors task. As long as there are barriers to accessing data, basic research could languish. User Connections The committee was impressed with the establish- ment of an integrated FAA/NASA Aviation Safety R&D Plan, an Aviation Safety Working Group, and an Aviation Safety Program Executive Council, all to en- sure greater coordination of FAA/NASA research. Ef- fective use of these groups will be vital to establishing post-2004 safety research goals. In some areas NASA seems to be pursuing tech- nologies or tools that have reached maturity or are complementary to items already available in industry or other government agencies. This is true, for example, with Performance Data Analysis and Reporting Sys-

OCR for page 71
ASSESSMENT OF THE AVIATION SAFETY PROGRAM tern (PDARS), Aviation Performance Measuring Sys- tem (APMS), and the virtual reality maintenance work. It is critical that regular, open, candid product benchmarking and communications occur among NASA, FAA, industry, and other research entities in order to avoid duplication, to ensure that valuable and limited resources are effectively and efficiently allo- cated, and to sustain world-class research standards and products. Assessment by Subproject System wide Accident Prevention Subproject SWAP is the AvSP subproject devoted to human factors and its relationship to error mitigation. The fo- cus of the research is primarily in error modeling, train- ing procedures, and maintenance procedures. SWAP is also responsible for identifying crosscutting issues in human factors that relate to the AvSP as a whole or to other subprojects and tasks under the AvSP purview. SWAP is broken into four tasks: Human Performance Models, Maintenance Human Factors, Crew Training, and Program Human Factors. SWAP is funded at a net value of $5.0 million in FY03 and $5.1 million in FY04. Human Performance Models Task The Human Performance Models task utilizes cog- nition and perception models to detect and analyze hu- man error and to develop tools for system design. The task works primarily with five human performance models: Air Man-Machine Integration Design and Analysis System (AirMIDAS), ACT-R/PM, A-SA, D- OMAR, and IMPRINT/ACT-R. Each model uses a dif- ferent cognitive approach and each has a different ap- plication to sources of pilot error. The Human Performance Models task has also developed a track- ing system, the Crew Activity Tracking System, to pre- dict operator behavior and to interpret operator actions. The Human Performance Models task of SWAP is funded at a net value of $1.5 million in FY03. The activities in this area are appropriately weighted toward fundamental research. The goal is clearly to create state-of-the-art modeling techniques. While resources seem adequate for the stated goals, the 5-year program constraint appears to limit the long- term potential of this core research area. Error analysis appears to focus on error as devia- tion from nominal procedure rather than considering 95 which deviations are dangerous and which are merely alternative but still acceptable ways to accomplish the task. These alternative methods may in some cases be better than the nominal (e.g., under off-normal condi- tions). Expanding the scope of work to include accept- ability analysis may broaden the potential application of this effort. While application of NASA human performance modeling research to other efforts at NASA, such as synthetic vision research, would seem promising, there is little to show as yet. There appears to be no connec- tion with human performance modeling at other gov- ernment agencies. Finding: Collaboration with Other Agencies. There appears to be no substantive interface with human performance modeling at other government agen- cies such as the Air Force Laboratory's Human Per- formance Modeling Integration Program, the De- partment of Defense, or government laboratories such as the Human Emulation Laboratory at Sandia National Laboratories. NASA is not part of the Hu- man Performance Modeling Special Interest Area.~2 Recommendation: Collaboration with Other Agen- cies. NASA should conduct collaborative research with both the Defense Advanced Research Projects Agency and the DoD to leverage techniques devel- oped by these other agencies for piloting, decision making, estimating human error in automated sys- tems, and vigilance. There is a well-documented, short-term plan with reasonable milestones, but the long-range vision and plan for this initiative lack definitive goals and metrics. Development of a method for comparison across mod- els is encouraged, since current metrics vary from model to model. Finding: Human Factors Outreach. Much of the Human Performance Models work was done by human factors engineers for human factors engi- neers. There is too little outreach to NASA engineers in other disciplines who should be future users of these models. Additionally, program deliverables and their purposes were not clearly articulated. 12See .

OCR for page 71
96 Recommendation: Human Factors Outreach. The NASA Human Performance Models group should work with the managers of all internal aviation re- search programs to identify each manager's need for human performance models and to support the testing of emerging models against human-in-the- loop simulation as well as flight demonstration. Recommendation: Human Factors Outreach. NASA human factors programs should publish a book or CD on the state of human performance modeling to communicate what can realistically be done in this type of modeling and to measure progress in this research area. Recommendation: Human Factors Outreach. NASA should create, document, and apply more clearly defined off-ramps for high-TRL Human Performance Models. NASA Ames has maintained an excellent reputa- tion for sponsoring and convening human performance modelers for several decades. It is essential to strive for continual high quality since human lives are af- fected by the accuracy of the safety estimates derived from these models. The committee applauds the par- ticipation of academia in the NASA Ames aviation safety work but strongly urges outreach to the govern- ment agencies listed in the finding on collaboration, above. In addition, this group has only been able to apply models to a limited number of real-world prob- lems, such as taxiing errors. The committee feels that these models can be tested and improved by applying them to additional real-world problems. NASA is developing tools in this area for others to use. However, actual and potential users, both manag- ers and researchers, should be more clearly iclentified so their input can be solicited when research and appli- cations are being identified and prioritized. Maintenance Human Factors Task The Maintenance Human Factors task is designed to develop "guidelines, recommendations, and tools directly to maintenance personnel and managers"~3 through a combination of research in understanding human error in maintenance and developing mainte- i3B. Karlki, NASA-Ames, "Maintenance Human Factors," ques- tionnaire completed in January 2003 (see Appendix D). AN ASSESSMENT OF NaSA 'S AERONAUTICS TECHNOLOGY PROGRAMS nance tools and aids to enhance safety. The mainte- nance program focuses on risk analysis, resource man- agement, advanced displays, and human error baselines. The effort is funded at $1.1 million net for FY03 and FY04. The importance of human error in the maintenance of aircraft was underscored recently by the US Air- ways Express Air Midwest Flight 5481 accident. The National Transportation Safety Board concluded in May 2003 that the probable cause of the accident, in combination with several other factors, was improp- erly set elevator control cables a maintenance over- sight. In this case, maintenance personnel skipped criti- cal steps outlined in the maintenance manual because they felt the steps were superfluous. This maintenance human factors initiative is criti- cal to reducing maintenance errors as well as to pre- venting injuries to personnel and damage to equipment. Industry applauds the effort. There are many facets to this program, but the resources seem limited relative to the need. This research group has made significant con- tributions in raising maintenance human factors aware- ness within the aviation community. However, this type of research is still in its infancy and just beginning to receive enough attention to identify data sources from which to generate statistically sound trend information. Finding: Maintenance Data Collection. Sources from which to collect data have been identified, but barriers to collection and processing seem to be slowing productive research. Recommendation: Maintenance Data Collection. NASA should develop a clear plan to include inspec- tion data and information from maintenance tech- nician training in its research data set. There is a coherent short-term plan for each of the projects, but the long-range strategic goals seem to be disjointed. The process used for selection of the par- ticular research topics was unclear to the committee. All are potentially useful tools at some level but lack the anchor of a long-term research mission. Specifi- cally, the committee is uncertain how the virtual reality and augmented reality work differs from or comple- ments what industry uses already and how such work will be applied to real-world maintenance error mitiga- tion. There also does not appear to be a systems analy- sis approach to setting priorities for the research effort.

OCR for page 71
ASSESSMENT OF THE AVIATION SAFETY PROGRAM Finding: Goals for Maintenance Research. The Main- tenance Human Factors task is an excellent activity but seems to lack clearly defined long-range goals. Recommendation: Goals for Maintenance Research. NASA's Maintenance Human Factors task should set clear and quantitative long-range goals and test its research against these goals annually. This is an area for long-term research and should be an area for de- veloping enduring core competencies. Finding: Virtual/Augmented Reality. The project to create virtual and augmented reality tools for main- tenance technicians seems to be operating without a clear understanding of what is available today in automation for maintenance technicians and the realities of an all-weather, real-world airline main- tenance operation. Recommendation: Virtual/Augmented Reality. NASA should formally assess the enhanced displays for maintenance research work, including what is currently in use by the airline industry, to deter- mine a more focused and practical approach to vir- tual and augmented reality tools for maintenance. The external community of maintenance human factors researchers was described as small, and there were said to be very close connections between agen- cies and academia. However, the allocation of roles and responsibilities among FAA, NASA, the Navy Safety Center, the Air Force Safety Center, and other research entities was not clear. Finding: Outreach to Community. There are a few omissions in the links to the outside community in this task. Recommendation: Outreach to Community. NASA should establish links to the Air Force Safety Cen- ter as well as airframe and power plant training in- stitutions. NASA should perform an active outreach to the maintenance technician unions for program planning, research vetting, and research participa- tion. NASA should collaborate with the Professional Aviation Maintenance Association on aviation maintenance research and with maintenance tech- nician schools such as the Stratford School to col- lect data and provide research results to enhance safety training. 97 C~ ~ 7 rew 1 raining 1 ash; The goal of this task is to develop training tech- niques and tools to help pilots avoid making errors that lead to accidents and to manage in-flight problems in situations brought about by external circumstances such as weather or system failures. The effort is funded at $1.9 million net for both FY03 and FY04. The Crew Training task of System Safety Technol- ogy has served the commercial aviation training com- munity for many years, producing excellent research work that could occur nowhere else. The current scope of activities is excellent; however, without a long-term core research plan, the projects seem disjointed. Sim~- larly, the individual subtasks in Crew Training are well planned but do not amount to a core training research program. Training research is inherently a long-term activity. Given the inability to go beyond the 5-year horizon for NASA program planning, the researchers in this task have tried to build longer-term research into the current 5-year plan; for example, they have devel- oped an anchor procedure for solving issues relating to flaps and auxiliary power units. In general, the committee found this research ef- fort to be productive and of high quality, with several activities in this task judged to be outstanding. The re- search in distributed team performance is clearly state of the art and is vital in developing flight as well as maintenance training programs. This group has also developed a number of high-quality training tools that have been distributed to the aviation community, par- ticularly the tool known as "How to Train Automa- tion." It is clear that core competencies within this group must be preserved. There is significant interaction and trust between this group and the aviation community operations and training personnel as well as labor unions. The ALPA training council had a meeting at NASA Ames in March 2003. Boeing will be at NASA Ames to review its internal research and development with NASA. These links keep NASA honest and enhance transition to industry. Finding: Outreach to Community. The Crew Train- ing task's already excellent user connection could be enhanced by greater interaction with entities outside the NASA aviation community, including high-level training decision makers at the officer level of major airlines and general aviation companies. Users could

OCR for page 71
98 be more involved than they currently seem to be in setting goals for NASA training research. Finding: Use of Milestones and Feedback. Clear metrics for understanding the impacts of NASA- developed training materials were not apparent. Recommendation: Use of Milestones and Feedback. NASA should institute a crew training quality as- surance program complete with feedback tools that measure adherence to goals and objectives, exit cri- teria, and status in regard to similar research being performed throughout the world. Or '' - ' . . .;' . ., ~ . . , ~ AN ASSESSMENT OF NASA 'S AERONAUTICS TECHNOLOGY PROGRAMS The committee identified several areas in which the training work could be expanded to have additional impact within the aviation community. In particular, there are needs and opportunities for research on main- tenance training that could be addressed in addition to flight crew training. Rotary wing crew training could also benefit from the research expertise gained through this task. The committee felt that some of the projects, such as research on the effects of low blood sugar on safety, would fit better in other venues like FAA's Civil Aero- space Medical Institute, which already has a long- stancling program in this area. Additionally, some of the research results (such as "How to Train Automa- tion") might better be disseminated through the FAA to avoid potential or perceived conflicts between regu- lator expectations and a respected research body such as NASA. Program Human Factors Task The goal of the Program Human Factors task is to identify crosscutting issues in human factors within the AvSP and to make specific human factors recommen- dations to other projects within the program. The effort is funded at $500,000 for both FY03 and FY04, in net dollars. The cockpit integration of the various and dispar- ate tasks of the aviation safety technologies is impor- tant and should be continuously and thoroughly ad- dressed. The Program Human Factors task at NASA Ames is designed to cut across multiple subprojects, in- cluding Synthetic Vision Systems, Weather Accident Prevention, and Single Aircraft Accident Prevention. Each of these subprojects is to perform its own internal human factors research. However, it appears that many key researchers in human factors are affiliated with Ames, making this an appropriate group to evaluate the overall safety program from a human factors perspec- tive. The group has completed a crosscutting look at is- sues arising as a function of humans interacting with synthetic vision. The study revealed that off-nominal procedures were weak; the technology was built, but procedures were poorly developed. This was the only work looking at full integration of synthetic vision with other existing and emerging technologies. This is a critical, real-world issue being addressed by no one else. The committee noted that the objectives of this task seem to have diminished over time, with corre- sponding reductions in allocated resources. Coupled with the 5-year life expectancy of research projects, the end result is that the plan to carry out the program seemed somewhat fragmented. The committee noted that there is only a single in-house researcher; all oth- ers come from outside contractors and academia. This threatens the future of the human factors core compe- tencies at Ames that are so essential to long-term re- search. Finding: Acceptance of Program Human Factors. The other projects within the Aviation Safety Pro- gram may be unresponsive to the recommendations of the Program Human Factors task. Recommendation: Acceptance of Program Human Factors. NASA management should foster greater accountability for the findings of the Program Hu- man Factors research and findings to ensure coop- eration within NASA so that human factors issues identified in Synthetic Vision Systems, Single Air- craft Accident Prevention, and Accident Mitigation are well considered by and integrated into all ap- propriate projects. The program is somewhat disconnected from the users. As with most of the Ames programs, the poten- tial users are quite broadly defined and interaction with users is not sufficiently documented. Human factors engineering has to be assertive to make clear its rel- evance, and thus the committee encourages coopera- tion and outreach with both industry and NASA Lan- gley and broad dissemination of research results. NASA should benchmark against similar external work, such as military projects like those at the De- fense Advanced Research Projects Agency, Big Pic-

OCR for page 71
"at ' '; ; 1 ASSESSMENT OF THE AVIATION SAFETY PROGRAM sure, and Quiet Knight and in forums such as the FAA and the Society of Automotive Engineers and leverage the results of that work. NASA researchers should present results of their work at the Institute of Electri- cal and Electronics Engineers ant! the American Insti- tute of Aeronautics and Astronautics to improve out- reach to potential users. Finding: Human Factors for Commercial Carriers. The research in this area only considers commer- ~ ~ cla1 alr carriers. Recommendation: Human Factors for General Aviation and Rotorcraft. NASA's Program Human Factors research should add general aviation and rotorcraft to its work on human factors, as it could have great impact in these areas. Aviation System Monitoring and Modeling Subproject The ASMM subproject develops technologies to view aviation safety from a systemwide perspective, develops metrics for the safety of the NAS, and pre- dicts systemwide effects of changes to the NAS. The subproject is composed of four tasks: Data Analysis Tool Development, Extramural Monitoring, Modeling and Simulations, and Intramural Monitoring. ASMM has $8.4 million in net funding for FY03 and $8.6 mil- lion in FY04. Data Analysis Tool Development Task The Data Analysis Tool Development task ana- lyzes both digital and textual data. This work tends to be low-TRL. The task develops concepts that are then instantiated in some of the AS MM modeling efforts. This task emphasizes tool design and development over modeling. Currently, the task focuses on two major ar- eas: digital data analysis tools and textual data analysis tools. The first set of tools, a system known as the Profiler, takes digital data from a system like the Avia- tion Performance Measuring System to generate and evaluate flight signatures. From these signatures, the researchers produce a list of atypical flights, identify the atypical parameters, and summarize the results. The second set of tools, known as PLADS (which stands for the steps in the preprocessing: Phrase ID, Leave, Augment, Delete, Substitute) and the Automatic Lan- guage Analysis Navigator (ALAN), preprocesses and 99 processes the kind of text data that would be found in the Aviation System Reporting System. The goal is to identify atypical situations without any a priori infor- mation merely by sifting through the flight data. This task is funded at $1.7 million in FY03 and FY04. The committee was impressed with the work of the contractors and their knowledge of analytical methods. However, it was concerned that the expertise for devel- oping this system is contractor-based and is not part of the NASA Ames knowledge base. The committee was generally impressed with the Profiler work and its abil- ity to identify atypical parameters from signatures. The committee also found the statistical methods used to be sound. In the text area, NASA does not seem to have le- veraged existing software in use by the Securities and Exchange Commission, the Defense Advanced Re- search Projects Agency, and the intelligence commu- nity. Data mining in the textual domain is a widely stud- ied problem, and the committee suggests that the researchers build on existing methodologies. In addi- tion, the text data research work should be dissemi- nated and benchmarked at major text search venues such as the Text Retrieval Evaluation Conference spon- sored by the National Institute of Standards and Tech- nology. The committee was encouraged to see collabora- tion with Office National cl'Etudes et de Recherches Aerospatiales, the French research agency. Such col- laboration should be extended further to other foreign agencies to assure quality benchmarking, including the Japan Aerospace Exploration Agency, the Depart- amento do Aviaco Civil (Brazil), the National Aero- space Laboratory (Netherlands), the State Research Institute of Aviation Systems (Russia), and the Defence Science and Technology Laboratory (United King- dom). Finding: Use of Milestones. The intended path to technology maturation for these data mining tools was not clear. In particular, it was unclear how data mining research was divided among the low-TRL too! development work in this task, the work on data mining applications taking place in the Extramural Monitoring task, and the work on Aviation Perfor- mance Measuring System analysis in the Intramu- ral Monitoring task. Recommendation: Use of Milestones. NASA should define clear goals and objectives, exit criteria, and a

OCR for page 71
100 set of milestones for technology transfer or for the next level of development. Extramural Monitoring Task . . . ~ . - ,. . I. . - ,. . ~ Am ASSESSMENT OF NASA 'S AERONA UTICS TECHNOLOGY PROGRAMS The Extramural Monitoring task strives to create a database of information to serve as the repository of aviation safety events and trends and the basis for avia- tion safety decision making. In particular, the task works with two databases: the National Aviation Op- erations Monitoring Service (NAOMS) and Aviation Safety Reporting System (ASRS), with most of the task investment in NAOMS. The overall funding for Extra- mural Monitoring is $2.0 million. NAOMS consists of a longitudinal survey of air- craft operators, gathering information about safety-re- lated experiences of pilots, cabin crews, and mainte- nance operators for both general aviation and air carriers. NAOMS is a random survey in which staff proactively question active operators in a telephone call. It provides statistically reliable results about the frequency of occurrence of safety-related incidents. In contrast, the ASRS is a joint FAA-NASA re- porting system that asks for the voluntary participation of operators who have experienced a safety-related problem. ASRS is funded by the FAA, although NASA administers the program. To encourage submissions to ASRS, NASA makes sure that the reporter remains anonymous. The FAA had agreed that an ASRS report cannot be used as evi- dence to substantiate an alleged violation in an enforce- ment action.~4 Only a small portion ($250,000 of $2.0 million) of the Extramural Monitoring budget supports ASRS-re- lated activities. That portion of the budget addresses data mining techniques applied to the ASRS database. The NAOMS approach is built on research and implementation of national surveys such as those of the Bureau of Labor Statistics. The NAOMS sampling methods have been grounded in sound interview poll- ing science; however, the interviews are conducted by professional pollsters, not aviation experts. The com- m~ttee has some concern about the level of accuracy attained by pollsters who have no expertise in the area in which they are conducting the telephone interview. The committee is also concerned about potential Remark Blazy, 1999. "We all know about ASRS, but what's an ASRP?" FAAviation News Magazine, October. redundancy between NAOMS data and data available from the air carriers or through the ASRS database. The NAOMS project seems to be developing a meth- odology to establish trends in aviation safety perfor- mance that are already available through other sources within the industry and government. For example, NAOMS appears to duplicate what many airlines are already doing both voluntarily and in FAA-mandated programs to track trends for example, in engine shutdowns. The NAOMS program may become more useful when applied to the general aviation commu- nity, however. NASA's decision to collect its own primary data in this case should rest on the type of research NASA wants to perform and whether that research can be supported by information obtained from the airlines. At this point, the committee does not see a compelling argument for independent data collection. Greater interaction with the Air Transport Association and the airlines might help to clarify the usefulness of this effort. The ASRS program has been around for many years. It is highly trusted by the pilot community and is growing in acceptance by the maintenance technician group. Because the program provides lim- ited immunity from certificate action by the regula- tor for errors by pilots, mechanics, and dispatchers (not willful acts), some tasks within the regulatory community resent the program, while others within the research community disparage its value because the inputs are voluntary. In truth, the threat of a cer- tificate action strongly encourages the submission of an ASRS. Unfortunately, the ASRS program is currently resourced to input less than 25 percent of the reports received into the database. Direct follow- up for additional information from the reporting par- ties can rarely be accomplished. Significantly greater volumes of data are anticipated from emerging Air- plane Safety Action Partnership (ASAP) programs, with no anticipated increase in research resources. This could create a serious shortfall in data available to researchers. While the committee is aware that the funding for the database collection work is pro- videcl by the FAA, not NASA, NASA is still respon- sible for maintaining the ASRS program. The com- mittee finds the defined ASRS activity for NASA to be much larger than its resource allocation; one or the other requires modification. It is important that when gaps in the ASRS data occur, phone calls should be made to fill in what is missing. The lack of resources to handle ASRS in a

OCR for page 71
ASSESSMENT OF THE AVIATION SAFETY PROGRAM statistically sound manner is a significant issue in un- derstanding the safety trends in the NAS. There are many opportunities to accomplish more research with the data available through the ASRS sys- tem. It was not clear if there were plans in this research task to optimize the joint use of ASRS and NAOMS. Finding: Aviation Safety Reporting System. Regret- tably, the Aviation Safety Reporting System data- base is only inputting about 25 percent of the sub- mitted reports. Interviews to follow up on Aviation Safety Reporting System submissions are very lim- ited owing to the lack of resources. The industry believes that the Aviation Safety Reporting System database has been underutilized for some time. The National Aviation Operations Monitoring Service is consuming the majority of the resources in this project area. Recommendation: Aviation Safety Reporting Sys- tem. NASA should combine the National Aviation Operations Monitoring Service methodology and resources with the Aviation Safety Reporting Sys- tem program data to identify aviation safety trends. Modeling and Simulations Task The Modeling and Simulations task seeks to incor- porate human performance models into an analysis of systemwide operations to identify safety-related char- acteristics and predict system response to safety inter- ventions. This program is not responsible for model development, but it incorporates models from other research efforts (such as the AirMIDAS mode! devel- oped in the SWAP program) into a larger, systems- level approach. The task is funded at $1.5 million in FY03 and $1.6 million in FY04. The committee applauds NASA's efforts to inte- grate the various performance models with models of the aircraft and air traffic control systems. This is bold and difficult work and is the kind of research in which NASA should be engaged. The TRL is low, but that is a quality of long-range research that can only be ac- complished by NASA. The weaknesses of the program seem to be a lack of interconnectivity and integration of tools as well as a limited ability to include issues such as clear air turbulence effects on traffic conflict and quality of performance. There is also no collabora- tion between the program and other programs that model environmental safety and noise. 101 The Reconfigurable Flight Simulator and Object Based Event Scenario Trees modeling programs are not tied to NAS models built using the FAA Consolidated Operations and Delay Analysis System and Aviation System Performance Metrics. Nor was there a tie to the Total Airspace and Airport Modeler, which has been validated by Eurocontrol, or the Traffic Organization and Perturbation Analyzer model (developed by the National Aerospace Laboratory in the Netherlands), which has been used to estimate the safety-capacity relationship that may be affected by airports at high operational workloads. Finding: Outreach to the Modeling Community. The modeling programs in this area have excellent potential but appear to lack coordination with other similar modeling programs. Recommendation: Outreach to the Modeling Com- munity. This task should benchmark its perfor- mance against other modeling implementation ef- forts and consolidate programs where possible to achieve a master system performance, capacity, and safety model. Intramural Monitoring Task Intramural Monitoring refers to internal quality assurance and safety functions within each air carrier and air traffic management organization. The Intramu- ral Monitoring products are the Aviation Performance Measuring System (APMS) and the Performance Data Analysis and Reporting System (PDARS). The APMS project is designed as a tool for analyzing aircraft flight data. APMS provides envelope data for each flight pa- rameter in typical flights, provides information about atypical flights, and provides descriptive statistics on phase-of-flight performance. PDARS is designed to collect, process, and analyze air traffic management data. It generates daily reports, shares data among fa- cilities, supports exploratory and causal analysis, and archives data for developing baselines. Its major strength is in the seamless integration of data from multiple sources. The overall task emphasis is on safety risk management. The task received $3.18 million in FY03 and expects $3.25 million in FY04. Some committee members worried that APMS and PDARS were not novel. The committee believes com- peting and sometimes superior systems are already used by airlines.

OCR for page 71
102 l .. . . .. ... AL ASSESSMENT OF NaSA 'S AERONAUTICS TECHNOLOGY PROGRAMS Overall, the APMS program has been in the refine- ment stage for several years. A target and milestones for technology transfer or the next level of develop- ment were not clear. Benchmarking of APMS against similar programs in other government arenas and academia seems to be lacking. The APMS tools to mine anomalous data are en- tirely appropriate and useful for airline flight opera- tional quality assurance (FOQA) programs. However, there are significant barriersamong them litigation issues to centralizing a general FOQA database at a government agency in the near term. This creates bar~- ers to close interaction with the industry. To make this array of activities more complete, emphasis and resources in this program need to shift further to integrating APMS and other complementary, commercially available FOQA software into an inte- grated operational efficiency and risk model. Finding: Aviation Performance Modeling System. The APMS software is mature in its development and is ready for the off-ramp to the marketplace. Recommendation: Aviation Performance Modeling System. NASA should redirect the APMS resources to pursue integrated data risk model research. The weather overlay work is a clear example of the kind of research that needs to be emphasized. Finding: Performance Data Analysis and Report- ing System. As a safety analysis tool, PDARS was well designed and is being utilized extensively by air traffic control management. PDARS is useful for airspace design, but it is at a fairly high TRL and is ready to be turned over to industry. The committee identified only one remaining gap in the research activity- data source integration. Recommendation: Performance Data Analysis and Reporting System. NASA PDARS resources should be used to integrate PDARS data with traffic and weather information to feed NASA's modeling and simulation activities. In addition, methods to inte- grate the Flight Operations Quality Assurance (FOQA) program and Airlines Safety Action Part- nership and Aviation Safety Reporting System in- formation into the higher level models should be developed.

OCR for page 71
Appendixes 1 . .

OCR for page 71