Summary

BACKGROUND

Real-time system-wide safety assurance (RSSA) is one of six focus areas for the National Aeronautics and Space Administration (NASA) aeronautics program. NASA envisions that an RSSA system would provide a continuum of information, analysis, and assessment that supports awareness and action to mitigate risks to safety. NASA’s research plans state that development of an RSSA system for the national airspace system (NAS) will necessitate automating safety assurance of air transportation system components, integrating component-level systems, and reducing the safety assurance cycle time until real-time safety assurance is achieved at the system-of-systems level. The safety assurance system envisioned by NASA will combine air traffic and onboard aircraft technologies as well as automated data mining capabilities for continuous safety monitoring and threat prediction.¹ This system is expected to maintain or exceed the current level of aviation safety while accommodating global increases in air travel and rapid introduction of new technologies. The RSSA system would not be expected to directly address issues related to design, development, training, or maintenance because the detection of problems in these areas and the process of implementing corrective actions falls outside the short time horizon of an RSSA. Other systems are already in place to address these aspects of aviation safety.

NASA envisions that the process of developing a comprehensive and fully functional RSSA system would include three intermediate milestones:²

• Domain-Specific³(Real-time) Safety Monitoring and Alerting Tools (2015-2025). Expanded system awareness through increased access to safety-relevant data and initial integration of analysis capabilities; improved safety through initial real-time detection and alerting of hazards at the domain level; and decision support for limited, simple operations.⁴

___________________

¹ NASA, 2015, NASA Technology Roadmaps TA 15: Aeronautics, NASA, Washington, D.C., https://www.nasa.gov/sites/default/files/atoms/files/2015_nasa_technology_roadmaps_ta_15_aeronautics_final.pdf.

² Briefing by J. Nowinski at the first meeting of the Aviation Safety Assurance Committee, January 23, 2016, Washington, D.C., p. 6.

³ One example of a “domain-specific tool” would be a tool that monitors the NAS and determines the system state as it applies to a specific class of aircraft operating in the airspace near a specific airport. After such a tool is demonstrated and validated at that airport, continued development could expand its applicability to other classes of aircraft and other airports.

⁴ In the context of this milestone, “limited, simple operations” are intended by NASA to refer to, for example, tools with limited automated decision-making and mitigation capabilities that could be demonstrated and validated in low-risk operations, such as those involving small unmanned aircraft systems (UAS) operating in unpopulated areas.

• Integrated Predictive Technologies with Domain-Level Application (2025-2035). NAS-wide availability of more fully integrated real-time detection and alerting for enhanced risk assessment and support of initial assured human and machine decision support for mitigation response selection for more complex operations.
• Adaptive Real-time Safety Threat Management (2035-2045). Fully integrated threat detection and assessment that support trusted methods for dynamic, multi-agent planning, evaluation, and execution of real-time risk-mitigating response to hazardous events.⁵

Maintaining the safety of the NAS as it evolves will require integration of a wide range of safety systems and practices, some of which are already in place and many of which need to be developed. Maintaining system safety into the future will require rapid detection and timely mitigation of safety issues as they emerge and before they become hazards. This report identifies challenges to establishing an RSSA system and the high-priority research that should be implemented by NASA and other interested parties in government, industry, and academia to expedite development of such a system. In order to assess NASA’s vision for an RSSA system and to develop a national research agenda consisting of high-priority research projects, a challenge assessment was conducted in four fundamental system element development areas.

• Concept of Operations and Risk Prioritization. A clear concept of operations (CONOPS) is needed to define the scope of an RSSA, to understand how it would work, and to establish the framework for other areas of research. The system’s capabilities will need to increase in sophistication as the NAS⁶ continues to evolve and improve, while also accommodating continued growth in conventional air traffic and a wide range of new entrants. This report identifies three classes of new entrants that are of particular interest to the development of an RSSA system: UAS,⁷ on-demand mobility,⁸ and the increasing pace of commercial space operations.
• System Monitoring. Aviation safety assurance begins with a monitoring function that observes the system state by fusing varied and complex data from a multitude of sensors.⁹ Issues of interest include identifying, characterizing, and collecting high-quality data. Additionally, data regarding operator performance cannot currently be collected in a timely fashion, or at all, in part because of privacy and related concerns.¹⁰
• System Analytics. Sophisticated algorithms and computational architectures will play a central role in the assessment function by interpreting and analyzing the state of the NAS and identifying elevated risk states, which then form the basis for mitigating actions to maintain safe operations. The large volume and heterogeneity of NAS data and the need to align and fuse data from multiple sources make it particularly difficult to develop algorithms, especially machine learning algorithms, that can identify and characterize existing and emergent risk states.

___________________

⁵ In the context of this milestone, “hazardous events” also refers to hazardous trends and conditions.

⁶ The NAS is “the common network of U.S. airspace; air navigation facilities, equipment, and services; airports or landing areas; aeronautical charts, information and services; rules, regulations, and procedures; technical information; and manpower and material.” FAA, 2013, Integration of Civil Unmanned Aircraft Systems [UAS] in the National Airspace System [NAS] Roadmap, Washington, D.C., https://www.faa.gov/uas/media/uas_roadmap_2013.pdf. Some NAS facilities are jointly operated by the FAA and the Department of Defense. The NAS includes all aircraft operating in U.S. airspace, both foreign owned and domestic.

⁷ An “unmanned aircraft” is, as the name implies, an aircraft that has no onboard pilot. In this report unmanned aircraft are assumed to have no humans on board either as flight crew or as passengers. A UAS is an unmanned aircraft and its associated elements, including ground control and communications equipment.

⁸ “On-demand mobility” (ODM) is an emerging concept for commercial aviation that would feature small aircraft providing on-demand transportation for individuals or small groups of passengers within urban areas, over relatively short intercity distances, and in some cases over longer distances for transportation to or from small and underserved airports. (Although some ODM concepts focus on ground transportation, this report refers to ODM exclusively in terms of aviation.)

⁹ Data fusion involves correlation and synthesis of data from heterogeneous data sources with different formats, timing, accuracy, and other characteristics.

¹⁰ In the aviation community, the term “operator” is used to refer both to individual human operators (e.g., pilots and air traffic controllers) or to the organizations that operate aircraft (e.g., airlines and government agencies). This report follows the same convention. Each time “operator” appears in the report, the specific meaning should be clear based on the context.

• Mitigation and Implementation. The ability to mitigate elevated risk states on a much faster time scale than existing safety management systems is limited. As aviation system complexity increases, so does the risk of unintended consequences due to system actions and recommendations. Success requires that human operators trust system outputs, which include alerts, decision support, and independent actions. Accepted approaches for verification, validation, and certification of real-time system solution sets are lacking.

OVERARCHING VISION COMMENTARY

Decades of continuous efforts to address known hazards in the NAS and to respond to issues illuminated by analysis of incidents and accidents have made airlines the safest mode of transportation. The task of maintaining a high level of safety for commercial airlines and other operators is complicated by the dynamic nature of the NAS.¹¹ The number of flights by commercial transports is increasing, air traffic control systems and procedures are being modernized to increase the capacity and efficiency of the NAS, increasingly autonomous systems¹² are being developed for aircraft and ground systems, and small aircraft—most notably UAS—are becoming much more prevalent. As the NAS evolves to accommodate these changes, aviation safety programs will also need to evolve to ensure that changes to the NAS do not inadvertently introduce new risks. In this context, the vision that NASA holds for an RSSA system is well founded.

A potentially confusing facet of NASA’s RSSA research lies in the descriptor “real time.” A common understanding of “real time” is that it describes events that occur at the same time or nearly so. Some elements of an RSSA system would indeed occur in real time, just as the Traffic Collision Avoidance System (TCAS) operates in real time to continuously monitor an aircraft’s position and velocity vector relative to the terrain and to other aircraft to immediately alert pilots when the risk of a collision exceeds a programmed threshold. Other elements of an RSSA system, however, could operate over a period of minutes, hours, or even days to look at operational trends over these time scales to identify risks that cannot be identified in real time.

Additionally, a safety assurance system, as defined by the International Civil Aviation Organization (ICAO), “consists of processes and activities undertaken by the service provider to determine whether the safety management system (SMS) is operating according to expectation and requirements.” An SMS is more comprehensive in that it uses “a systematic approach to managing safety including the necessary organization structures, account-abilities and policies and procedures.” To successfully achieve NASA’s vision, the latter will be required.

The committee’s vision of an in-time aviation safety management system appears in Box S.1. This description does not specify the use of any particular programmatic approach for achieving the vision. Chapter 5, however, notes that an approach with interim deliverables would facilitate development of a consensus in the aviation community to support IASMS research. The approach envisioned by NASA for development of an RSSA is structured to provide such deliverables.

The committee’s vision for an IASMS is summarized in the following recommendation:

Recommendation. In-time Aviation Safety Management. The concept of real-time system-wide safety assurance should be approached in terms of an in-time aviation safety management system (IASMS) that continuously monitors the national airspace system, assesses the data that it has collected, and then either recommends or initiates safety assurance actions as necessary. Some elements of such a system would function in real time or close to real time, while other elements would search for risks by examining trends over a time frame of hours, days, or even longer.

___________________

¹¹ In this report, “commercial transports” refers to aircraft operated by regional and major passenger airlines as well as cargo airlines.

¹² Increasingly, autonomous systems lie along the spectrum of system capabilities that begin with the abilities of current automatic systems, such as autopilots and remotely piloted (nonautonomous) unmanned aircraft, and progress toward highly sophisticated systems that would enable, for example, UAS that could operate independently within civil airspace, interacting with air traffic controllers and other pilots just as if a human pilot were on board and in command (National Research Council, 2014, Autonomy Research for Civil Aviation: Toward a New Era of Flight, The National Academies Press, Washington, D.C.)

HIGH-PRIORITY RESEARCH TOPICS

In order for an IASMS to properly function in accordance with the intended vision, research is required in each of the four system elements described above: CONOPS and risk prioritization, system monitoring, system analytics, and mitigation and implementation. For each element, existing technologies and analytical capabilities cannot handle, in-time, the vast amount of data and information needed by an IASMS.

Research priorities are identified based on the understood difficulty of completing each item and the urgency with which they should be initiated so that the research output will be available in a manner that supports the intermediate milestones identified by NASA.

The committee identified 10 high-priority research projects that it recommends for consideration by agencies and organizations in government, industry, and academia with an interest in developing an IASMS for the NAS. All 10 are judged to be both difficult and urgent; if they were not, they would not have been designated as a high priority. As indicated in the following, two of the high-priority research projects address an IASMS concept of operations and risk prioritization, two address data collection for system observation, three address system analytics, and three address mitigation and implementation.

For most of the research projects, meeting the needs of an IASMS will likely require a mix of new technologies, improvements to existing technologies, and the adaptation of existing technologies developed for other applications. Each research project, as applicable, will need to determine the appropriate mix for that project.

The research project IASMS Concept of Operations and National Airspace System Evolution is judged to be of the highest priority (see Chapter 6). The report does not otherwise address the relative priority of the high-priority research projects, because execution of most of the projects is most likely to be successful if they proceed in an iterative and integrated fashion that accounts for the many interactions among the different projects in Chapters 2 to 5.

IASMS CONCEPT OF OPERATIONS AND RISK PRIORITIZATION

IASMS Concept of Operations and National Airspace System Evolution

This research project would develop a detailed concept of operations for an IASMS using a process that considers multiple possible system architectures, evaluates key trade-offs, and identifies system requirements. This would (1) establish the framework upon which all other IASMS research is conducted, (2) identify the near-term potential of IASMS research to enhance the safety of the NAS and to engender stakeholder support for and trust in an IASMS, and (3) facilitate updates to the CONOPS as the NAS evolves.

Developing a detailed CONOPS will be extremely difficult and time consuming because an IASMS will be a complex and dynamic system of systems and because of the many factors to be considered and the difficulty of assessing the trade-offs and interactions among them. This research is urgent because of the complexity of achieving IASMS goals and because it will establish the framework upon which all other research projects flow. Developing a detailed IASMS CONOPS will also define timelines for infrastructure investment strategies that would most efficiently support development of an IASMS.

Identifying and Prioritizing Risks

This research project would develop processes to identify and prioritize risks that are relevant to an IASMS and that threaten the safety of the current and evolving NAS. This would lead to approaches for identifying emerging risks and for prioritizing known and emerging risks that fall within the scope of the IASMS CONOPS.

The traditional approach to risk assessment is based on an evaluation of the probability of occurrence and the consequence of an event. The highest risks occur when the consequences of an event are the most severe and the probability of it occurring is the highest over some period of time. An ongoing process of identifying and prioritizing risks that an IASMS will address is important because additional risks will emerge. These new risks will arise due to changes in operations in the NAS, technological advances, increased connectivity, the implementation of next-generation airspace procedures such as delegation of separation, and other exogenous and internal threats. Also, as the safety of various elements of the NAS improves and as the probability threshold for a risk to be mitigated lowers, the number of elevated risk states that should be considered for mitigation will increase. Because any mitigation approach will introduce some cost into the system, risk prioritization is needed to facilitate development of an affordable IASMS.

This research project will be difficult to complete largely because of the uncertainties associated with identifying emerging risks. This research is urgent because it is essential to the development of an IASMS CONOPS scope.

SYSTEM MONITORING

Data Fusion, Completeness, and Quality

This research project would develop methods to automatically collect, fuse, store, and retrieve data from different sources and with different formats, timing, accuracy, and other characteristics. The range of IASMS capabilities that can be successfully implemented will be limited by the completeness of the data available, by its quality and

consistency, by the ability to fuse it in the time scales of interest, by the ability to store it for future use, and by the relative cost and value of obtaining additional or higher-quality data sources as required.

This research will be difficult to achieve given the substantial advances that are needed to develop the ability to define, acquire, understand, fuse, and store the data required to support planned IASMS capabilities. This project is urgent because it is fundamental to the success of an IASMS risk identification and prioritization process and because some components will likely take years to complete.

Protecting Personally Identifiable Information

This research project would develop methods of de-identifying and/or protecting sensitive data in a way that does not preclude effective data fusion. This would help achieve the vision for an IASMS by developing systems that will permit the automated fusing of large data sets without compromising the identification of the operator. For information to be used for in-time monitoring and assessment and to be stored for future use, advances in technology (and changes to regulatory policy) are needed to address operators’ concerns regarding unauthorized disclosure of identifiable data. To meet the needs of an IASMS, de-identification methods will need to operate quickly and without loss of key operational safety data.

This research will be difficult to complete because the source data will be generated from unique and sometimes proprietary systems. This research project is urgent because of the time that it will take to develop improved methods for de-identifying and protecting data and to then develop a broad consensus among stakeholders—including operational personnel, unions, and the leadership of airlines, other operators, the Federal Aviation Administration (FAA), and original equipment manufacturers—that these methods are adequate.

SYSTEM ANALYTICS

In-time Algorithms

This research project would develop robust and reliable algorithms that can assess large volumes of heterogeneous data of varying quality to simultaneously identify and predict elevated risk states of many different types and that are fast enough to meet in-time requirements.

This research project would be difficult to complete because of the growing complexity of the NAS and because of the large and growing number and variety of aircraft operating in the NAS, including new entrants. In addition, this research project faces significant uncertainties regarding the ability to acquire all of the data needed to monitor the NAS, to assess the system state, and to detect elevated risk states. This research project is urgent because in-time algorithms will form the core of the monitoring, detection, prediction, and mitigation tasks of the IASMS.

Emergent Risks

This research project would develop approaches for continually mining historical data for detecting previously unknown anomalies and their evolution, to characterize emergent risks, and to update the IASMS risk assessment algorithms.

As with the preceding In-time Algorithms research project, this research project will be difficult to complete because of the growing complexity of the NAS and because of the large and growing number and variety of aircraft operating in the NAS, including new entrants. This research is urgent because it will take a long time to develop the new classes of offline data-driven methods,¹³ machine learning and data mining algorithms, and analysis and prediction techniques that will be needed for each functional element (monitor, assess, and mitigate) of the IASMS to address adequately the hazards posed by emergent risks.

___________________

¹³ “Offline analysis” refers to analysis of stored data as opposed to online analysis of streamed data in real time or near-real time.

Computational Architectures

This research project would support the design of data repositories and computational architectures that support online detection of elevated risk states and offline analysis to detect and characterize emergent risks and to update the IASMS risk assessment algorithms. Existing computational architectures lack the ability to handle large volumes of heterogeneous data and dynamic analytics workflows to support in-time analysis.

This research project will be difficult to complete because research and development focused on other applications will not meet the unique needs of an IASMS in terms of scope; spatial and temporal complexities; the need for timely processing of large volumes of streaming and stored heterogeneous data with varying levels of quality and frequency; and the need to provide a reliable, fault tolerant, and secure system that degrades gracefully when adverse situations (e.g., regional power failures) and malicious threats are launched against the system. This research is urgent because data repositories and computational architectures will provide the backbone of the IASMS operational system and are therefore needed early in the IASMS research effort.

MITIGATION AND IMPLEMENTATION

In-time Mitigation Techniques

This research project would, for the high-priority risks that fall within the scope of the IASMS CONOPS, identify those risks for which adequate mitigation techniques do not currently exist and develop approaches and technologies necessary to implement timely mitigation.

This research project will be difficult to complete because of the need for new instrumentation, advanced analytic methods, and sophisticated prediction capabilities that take into account the increasing complexities and uncertainties in the evolving NAS, particularly with respect to new entrants. This research project is urgent because the success of an IASMS is dependent on near- and long-term mitigation schemes to maintain the safety and efficiency of the NAS and because of the long time it will take to achieve project goals.

Trust in IASMS Safety Assurance Actions

This research project would identify factors specific to human trust in IASMS safety assurance actions. IASMS will rely on systems that are growing in functionality and decision-making capabilities. Many factors, such as the frequency and complexity of operator interactions with the system, will need to be understood and addressed in order to foster operator trust in the system and to create the proper workload balance between the operator and a system so that the operator does not become overloaded. Change management processes will be critical when the system is deployed. The examination of the preceding factors, however, will need to occur at a much earlier stage than typical change management processes to assist in shaping the CONOPS, design, and implementation of an IASMS.

This research project will be difficult to complete because creating an IASMS that operators will trust, and therefore use, will require a thorough understanding of the potential capabilities, nuances, and emergent properties of IASMS. The research is urgent because operator trust is a relatively new field, and this research project therefore does not have a large body of work to use as a resource. In addition, the results of this research project will be most effective if they are available early enough in the IASMS development process in order to support the design and development of IASMS.

System Verification, Validation, and Certification

This research project would develop practical methods for verifying, validating, and certifying an IASMS. A system as complex as an IASMS, which can influence immediate operations, will need to be certified before it becomes operational. Although research in this area is already under way to support related applications, such as certification of a UAS traffic management (UTM) system and autonomous cars, existing research will not meet the

unique needs of an IASMS because an IASMS will be much more complex than a highly automated/autonomous aircraft, and it will need to monitor and assess the operational safety of all existing and new entrants that will be operating in the NAS, encompassing the existing air traffic management (ATM) systems as well as UTM systems.

This research project will be difficult to complete because development of certification standards for an IASMS will require a new approach to certification that promotes rapid and yet safe changes to the system. This research project is urgent because of the expected long lead time involved in creating viable verification, validation, and certification (VV&C) processes that can be standardized and applied to other ATM systems or to develop an alternate approach to VV&C.

ECONOMIC CHALLENGE

The report identifies 13 technical challenges that are addressed by the recommended high-priority research projects, discussed above. In addition, there is one economic challenge, as detailed below.

Operators’ Costs and Benefits

Operators’ perception of the cost-to-benefit ratio of an IASMS may be so high that it will impede its implementation. Some aviation safety programs, such as the Aviation Safety Action Program (ASAP), could be implemented without new equipage. Many of the future data sources needed for the successful adoption of a fully functional IASMS, however, will require new and sophisticated onboard equipment along with the adoption of ground infrastructure and data processing capability. Airline operations in the NAS are already extremely safe—and given the limited financial resources of airlines and other operators—the ability to adopt new and potentially costly investments in a new safety system such as an IASMS will not easily pass the traditional cost-to-benefit ratio for adoption. If that is the case, widespread use is unlikely to occur unless and until regulatory mandates are issued.