4
System Analytics
The generic IASMS CONOPS presented in Figure 2.1 depicts the key role that the assessing function plays in interpreting and analyzing the state of the NAS and identifying elevated risk states, which then form the basis for mitigation to maintain safe operations. The assessing function will be enabled by sophisticated analytics functions and algorithms that (1) identify and characterize known risk states in the time frame of interest to an IASMS and (2) examine large volumes of stored flight and ground operations data with anomaly detection methods to identify and characterize emergent risks and to update IASMS risk assessment algorithms.
Developing in-time algorithms for identifying and predicting elevated risk states presents many challenges. The lack of a well-defined CONOPS for IASMS (see Chapter 2) makes it hard to define the scope of the in-time algorithms, to define algorithm performance requirements, and to develop verification and validation (V&V) methods for the developed algorithms. The large volume and heterogeneity of NAS data (from commercial transports, UAS, commercial spacecraft, and general aviation aircraft) and the need to align and fuse data from multiple sources (from ground and air operations, ADS-B systems, pilot communications from individual aircraft, and reports of weather conditions throughout the NAS) make it particularly difficult to develop new or improved algorithms, especially machine learning algorithms, and/or make new applications of existing algorithms in order to identify and characterize risk states. Overcoming many of the other challenges addressed in Chapters 2 and 3 will also create demands for more capable analytical systems.
The complexity and size of the NAS implies that a large number of factors can influence system safety. Identifying and characterizing elevated risk states will require developing sophisticated machine learning algorithms that can operate on heterogeneous data of varying quality. Furthermore, the continually evolving NAS will require development of advanced anomaly and hazard detection algorithms that can operate on large volumes of historical operations data to characterize and predict emergent risks and to reprioritize known risks.
In-time safety assessment for a large number of risk factors will require the development of computational architectures for data input and output devices, processing capabilities, and storage that can work with high-volume and high-speed streaming of data from multiple sources. The FAA’s SWIM architecture (see Chapter 3), if scaled up to include more distributed architectures and networked repositories, will have the potential to support in-time elevated risk identification and offline anomaly detection algorithms for characterizing emergent risks.1
___________________
1 “Offline analysis” refers to analysis of stored data as opposed to online analysis of streamed data in real time or near-real time.
This chapter identifies three key challenges and three high-priority research projects:
- Challenges
- In-time Algorithms
- Emergent Risks
- Computational Architectures
- Research Projects
- In-time Algorithms
- Emergent Risks
- Computational Architectures
CHALLENGES
In-time Algorithms
Challenge Summary Statement: Existing algorithms for identifying and predicting elevated risk states lack the ability to integrate the diversity of data sources of varying quality anticipated for an IASMS.
Developing the algorithms for an IASMS will be a key challenge because the NAS will experience large increases in air traffic by commercial transports and new entrants, such as UAS, whose interactions with manned aircraft are still being investigated. Moreover, the introduction of trajectory-based operations for commercial transports will change the role of ATM systems from tactical to more strategic decision making that will be distributed and partially automated.2 This will introduce new issues for the algorithms and analytics developed for monitoring, detection, and mitigation of known and emergent risks.
Whereas the traditional role of an IASMS would be to focus on known, high-priority risks, significant changes in airspace operations could result in a reprioritization of known risks and the introduction of new risks. The likelihood of some known risks (see Figure 2.3) may decrease significantly with the introduction of new devices (e.g., ADS-B) and increased automation, but new high-priority risks will likely emerge, and risk prioritization will continue to evolve with changes in operations and as new information becomes available (see “Identifying and Prioritizing Risks,” in the Challenges section of Chapter 2).
The large number of relevant factors and the interactions among them will complicate the process of studying and analyzing some high-priority risks. For example, factors related to midair/near midair collisions include trajectory-based operations, interactions between UAS and manned aircraft, the impact of environmental conditions such as turbulence and weather on flight and ground operations, and the cognitive and emotional states of operators.
The challenges in Chapter 3, Systems Monitoring, are also relevant to this challenge. In-time algorithms will require large volumes of heterogeneous, multimodal data, and the ability to process them in in a timely fashion so that an IASMS can monitor ground and air operations and identify and characterize the current state of NAS. Data quality and completeness as well as data fusion will impose requirements on the data-driven state identification methods regarding the ability to process data from multiple sources of varying levels of uncertainty to determine their impact on the reliability of the assessment function as it detects elevated risk states. Key factors related to uncertainties include accuracy, timeliness, completeness, availability, and reliability of the data. Research into uncertainty management and risk analysis methods will enable the detection of elevated risk states and the generation of alarms with low false-alarm rates. For example, in-time monitoring, hazard detection, safety analysis, and
___________________
2 The implementation of NextGen improvements in navigation, communications, surveillance, and automation will enable the ATM system to adopt trajectory-based operations. This will enable the more efficient use of the NAS in the air and on the ground at airports. Full implementation of trajectory-based operations will require operators to install Automatic Dependent Surveillance-Broadcast (ADS-B) and other new equipage on their aircraft (Federal Aviation Administration, 2016, The Future of the NAS, Washington, D.C., https://www.faa.gov/nextgen/media/futureOfTheNAS.pdf.
prediction algorithms for an IASMS will need to operate in an anytime3 manner, and this will require algorithms to account for the different sources of uncertainty and to compute the contextualized risk associated with hazards in dynamically evolving situations.
Developing algorithms to address human performance states is a particularly difficult issue because there is little understanding of the comprehensive set of cognitive and emotional states and conditions of individual humans and their interactions in distributed environments that provide an understanding of normal behavior. In contrast, there is significant understanding and maturity of the laws of physical processes that lead more directly to assessing normalcy in those systems. Individual human responses, behaviors, cognitive states, and emotional states cannot yet be measured or predicted reliably or accurately. If sufficient data are identified and collected on human performance (see Chapter 3), then quantifying and evaluating these data on a relevant time scale is one factor that is needed for an IASMS to achieve its full potential.
Emergent Risks
Challenge Summary Statement: The complexity of the evolving NAS will result in anomalies with unknown root causes, making it hard to develop algorithms that analyze and predict the effects of emergent risks before accidents or incidents occur.
Emergent risks associated with the growth in air traffic by commercial transports, UAS, on-demand mobility, and commercial space operations are discussed in “National Airspace System Evolution,” in the Challenges section of Chapter 2. The challenge discussed here is related to the effect of emergent risks on requirements for IASMS analytics to support the discovery and analysis of the effects of emergent risks and to guide the design and development of mitigation procedures to address newly identified risks.
Developing the ability to analyze and predict the threat posed by emergent risks will be a key challenge because the threat posed by emergent risks could increase in frequency and severity as the NAS evolves. Factors related to this challenge include the difficulty of interpreting and tracking the evolution of previously unknown anomalous situations in historical data, especially when they involve complex operational scenarios. In general, analysis schemes will be needed for multidimensional, time series data. These data will be generated by monitoring the state and trajectories of multiple aircraft. The quality of some key data segments may be highly uncertain in that to some extent they may be inaccurate, misleading, missing, or misaligned. Furthermore, the nominal modes of operation of the NAS are not fully known, especially with respect to new entrants. This makes it difficult to differentiate anomalous situations from nominal situations and to establish the root causes of the former with any degree of certainty. As a result, it will also be difficult to establish and validate elevated risk states resulting from emergent risks and to study their longer-term consequences on the operations of the NAS. The form of data relevant to emergent risks and their features may be hard to identify, and the operational data available may be incomplete.
Computational Architectures
Challenge Summary Statement: Existing computational architectures lack the ability to handle large volumes of heterogeneous data and dynamic analytics workflows, both of which are necessary to detect elevated risk states, to detect and characterize emergent risks, and to update the IASMS risk assessment algorithms.
Computational architectures typically focus on data sources, storage, computing mechanisms, and the presentation and delivery of results to the user. Developing the computational architectures needed by an IASMS
___________________
3 Anytime algorithms are required to produce the best results they can within a given time bound for computation. In other words, they produce a result even if they are interrupted before completion (S. Zilberstein, 2017, Using anytime algorithms in intelligent systems, AI Magazine 17(3):73).
will be a key challenge because these architectures will deal with big data4 acquired from a variety of sources, some of which may be stored (such as past aircraft trajectories, or runway configurations). Much of this will be streaming (e.g., data related to current aircraft locations, weather conditions, and pilot-to-pilot and pilot-to-air traffic controller communications). Analysis of these data to identify the occurrence of known hazards (e.g., loss of control) in real time as well as over periods of hours or days to look for anomalies and emerging risks presents many difficulties. Parallel computing architectures, perhaps in the form of extensions and derivatives of the current Hadoop5 infrastructure (such as Apache Hama6), will have to be developed for processing of streaming and stored data.
As the ATM system evolves, it is unclear how data collection methods and systems can scale up to collect the very large volumes of heterogeneous, multimodal data that are going to be generated from multiple sources and multiple regions in the airspace. These include different commercial airlines, UAS operators, ground operations systems, environmental data such as weather conditions and turbulence, and operational data from the ATM system. A key part of this challenge will be how to scale up the current data exchange standards established in the SWIM program to accommodate the collection and distribution of the large volumes of heterogeneous, multimodal data from operations of the NAS, especially as the system evolves. The centralized SWIM architecture may need to be revamped to set up a shared, distributed computational architecture that includes networked repositories and computational systems to support online IASMS analytics operations that cover ground operations and the airspace.
Another key element of this challenge is the need for computational architectures to support multiple data sources and consumers of various components of the data. This raises additional issues, including the need for infrastructure for data abstraction, alignment, and integration. In addition, advanced software that interconnects various elements of the architecture is needed to build analytics workflows that can be configured for a variety of data sources and algorithms described above. Configuration and reconfiguration of computational workflows would also benefit from the ability to incorporate new analytics components to enable newer functionalities. Another necessary element would be a management layer in the architecture to ensure the seamless execution of both online and offline analytic tasks.
___________________
4 One can separate big data and “regular-size” data based on the presence of a set of characteristics commonly referred to as the four V’s: volume, variety, velocity, and veracity. The volume of data collection is pervasive across industries including finance, manufacturing, retail, health, security, technology, and NAS operations. Furthermore, in the vocabulary of big data, petabytes and exabytes have now replaced terabytes. To put these volumes into perspective using the classic grains of sand analogy: if a megabyte is a tablespoon of sand, a terabyte is a sandbox 2 feet wide and 1 inch deep, a petabyte is a mile-long beach, and an exabyte is a beach extending from Maine to North Carolina. The variety comes from structured, semistructured, and unstructured data accumulated from multiple sources, and includes traditional transactional data, user-generated conversations, sensor-based data, and spatial-temporal data. The velocity of data creation is a hallmark of big data, and it has important implications for “real-time” predictive analytics in various application areas, ranging from finance to health. Simply put, analyzing “data in motion” presents new challenges because the desired patterns and insights are moving targets, which is not the case for static data. Veracity pertains to the credibility and reliability of different data sources, which may have varying degrees of noise, corruption, and incompleteness. In addition, aligning large volumes of time-series data from heterogeneous sources can be a challenge, as is deriving deep semantic knowledge from a combination of these data sources (A. Abbasi, S. Sarker, and R.H. Chiang, 2016, Big data research in information systems: Toward an inclusive research agenda, Journal of the Association for Information Systems 17(2):1-32).
5 Hadoop (created by Doug Cutting and Mike Cafarella in 2005) makes it possible to run applications on systems with thousands of commodity hardware nodes, and to handle thousands of terabytes of data. Its distributed file system facilitates rapid data transfer rates among nodes and allows the system to continue operating in case of a node failure. This approach lowers the risk of catastrophic system failure and unexpected data loss, even if a significant number of nodes become inoperative. Consequently, Hadoop quickly emerged as a foundation for big data processing tasks, such as scientific analytics, business and sales planning, and processing enormous volumes of sensor data, including from Internet of Things sensors. It is part of the Apache project sponsored by the Apache Software Foundation. (See TechTarget, “Hadoop,” in “Essential Guide: Managing Hadoop Projects: What You Need to Know to Succeed,” last updated September 2016, http://searchcloudcomputing.techtarget.com/definition/Hadoop?.)
6 K. Siddique, Z. Akhtar, E.J. Yoon, Y.S. Jeong, D. Dasgupta, and Y. Kim, 2016, Apache Hama: An emerging bulk synchronous parallel computing framework for big data applications, IEEE Access 4:8879-8887.
RESEARCH PROJECTS
In-time Algorithms
Research Project Summary Statement: Develop robust and reliable algorithms that can assess large volumes of heterogeneous data of varying quality to simultaneously identify and predict elevated risk states of many different types, and that are fast enough to meet in-time requirements.
This research project would help achieve the vision for an IASMS, because IASMS will be dealing with a new and evolving environment for flight management operations, which could create situations where a lack of knowledge about how hazards evolve may hamper detection and decision making. Addressing this shortfall will require development of advanced machine learning methods to analyze large volumes of heterogeneous data and find anomalous patterns and precursors to hazards. Another requirement will be the development of interfaces with operators that can succinctly inform them of both impending hazards and corrective action to mitigate the hazards. Additional background information related to this research project appears in the discussion of the corresponding challenge earlier in this chapter.
This research project will be difficult to complete because of the growing complexity of the NAS and because of the large and growing number and variety of aircraft operating in the NAS, including new entrants. In addition, this research project faces significant uncertainties regarding the ability to acquire all of the data needed to monitor the NAS, to assess the system state, and to detect elevated risk states. This research project is urgent because in-time algorithms will form the core of the monitoring, detection, prediction, and mitigation tasks of the IASMS.7
Advances in supervised, semi-supervised, and unsupervised machine learning algorithms will be needed to reliably characterize known hazards and to discover new hazards, all while taking into account the multidimensional operational space of aircraft, their flight trajectories, human performance states, key inputs to human performance, and environmental factors.
Predictive algorithms will be needed to follow evolving situations and to prognosticate the occurrence of adverse events that can degrade safety. In addition, software tools will need to be able to conduct what-if analyses to support ground control and air traffic control to study the effects of safety assurance actions being applied in evolving hazardous and anomalous situations.
For both state identification and state prediction, this research project will need to consider the confluence of factors arising from aircraft operations and trajectories, interactions between UAS and manned aircraft, the density of air traffic, and weather conditions. Even when any one of these by itself does not imply an elevated risk state, some combination of these factors could indicate that hazards exist. Eventually, system analytics and data mining methods will need to be extended to support V&V methods both to assure that an IASMS will operate safely and to determine appropriate operational boundaries.8
This research project will address approaches to V&V to the extent that new methods will be needed to ensure the correctness of advanced algorithms. The complexity of this problem may necessitate the development of test beds, such as the Shadow Mode Assessment Using Realistic Technologies for the National Airspace System (SMART-NAS) project.9 The high dimensionality and the uncertainty in the data will necessitate verification schemes that are based on stochastic and Monte Carlo simulation methods.
IASMS algorithms must be robust and reliable given the variability in the quality and completeness of the available data. An operational IASMS will, of course, rely to a large extent on streaming operational data. Much of that data is not currently being collected, and so this research project (and some others) will need to rely on data produced by models and simulations. Data storage schemes and computational processing architectures for
___________________
7 As shown in Figure 2.1, the generic CONOPS defines a process whereby (1) operational data is extracted from the NAS and fed into a risk monitoring system that determines the system state; (2) the system state is continuously assessed for elevated risk states; and (3) when an elevated risk state is detected, a mitigation process is triggered to implement a safety assurance action that reduces the identified risk level.
8 For more information on VV&C, see both “System Verification, Validation, and Certification” sections in the Challenges and Research Projects sections in Chapter 5.
9 NASA’s SMART-NAS test-bed provides simulation and testing support for NASA’s air traffic management research.
big data, as discussed above, will also play an important role in this research. These algorithms will form the core of the in-time IASMS assessment and mitigation functions, and they are essential from maintaining the effectiveness of an IASMS as the NAS evolves.
Emergent Risks
Research Project Summary Statement: Develop approaches for continually mining historical data for detecting previously unknown anomalies and their evolution to characterize emergent risks and to update the IASMS risk assessment algorithms.
This research project would help achieve the vision for an IASMS because well-defined metrics to characterize safety margins and risk thresholds need to be established along with the ability to track these metrics as the NAS evolves. Although research in this area is already under way to study anomalies and emergent risks for individual aircraft, existing research will not meet the unique needs of an IASMS because of the complexity of an IASMS in terms of the scale, the heterogeneity, and the uncertainties in characterizing the airspace and time frame of interest to an IASMS. This research project will be difficult to complete because of the growing complexity of the NAS and because of the large and growing number and variety of aircraft operating in the NAS, including new entrants. This research is urgent because it will take a long time to develop the new classes of offline data-driven methods, machine learning and data mining algorithms, and analysis and prediction techniques that will be needed for each functional element (monitor, assess, and mitigate) of the IASMS to address adequately the hazards posed by emergent risks. Additional background information related to this research project appears in the discussion of the corresponding challenge earlier in this chapter.
Large amounts of historical data from air and ground operations over long periods of time are required to assess emergent risks. New and sophisticated data preprocessing methods are needed to clean, curate, and achieve established norms for data quality before the data are provided to IASMS analytics and machine learning algorithms. Furthermore, growth in the number and variety of new entrants, especially UAS; data integration and alignment; and feature extraction will be increasingly difficult.
This research project is not intended to conduct what-if analyses that explore a list of risks posed by researchers. Those analyses will be conducted by the research project on identifying and prioritizing risks (see Chapter 2). Rather, this project will be a data-driven effort that seeks to identify unknown near-term risks that are growing in magnitude and should therefore be considered for inclusion within the scope of an IASMS.
Existing semi-supervised and unsupervised learning methods that can operate on large amounts of complex historical data are needed to support discovery and characterization of anomalous operations and potentially unforeseen circumstances that may lead to elevated risk states. An important consideration for successful application of unsupervised algorithms is the application of feature selection to reduce the dimensionality of the space by removing redundant and irrelevant features. This will make the anomaly detection algorithms computationally efficient with outputs that are more robust in terms of false positives and false negatives, both of which need to be reduced. The anomaly detection methods developed to address emergent risks may also prove to be useful in providing new insights into the causes of known hazards. These anomaly detection methods will require the involvement of human experts to aid in the interpretation of anomalies and to initiate analyses to find root causes. Once the risks of anomalies have been characterized, the experts may direct updates to the in-time risk detection and identification algorithms.
Computational Architectures
Research Project Summary Statement: Support the design of data repositories and computational architectures that support online detection of elevated risk states and offline analysis to detect and characterize emergent risks and to update the IASMS risk assessment algorithms.
This research project will provide the core infrastructure that will provide the basis for the algorithms used for online and offline elements of an IASMS. With the explosion of big data applications across business, engineering,
medicine, and scientific research, distributed cloud architectures and accompanying computational architectures are being developed and deployed. This research project will be difficult to complete because research and development focused on other applications will not meet the unique needs of an IASMS in terms of scope and spatial and temporal complexities; the need for timely processing of large volumes of streaming and stored heterogeneous data with varying levels of quality and frequency; and the need to provide a reliable, fault-tolerant, and secure system that degrades gracefully when adverse situations (e.g., regional power failures) and malicious threats are launched against the system. This research is urgent because data repositories and computational architectures will provide the backbone of the IASMS operational system and are therefore needed early in the IASMS research and development effort. Additional background information related to this research project and to big data as it applies to an IASMS appears in the discussion of the corresponding challenge earlier in this chapter.
Computational architectures for IASMS face a scaled-up version of the big data challenge. This includes the need to effectively organize and extract relevant information from large volumes of frequently changing streaming data generated by multiple, heterogeneous, and autonomous sources in the NAS. In addition, there is the need for in-time analysis using statistical and machine learning techniques developed for risk identification and prioritization. Simultaneously, the architecture will have to support offline analysis of large volumes of stored heterogeneous data of varying quality and frequency for detection of previously unknown hazards and emergent risks. The potentially large number of airlines and other operators, the need to control UAS and manned aircraft in the same airspace, the lack of data collection standards and requirements (which will make data fusion more difficult), and the lack of well-defined repositories present significant difficulties in advancing research on computational architecture to support IASMS algorithm development, decision making, and visualization of complex data for flight and ground operations. Modern commercial transports have substantial on-board processing and data storage capacities, and this could facilitate the availability of data needed by an IASMS. It is not anticipated, however, that the computational architecture will rely on on-board systems because of the proprietary nature of those systems and the high cost of modification.
Many of the organizations that are most involved in the development and use of big data have been developing technologies in other domains to address the problems of in-time processing and analysis of large data streams. Even so, the extent to which these approaches can be applied to an IASMS remains to be seen.
One goal of this research project will be to develop technology-independent reference architectures and categorization of related implementation technologies and services to enable the development of a big data architecture suitable for an IASMS. Such an architecture will need to research a four-layered abstraction model to explore complex and evolving relationships among data. The four layers are the physical layer, the data layer, the computing layer, and the data analytics layer.10 Research related to the physical layer will include approaches for handling streaming and stored data, it will be scalable to assure redundancy and fault tolerance, and it will allow data transfer at high rates and efficient support for computation. Research related to the data layer will address core functionalities, such as data organization, to enable information exchange and fusion and to ensure that all distributed storage devices can support common goals while facilitating fast access and retrieval of the data in both the streaming and stored models of operation. Research related to the computing layer will address the needs for data modeling and query, and it will develop tools that facilitate retrieval of structured and nonstructured data while also supporting advanced computational architectures, such as Spark11 and Storm12 and their future evolutions to
___________________
10 The four-layer architecture is a widely used, generic template used in big data applications. However, some researchers and practitioners have proposed a different four-layer scheme, such as (1) data source, (2) data storage, (3) data processing/analysis, and (4) data output. See B. Marr, 2016, Big Data in Practice: How 45 Successful Companies Used Big Data Analytics to Deliver Extraordinary Results, John Wiley & Sons, Hoboken, N.J.
11 Apache Spark is an open-source cluster-computing framework for tackling big data problems. Spark provides programmers with an application-programming interface centered on a data structure called the resilient distributed dataset, a read-only multiset of data items distributed over a cluster of machines that is maintained in a fault-tolerant way. Spark supports a variety of systems, including Hadoop. See J.G. Shanahan and L. Dai, 2015, Large scale distributed data science using Apache Spark, pp. 2323-2324 in Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Association for Computing Machinery, New York, N.Y.
12 Apache Storm is an open-source, distributed, and scalable high-speed stream processing computational framework for applications such as real-time analytics, online machine learning, and continuous computation. See “Apache Storm,” http://storm.apache.org, accessed December 8, 2017.
allow for data-intensive computing for statistical analysis and machine learning algorithms. Research related to the data analytics layer will provide the computational abstractions to support the analytics and mining functions for in-time and emergent risk analysis while also providing the textual and graphical interfaces to support effective human-machine interactions.
This research project will also develop visual and configurable schemes for generating workflows that support the data analysis tool chain: acquisition, data cleaning and alignment, preprocessing, curation, analytics and mining, and generation of actionable information to support automated as well as human-in-the-loop decision making. This project would also investigate secure repositories and computational architectures that scale with the four V’s (volume, variety, velocity, and veracity) associated with big data applications.
