3
System Monitoring
The first step in the operation of an IASMS is a monitoring function that observes and characterizes the system state by collecting, fusing, and assessing data from a variety of sensors (see Figure 1.1 in Chapter 1). The scope and accuracy of an IASMS are thus directly related to the completeness and quality of the data that are available for an IASMS to observe the NAS and determine its system state. This chapter describes challenges for system observation and recommends research projects to address them.
System observation challenges include identifying, characterizing, and collecting data to be used for the intended scope of the IASMS, as established by its CONOPS; see Chapter 2). While some timely and accurate data sources exist, particularly for commercial transports, not all operational parameters are readily observed in a timely and accurate fashion. Some parts of the system are not observable at all, or only with significant effort and cost (e.g., small UAS) operating in unauthorized airspace), while others are not easily observed due to privacy or other policy considerations (e.g., pilot performance). The data from various sources differ in quality. The quality of data may also vary with time as sensors or system conditions change. Last, correlation and fusion of data sources can be challenging due to variations in key parameters, such as timing accuracy and latency, for data collected from different sources.
Resolving these challenges requires research in several key areas. A set of data needs to be identified and assembled via existing or new sources to enable sufficient system observations. Data sources need to be evaluated and monitored for quality and then fused to enable observations that meet the needs of an IASMS at a cost consistent with the value that the data provides. In addition, appropriate measures are needed to protect the privacy and proprietary concerns of data providers and/or the individuals being observed.
This chapter identifies three key challenges and two high-priority research projects:
- Challenges
- Data Completeness and Quality
- Data Fusion
- Collecting Data on the Performance of Human Operators
- Research Projects
- Data Fusion, Completeness, and Quality
- Protecting Personally Identifiable Information
CHALLENGES
Data Completeness and Quality
Challenge Summary Statement: Successful and efficient implementation of an IASMS requires identification, characterization, storage, and retrieval of the required data subject to availability, completeness, quality, and cost considerations.
Meeting the requirements of an IASMS for complete, high-quality, and affordable data will be a key challenge because some required data are not available and/or cannot be fused in the time frame of interest to an IASMS; some are not systematically stored or are not retrievable; some are expensive to obtain; some are burdened by accessibility and use constraints due to proprietary or privacy concerns; and some are not consistently available with sufficiently high quality, particularly with regard to accuracy and timeliness. While data acquisition is a general-purpose issue for any NAS decision support system, an IASMS has unique requirements for data availability, quality, and timeliness that need to be addressed early in system development because the success of an IASMS is highly dependent on the data used.
A comprehensive IASMS would observe all operations and entities that impact safety risks of interest to an IASMS, both in real time and over some period of history.1 This is not currently possible because available data sources do not completely cover the needed observations. For example, there is little historical data on new entrants to the NAS, particularly UAS, ODM aircraft, and the increasing pace of commercial space launch and reentry operations. Unlike commercial transports, for which there exists an extensive historical record of normal operations, incidents, and accidents, no such record exists for new entrants. This will be an issue for developing algorithms to detect relevant safety issues. It is also not yet possible to acquire data on new entrant operations in real time or close to real time. While some space operations provide telemetry data, most do not, and while they may be surveilled by Department of Defense (DoD) radar systems, that information is not currently available to the FAA for air traffic control purposes. Standards and data streams do not yet exist for UAS operational data reporting and storage, and some UAS missions are inherently unpredictable because they typically do not follow a fixed flight plan. Likewise, general aviation aircraft do not necessarily file a flight plan if operating under visual flight rules, and surveillance data on general aviation aircraft are only available under a limited set of conditions. New data sources such as ADS-B will likely be needed to bridge existing gaps in observational data.
The means for data collection (as well as for command and control operations) in real time typically involves wireless links from aircraft to terrestrial or satellite-based systems as well as ground system-to-ground system networks. An IASMS could also take advantage of aircraft-to-aircraft communications systems that could become more prevalent in the future. Key factors regarding the collection of data from each of these sources include availability, latency, update rates, integrity, security, formats, avionics standards, implementation and service costs, spectrum regulation, and bandwidth utilization.
A successful IASMS will be trustworthy and capable of effectively detecting elevated risk states (that is, the system will experience few false negatives), and it will have very low false alarm rates (that is, few false positives). This could be quite difficult to achieve, since an IASMS will attempt to detect and assess the risk of rare events—and it will be especially difficult unless the quality of input data can meet the needs of an IASMS CONOPS. Many existing data feeds, however, such as the feed from the FAA’s System-Wide Information Management (SWIM)2 system, are composed of data from many sources of varying quality. The utility of these data in supporting an IASMS CONOPS is limited unless it is possible to determine the source and the quality
___________________
1 The actual scope of the observations conducted by an IASMS will be determined by the CONOPS.
2 The FAA’s SWIM system is an NAS-wide information system that supports development of the FAA’s Next Generation Air Transportation System (NextGen) modernization program. SWIM will generate standard data streams and interfaces for use within the NAS and by other members of the aviation community. SWIM is intended to replace legacy point-to-point system interfaces with a modern service-oriented architecture, thereby providing a common set of connections and data components. As of September 2017, SWIM streams are available for traffic flow management data, terminal radar data, flight plan data, and airspace data. As the system matures, it will also provide weather forecast data. Additional information on SWIM is available at https://www.faa.gov/nextgen/programs/swim (accessed December 8, 2017).
of each individual datum and to monitor for changes that would affect the accuracy and reliability of the data. Furthermore, the accuracy and reliability of the data must be explicitly quantified, such that for each risk of interest it is possible to determine key statistics, such as the probabilities of detection and of false alarms. For example, the SWIM Traffic Flow Management System provides a large set of flight data and flow information, including flight planning data, aircraft positions, airport and route status, and predeparture flight status information. Flight position data come either from en route radar, which provides accurate and reliable data, or from oceanic position reports, which provide data that are less accurate, less reliable, and less consistent in terms of quality. Data on the altitude of individual aircraft come from on-board sensors and thus vary in quality depending upon the accuracy of a particular aircraft’s on-board altitude measurement system. Some other data are self-reported by airlines, and the timeliness and reliability of these data vary across airlines and in some cases vary over time as airlines update their data collection and reporting systems. Future enhancements to the traffic flow management system will include position reports generated by the ADS-B system, and this will improve the accuracy, reliability, and consistency of position data, but ADS-B data will not necessarily be available for all flight tracks.
The sources and quality of data provided collected by an IASMS need to be understood and tracked over time to determine the quality of IASMS outputs. In addition, some data feeds will need to be stored in a retrievable way, which will be necessary to enable an IASMS to continuously analyze data over the last few days or weeks as necessary to detect elevate risk states that may arise over these time frames. Data storage and retrieval capabilities would also enable running data quality analyses, so that IASMS can determine when a specific data source was either unavailable or degraded for some reason (e.g., because data were being collected from a backup system with lower accuracy).
There is also a value proposition involved in selecting data for system observation. Some data sources are readily available in real time and easy to access. This is the case with data that can be accessed via SWIM feeds. In other cases, data that would be useful for an IASMS, such as position reporting for small UAS, may not be available or the data may be too difficult or too expensive to collect and access. The cost and availability of data will be an important consideration when developing an IASMS CONOPS and prioritizing the risks to be included within the scope of an IASMS (see Chapter 2). Put another way, it is important to identify which data are not necessary or worthwhile to collect.
Data Fusion
Challenge Summary Statement: To accurately detect safety risks, an IASMS will need to correlate and synthesize data from heterogeneous data sources with different formats, timing, accuracy, and other characteristics.
Developing the data fusion capabilities needed by an IASMS will be a key challenge because, while systems exist to assemble and fuse safety-related data weeks or months after operations are complete, the IASMS will require data to be fused in a much more timely fashion. The performance of an IASMS would be enhanced or in some cases enabled by the fusion of data that are not collected by current systems (e.g., human performance data).
Observations needed to detect an elevated risk state in the NAS will require data from many different sources. These sources will vary in accuracy and latency, they may overlap in coverage (e.g., multiple surveillance sources), and they may require correlation (e.g., flight plans and surveillance reports). An IASMS will need flight data, such as aircraft state and trajectory data, as well as nonflight data, such as human performance measurements or voice communications between controllers and pilots and in the cockpit and among the members of a single flight crew.3
Methods for fusing flight data for commercial transports are mature, as this is done today in both real-time and post-event analyses. For example, the SWIM Traffic Flow Management System synthesizes data from multiple surveillance sources, schedules, flight plans, and weather forecasts to generate information on current aircraft state and predicted aircraft trajectory. Aircraft position and velocity data are known or estimated, but other parameters
___________________
3 Recording voice communications within the cockpit and making them available to an IASMS raises privacy issues that are addressed in the next section and in “Protecting Personally Identifiable Information,” in the Research Projects section later in this chapter.
that may be of importance to an IASMS, such as bank angle or vertical speed, are not known well or at all. Trajectory intent is also imperfectly known, as flights are not always following a cleared flight plan, as is the case when controllers clear pilots to deviate around severe weather as the pilots deem necessary. To achieve IASMS goals, it may be necessary to fuse data from additional sources, such as from ADS-B reports or voice recognition of controller-pilot voice communications.
More data sources are available for post-event analysis than for in-time analysis. Some data cannot be obtained in a timely fashion (e.g., flight recorder data). Also, noncausal4 post-processing algorithms can be used to produce more accurate flight state data.
As previously noted, comprehensive flight data are not available for flights operating under visual flight rules, for UAS operations, or for commercial space flights. Data fusion is thus also more difficult, and real-time (or near-real-time) modeling and prediction may be necessary to predict or infer vehicle status and intent based on the limited data available and in the time frame of interest for an IASMS.
Data fusing can also raise sensitivity considerations, particularly with regard to storage. Archiving data and making them available for post-event processing may lead to identification of safety issues that are sensitive to some aviation system stakeholders. Data fusion may also reveal sensitive non-safety-related information, such as details of a carrier’s business objectives or proprietary design features of an aircraft. This risk is increased when data are stored and are made available for post-event analysis.
Collecting Data on the Performance of Human Operators
Challenge Summary Statement: Data regarding operator performance that are essential to achieving the full potential of the envisioned IASMS cannot be collected in a timely fashion or at all, in part because of privacy and related concerns.
Providing an IASMS with sufficient, timely data on the performance of operators will be a key challenge because of privacy and related concerns. Much of the data necessary to perform in-time safety analysis are associated with the actions and performance of an individual pilot, controller, or other member of the aviation industry. Individuals will be reluctant to agree to disclose performance data that could jeopardize their livelihood, subject them to regulatory or company sanctions, or violate their personal privacy. Similarly, information held by private companies and airlines, including data on the performance of their staff, will be difficult to obtain due to potential liability issues.
There are currently in place systems to improve operator safety performance, and these systems have greatly contributed to aviation safety. These systems are part of airline safety management systems and include FAA-approved programs like the Aviation Safety Action Program (ASAP) to facilitate reporting of safety problems by employees and the Flight Operations Quality Assurance (FOQA) program to capture and disseminate flight monitoring data. These programs provide the foundation of the Aviation Safety Information Analysis and Sharing (ASIAS) program (see Chapter 1).
To mitigate the concerns of operators regarding privacy, enforcement actions, and liability and to thereby encourage the submission of data that would otherwise be unobtainable, the U.S. Congress provided the FAA with the authority to protect from public disclosure information provided under an approved program such as the preceding.5 This has been extremely successful in obtaining safety information and maintaining the confidentiality of information. Similar protections may be necessary for information provided by individual and corporate operators to NASA for research purposes and ultimately to the entity entrusted with operating the IASMS. Without such legal protections and trust, valuable safety information may not be available for analysis, which could limit the full potential of an IASMS.6
___________________
4 In real time, only data on events that have already occurred can be used. In post-event processing, this restriction is not present (e.g., future positions of a flight can be used in refining the “present” position).
5 Protection of Voluntarily Submitted Information, 49 USC 40123, 1996.
6 There are analogous privacy, liability, and security concerns in automotive data management, especially as the industry moves toward high-volume vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), and vehicle-to-cloud-to-vehicle (V2C2V) communications. Research devoted to solving automotive problems may also provide solutions for analogous aviation data management issues.
Current systems such as those listed above are not designed to collect and assess data in the short time frames of interest to an IASMS. Because many accidents and incidents can be traced to human error (individual or collective) or miscommunication, it is important to identify states and conditions that indicate an elevated risk state that involves human error and/or miscommunication. Key issues include how to identify those factors that contribute to elevated risk states related to human performance (e.g., high emotion, fatigue, or inattention, etc.) and how to develop noninvasive data collection methods that will be acceptable to operators, particularly regarding privacy concerns.
Operators sometimes face high levels of stress. The combination of factors such as insufficient rest, hazardous weather, illness, emotional state (e.g., because of the death of a family member), and concerns about their competency to handle operational requirements (especially for inexperienced general aviation pilots on solo flights) can affect an operator’s ability to focus or contend with an urgent situation.7 This highlights the importance of understanding preexisting emotional or cognitive issues that might affect task performance as well as the need for timely data on the state of individual operators.
Currently, there is a significant body of work that quantifies and assesses the performance of pilots. This work is primarily limited to controlled environments, such as a laboratory or simulator, and it includes information gathered via post-flight and post-accident reports. In many studies, data on the internal state of pilots is based on qualitative self-reports of the subject’s cognitive and emotional state, such as perceived stress levels and situational awareness. Subjects may have difficulty describing their internal states, and there is also a risk that subjects may under- or over-report their emotional and cognitive processes depending on real or perceived consequences.8 Much of this work focuses on the final results of a mission or on the completion of a challenging task. Metrics indicating the internal state of the operators in real time or near-real time are far less mature.
The ability to rapidly identify, quantify, and evaluate human-system performance offers a significant advantage in increasing operational safety and efficiency.9 This information can directly aid the operator’s actions and performance through appropriate and well-timed feedback, and it can inform other operators in the NAS of impending issues and conflicts.
RESEARCH PROJECTS
Data Fusion, Completeness, and Quality
Research Project Summary Statement: Develop methods to automatically collect, fuse, store, and retrieve data from different sources and with different formats, timing, accuracy, and other characteristics.
This research project would help achieve the vision for an IASMS because the range of capabilities that can be successfully implemented in an IASMS will be limited if available data are inadequate in terms of completeness, quality, consistency, the ability to fuse them in the time scales of interest, the ability to store them for future use, or the relative cost and value of obtaining additional and/or higher quality data. This research project will be difficult to complete given the substantial advances that are needed to define, acquire, understand, fuse, and store the data required to support planned IASMS capabilities. This project is urgent because it is fundamental to the success of an IASMS and because some components will likely take years to complete. Additional background information related to this research project appears in “Data Completeness and Quality” and in “Data Fusion” in the Challenges section earlier in this chapter.
___________________
7 Many general aviation pilots have little flight experience. In contrast the flight crews of all commercial transports include at least one pilot who holds an airline transport pilot certification, which is the highest level pilot certification. This disparity in pilot experience is one factor that contributes to the high accident rate of general aviation relative to airlines. AOPA Air Safety Institute, 2017, 26th Joseph T. Nall Report, General Aviation Accidents in 2014, Washington, D.C., https://www.aopa.org/training-and-safety/air-safety-institute/accident-analysis/joseph-t-nall-report.
8 A.K. Webb, A.L. Vincent, A.B. Jin, and M.H. Pollack, 2014, Physiological reactivity to nonideographic virtual reality stimuli in veterans with and without PTSD, Brain and Behavior 5(2):e00304, doi: 10.1002/brb3.304.
9 K.R. Duda, Z. Prasov, S.P. York, J.J. West, S.K. Robinson, and P.M. Handley, 2015, “Development of an Integrated Simulation Platform for Real-Time Task Performance Assessment,” 2015 IEEE Aerospace Conference, pp. 1-9, doi:10.1109/AERO.2015.7118974.
One thrust of this research project would be to establish a picture of current and projected data completeness: What data are likely to be available? What data are needed but are unlikely to be available given current trends in research? What research and changes in policy would enable the collection of necessary data? This research project would begin the slow process of developing operators’ support for providing data that are proprietary, sensitive, and/or technically difficult to obtain. This will require consensus in the aviation community that the safety benefit of the information is great enough to convince operators that releasing the data is in the best interest of all aviation users, including themselves. Even then, in some cases it may be difficult to obtain requisite data due to changes in the NAS. For example, there are many envisioned UAS mission profiles in which UAS would operate within airspace shared by manned aircraft. Some UAS will be equipped with increasingly autonomous systems and some will have significantly different flight performance characteristics than manned aircraft. As a consequence the data required to assure safety of some UAS operations are not yet well understood.
Another thrust of this research project would be to (1) gain a complete and quantitative understanding of IASMS input data characteristics, and (2) develop algorithms and processes for monitoring, fusing, and updating this understanding over time as data quality changes and new data sources are added. This would include analyses of the accuracy, timeliness, and reliability of existing and projected input data as well as establishing relationships with the original data providers to anticipate changes. While this is particularly critical for establishing the accuracy of and developing trust in time-critical IASMS functions, it is also important that the quality of stored data be known and recorded so that it can be improved after the operations of interest have concluded, thereby improving the utility of post-operations analyses. Shortcomings in some data, such as gaps in receiving data from a particular source, will likewise need to be recorded and their implications understood. It would also be useful to advance existing capabilities to solve for incomplete data. Existing studies of the quality of NAS data from different sources will be useful to this research project, but the specific needs of IASMS functions will likely require additional studies and algorithm development. The difficulty of the research project will vary across the different data types and sources IASMS will collect. Analyses of data quality requirements will determine which IASMS functions are feasible, and they will provide a basis for setting alerting thresholds that result in a high probability of detecting elevated risk states and a low probability of false alarms.
Protecting Personally Identifiable Information
Research Project Summary Statement: Develop methods of de-identifying and/or protecting sensitive data in a way that does not preclude effective data fusion.
This research project would help achieve the vision for an IASMS because it would enable the timely and automated fusing of large data sets to help overcome the concerns of operators regarding privacy and related concerns. Additional background information related to this research project appears in “Collecting Data on the Performance of Human Operators,” in the Challenges section earlier in this chapter.
For information to be used for in-time monitoring and assessment and to be stored for future use, advances in technology (and changes to regulatory policy) are needed to address operators’ concerns regarding unauthorized disclosure of identifiable data. Research in this area has advanced the state of the art of text mining of narrative data when disparate data are combined from numerous sources, but current approaches greatly increase the potential for identifying the individual source of specific data. To overcome the concerns of operators and thereby enable an IASMS to achieve its full potential, de-identification of data will need to be done quickly and without the loss of critical operational safety data. This research project will be difficult to complete because source data will be generated from unique and sometimes proprietary systems. It is likely to be particularly difficult to convince pilots and other operators to agree to the monitoring of their cognitive and emotional states, their decision-making capacity, and their sense of spatial orientation for those who are in the risk mitigation decision chain. This research is essential because information concerning time of day, aircraft type, registration numbers, along with many other identifiable sources of information would be particularly valuable to an IASMS, and in many cases key operational and safety data will not be available from other sources.
This research project is urgent because of the time that it will take to develop improved methods for de-identifying and protecting data and to then develop a broad consensus among stakeholders (including operational personnel, unions, and the leadership of airlines, other operators, the FAA, and original equipment manufacturers) that these methods are adequate. This research project could assess the value to each stakeholder of collecting operator data so that all stakeholders understand the value of collecting and storing relevant data in terms of improving the safety of the NAS. For example, the research project could work with stakeholders to identify potentially unsafe conditions that could effectively be addressed using data on operator performance to identify data of particular interest and when it should be collected. Monitoring operators en route is contentious in part because of operators’ concerns about recording personal discussions on topics not directly related to operations (even though such communications are not prohibited). It may be less difficult, however, to achieve consensus regarding the collection of selected data in critical parts of the flight regime, such as, for example, upon arming of the instrument landing system when only flight operational communications should be discussed. It will also be important to assure that an IASMS has adequate cybersecurity protection in order to safeguard personally identifiable information. Cybersecurity is also important to assure the operation of the system as a whole, however, so it need not be included in this particular research project.10
___________________
10 See “Identifying and Prioritizing Risks,” in the Challenges section in Chapter 2, for more information on cybersecurity.
