5

Mitigation and Implementation

This chapter addresses issues related to the mitigation function of an IASMS and to the implementation of an IASMS once the necessary technologies and capabilities have been developed. As discussed in Chapter 1, an IASMS requires the ability to detect and mitigate elevated risk states on a much shorter time scale than existing safety management systems. The NAS is a complex and evolving system of systems, and as that complexity increases, the risk increases that the outputs of an IASMS (that is, the safety assurance actions) will in some situations have unintended consequences that create new and unanticipated safety risks. The effectiveness of an IASMS will be limited if operators do not trust the system’s safety assurance actions, including alerts, decision support aids, and independent actions. It will be especially important for an IASMS to engender operators’ trust as advanced IASMS are developed and given greater autonomy and authority. Aircraft operators represent an important potential source of data for an IASMS, but they will likely not support costly technical and infrastructure investments unless the safety benefits are worth the expense and as long as participating in an IASMS does not create a competitive disadvantage.

This chapter identifies five key challenges and three high-priority research projects:

• Challenges
  • In-time Mitigation Techniques
  • Unintended Consequences of IASMS Action
  • Trust in IASMS Safety Assurance Actions
  • System Verification, Validation, and Certification
  • Operators’ Costs and Benefits
• Research Projects
  • In-time Mitigation Techniques
  • Trust in IASMS Safety Assurance Actions
  • System Verification, Validation, and Certification

CHALLENGES

In-time Mitigation Techniques

Challenge Summary Statement: Existing mitigation techniques are limited in their ability to respond to many risks in the short time frame of interest to an IASMS.

Developing mitigation technologies that can cover the scope of issues to be addressed by an IASMS will be a key challenge because of many factors, including the short time frame of interest to an IASMS (see Chapter 1), the many different types of operations in the NAS (see Chapter 2), and the wide variety of known and emergent risks (see Chapters 2 and 4).

An IASMS will be expected to mitigate elevated risk states associated with a wide variety of ground and air operations, such as arrival and departure sequencing, vectoring and hold operations, four-dimensional trajectories (latitude, longitude, altitude, and time), and balancing of capacity with demand across various regions of the NAS.¹ The response to some of these risks will require urgent action by operators to safeguard individual aircraft. In addition, anomalies and elevated risk states can in some cases propagate rapidly across various operations and aircraft, making it difficult to identify root causes and to mitigate elevated risk states in a timely fashion. Emergent risks are also of particular concern. For example, navigation, communication, and surveillance operations in the NAS are migrating to digital and network-based operations, and this may introduce new cyber vulnerabilities that could fall within the scope of an IASMS.

Timely mitigation techniques will need to account for human operators in the decision-making loop and provide sufficient information to them in a timely manner so that they can take the appropriate actions to mitigate the evolving risks and hazards. Consider Asiana Airlines flight 214, which on July 6, 2013, during an approach at San Francisco National Airport, struck a sea wall. At the time the instrument landing system’s glide slope for the runway was out of service and so the flight crew was making a visual approach. The National Transportation Safety Board determined that the probable causes of the accident were the flight crew’s mismanagement of the airplane’s descent during the visual approach, the unintentional deactivation of the automatic airspeed control by the pilot flying the aircraft, the flight crew’s inadequate monitoring of airspeed, and the flight crew’s delayed execution of a go-around after they became aware that the airplane was below the minimum acceptable altitude and airspeed for the glide path. An IASMS could possibly have identified and responded to key factors that combined to increase the risk during this approach. Most importantly an IASMS could have identified that the aircraft approach continued below 500 feet of elevation even though Asiana Airlines requires that a stabilized approach be established by that point in the approach. Depending upon the control authority granted to an IASMS, the system could have initiated a go-around or advised the flight crew to initiate a go-around while still at a safe altitude. As it was, the flight crew did not initiate a go-around until the aircraft was at 100 feet, and they were unable to complete the maneuver before impacting the sea wall.²

Even in those situations where the IASMS has the authority to act independently in response to a particular elevated risk state, human operators may need to be informed of the risk and mitigating action so that they can judge whether to allow the IASMS to continue its response or to override the IASMS.

Unintended Consequences of IASMS Actions

Challenge Summary Statement: An IASMS could inject new risks into the NAS due to unintended consequences of actions that it recommends or initiates.

___________________

¹ The IASMS Concept of Operations and National Airspace System Evolution research project will generate detailed guidance regarding requirements for system mitigation.

² National Transportation Safety Board, 2014, Descent Below Visual Glidepath and Impact with Seawall, Asiana Airlines Flight 214, Boeing 777-200ER, HL7742, San Francisco, California, July 6, 2013, Accident Report NTSB/AAR-14/01 PB2014-105984, Washington, D.C., https://www.ntsb.gov/investigations/AccidentReports/Reports/AAR1401.pdf.

Predicting and minimizing the unintended consequences that could arise from the safety assurance actions of an IASMS will be a key challenge because of the complexity of the NAS and the innumerable ways in which an IASMS could interact with various elements of the NAS. The overall safety analysis and operational approval of an IASMS will need to address the possibility of unintended consequences, and this may require synchronization between different human and/or automation agents in the system to avoid conflicting actions or recommendations between different agents in the system.

The importance of this challenge was illustrated by the Überlingen TCAS accident in 2002. This accident was a midair collision of two aircraft, Bashkirian Airlines Flight 2937 and DHL Flight 611, near Überlingen, Germany. The accident investigation identified two immediate causes for the accident.³ First, the air traffic controller did not notice that the aircraft were on a collision course soon enough to maintain safe separation between the two aircraft. Thus, although he directed Flight 2937 to descend to avoid the collision, the projected path of the two aircraft still indicated that the risk of collision was so high that the Traffic Collision Avoidance System (TCAS) on each aircraft directed the flight crew to take action. In particular, the TCAS on Flight 2937 advised the pilot to ascend (countermanding the order from the air traffic controller), while the TCAS on Flight 611 advised the pilot to descend. The second immediate cause of the accident was that the pilot on Flight 2937 continued to descend even after receiving the TCAS warning to ascend. In other words, the pilot on Flight 2937 received conflicting advice from two agents, one human and one automated, each of which was not aware of the action being recommended by the other. Because of this lack of synchronization, and because the pilot erred in following the advice of the human agent instead of the automated agent, two aircraft and all 71 people aboard those two aircraft were lost.

Trust in IASMS Safety Assurance Actions

Challenge Summary Statement: The efficacy of an IASMS will be degraded if it is built without regard to the factors that influence operators’ trust in the system.

Ensuring that operators develop trust in the safety assurance actions of an IASMS (e.g., system alerts, decision aids, and independent actions) will be a key challenge because the complex, multifaceted, dynamic, computational nature of the system may lead to safety assurance actions that are unfamiliar, unexpected, and/or run counter to operators’ training and experience. While this section addresses trust with respect to the safety assurance actions, the issue of trust includes many other factors. Chapter 3 addresses other trust factors, such as the disclosure of personal or private information.

The issue of human trust in technology is growing in importance as new systems incorporate an increasing level of functionality and decision-making capabilities. This can be a significant barrier to the acceptance of some increasingly autonomous systems, such as advanced UAS, but this issue is not limited to unmanned vehicles. The number of other systems that provide capabilities such as process improvement and intelligent decision support are also on the increase, and IASMS certainly falls in the latter area.

Appropriate levels of trust are necessary to assure that operators use systems such as an IASMS to their full extent. This occurs when an operator’s trust matches a system’s capabilities. If an operator’s trust of a system exceeds the system’s capabilities, then the system could be misused. On the other hand, if an operator’s trust falls short of the system’s capabilities, then the system could be underused. Inappropriate trust materializes as a mismatch of expectations and eventually leads to operator overload, limited or no use of the system, and decreased system utility.⁴

Within the aviation community, interviews with operators confirm that, even if a system’s capabilities are impressive and even if the system operates as the system developers and manufacturers intended, operators will limit their use of a system if they do not trust the system to act appropriately. In the best case, if operator accep-

___________________

³ Bundesstelle für Flugunfalluntersuchung (BFU) (German Federal Bureau of Aircraft Accidents Investigation), 2004, Investigation Report AX001-1-2/02, BFU, Braunschweig, Germany, http://www.bfu-web.de/EN/Publications/Investigation%20Report/2002/Report_02_AX0011-2_Ueberlingen_Report.pdf?__blob=publicationFile.

⁴ J.D. Lee and K.A. See, 2004, Trust in automation: Designing for appropriate reliance, Human Factors 46(1):50-80.

tance is not addressed during the formation of the system, operators will use only a portion of the functionality. In the worst case, the operators will ignore the system. Both the functionality and the money spent on systems such as IASMS will be wasted if operators do not trust the system enough to use it or if they act in accordance with the system’s output only when the system concurs with the action that operators have already decided to take.

Less complex technologies, such as anti-lock brakes on cars, were slow to gain operator acceptance due in part to the fact that the proper use of the braking system (which anti-lock brakes imposed under specified conditions) ran counter to operators’ training and experience. This created a conflict of expectations between what the operators believed they could do versus what the technology could do. The acceptance of anti-lock brakes was mostly solved through increased familiarity with the technology and with positive experiences that accumulated over time. With an IASMS, however, its extraordinarily complex algorithms, its use of enormous amounts of data, and the wide variety of situations in which it may act, will make it difficult to engender the trust of operators solely through changes in training and experience with the use of the system.

An IASMS has the potential to draw conclusions and either recommend or initiate safety assurance actions that in some cases are unfamiliar, unexpected, and/or run counter to operators’ prior training and experience. In some situations operators will have very little time to decide whether to trust the output of an IASMS, and in some of those situations it will be vital that the operator make the correct decision because the consequences of making a wrong decision could be catastrophic to the crew, their passengers, and to people on the ground.

System Verification, Validation, and Certification

Challenge Summary Statement: There is no accepted approach to verification and validation that leads to certification of a software system as complex as an IASMS, particularly if, as expected, the system includes adaptive and/or nondeterministic algorithms.

Verification, validation, and certification (VV&C) of an IASMS will be a key challenge because of shortcomings in regulatory requirements certification of an IASMS as it is currently envisioned; a lack of certification standards to provide guidance for complying with regulatory requirements; and a lack of verification and validation (V&V) technologies for an IASMS that would permit conformance to the requirements that are likely to appear in certification guidance.

“Verification” refers to the processes for ensuring that a given product, service, or system meets its specifications. “Validation” refers to the process for ensuring that the product will fulfill its intended purpose. V&V methodologies are typically underpinned by well-established scientific principles. VV&C are critical steps on the path to engendering the trust necessary for operators to accept the outputs of a complex system. This is especially important for an IASMS because the outputs of the system may not be intuitive to the operator.

V&V processes currently employed in aviation are geared toward obtaining quantitatively predictable outcomes based on known inputs or stimuli. In the case of an IASMS, requirements would be deconstructed from the overall IASMS into hardware and software. While challenging, there are several accepted approaches to certification of hardware. (See, for examples, DO-160C.^5,6) In the case of traditional computer hardware, certification can also be achieved through service-life history.

FAA certification of software is performed through the methodologies detailed in DO-178C, Software Considerations in Airborne Systems and Equipment Certification, and in DO-278C, Software Standard for Non-Airborne Systems. The DO-178C/278C methodologies have been shown to provide reliable software that meets certification requirements with certainty.⁷ While inputs to these software systems may be stochastic (from, for example, air data sensors), and stochastic analysis may be used for certification of mechanical aircraft parts, stochastic approaches are not used to certify software. The DO-178C/278C methodologies are time consuming and expensive to complete,

___________________

⁵ DO-160C, which is published by RTCA, addresses environmental conditions and test procedures for airborne equipment.

⁶ For a complete list of RTCA standards and other documents, see “Standards and Guidance Materials,” https://www.rtca.org/content/list-available-documents, accessed December 28, 2017.

⁷ National Research Council, 2007, Software for Dependable Systems: Sufficient Evidence, The National Academies Press, Washington, D.C.

and they do not apply to the adaptive, nondeterministic systems (see Box 5.1) that are likely to be incorporated into an advanced IASMS because of the following reasons:

• An IASMS software system will be very complex due to the varied and large number of inputs and the complexity of the algorithms.
• An advanced IASMS is expected to be adaptive because it will modify its response to a given input over time. That is, after an initial training phase an IASMS will continue to “learn” as a result of its operational experience in different situations. Therefore, the IASMS will tend to accommodate changes in the NAS, whether the changes are a result of operating in different geographic regions; seasonal variations in air traffic; and evolution of the NAS as systems, equipage, and procedures are updated, air traffic flows are rerouted, new entrants become more common, and so on. Current certification standards are incompatible with adaptive/nondeterministic systems because those standards demand and expect, among other things, that systems will consistently respond in the same way to a given situation. In order to be certified using current VV&C procedures, the adaptive features of an IASMS would need to be locked down after a training phase and before the VV&C process, thus negating the adaptability of the system. This would not, however, resolve this issue if the IASMS is also nondeterministic.
• Verification technologies for adaptive/nondeterministic systems are being developed by the Defense Advanced Research Projects Agency (DARPA), the Air Force Research Laboratory (AFRL), and NASA, among others. In addition, the National Highway Traffic Safety Administration is developing standards for autonomous cars that are consistent with the risk assessment matrix in Chapter 2 (see Figure 2.2). It may be possible to use the research coming out of these efforts to support IASMS research. An IASMS, however, is substantially more complex than the systems that are the focus of ongoing research, and additional research will be needed to support VV&C of an IASMS. This is particularly true with regard to development of methods for continuous certification or certification in block updates, which would facilitate improvements in an IASMS to accommodate changes in the NAS.

• Requirements need to be developed for IASMS that are based on a consensus among stakeholders with regard to system reliability, the ability to detect elevated risk states, an acceptable level of false positives, a clear definition of what constitutes a risk condition, and the ability to minimize the creation of new risks. This consensus will be the basis that regulators will use to address the certification of an IASMS.
• The level of intended impact of an IASMS (which could range, for example, from increasing the situational awareness of an operator to the initiation of safety assurance actions on its own authority) will likely determine the assurance level for the software design and V&V testing. Higher assurance levels result in higher costs for V&V.

In summary, there is tension between the nature of adaptive (learning) systems and the need to certify them. Impressive functionality can be implemented with adaptive capabilities, but this comes at a cost of not meeting certification requirements for a component of the NAS. Likewise, a deterministic IASMS could be certifiable, but that would also limit its advanced capabilities. A key part of the VV&C challenge will be to find a proper balance between functionality with the ability to certify an IASMS, recognizing that improving the latter would enable improvements to the former.

Operators’ Costs and Benefits

Challenge Summary Statement: Operators’ perception of the cost-to-benefit ratio of an IASMS may be so high that it will impede its implementation.⁸

The perceived cost-to-benefit ratio of implementing an IASMS could be a key challenge because operators will need to fund the purchase and installation of IASMS-specific equipment on their aircraft or other aviation systems.

Some aviation safety enhancements, such as those arising from the Commercial Aviation Safety Team (CAST) and Aviation Safety Action Program, can be implemented without new equipage. Many of the future data sources needed for the successful adoption of a fully functional IASMS, however, will require new and sophisticated aircraft equipage, such as upgraded avionics and sensors, as well as new ground infrastructure and data processing capabilities. Airline operations in the NAS are already extremely safe; based on current accident rates the probability that any individual airline will experience a catastrophic accident is extremely low, even over a time frame of decades. In addition, airlines and other operators have limited financial resources. Therefore, the higher the cost of developing and implementing an IASMS, the more difficult it will be to demonstrate a satisfactory cost-to-benefit ratio. The issue of cost is especially problematic with the general aviation community. Even though general aviation operations are much more hazardous than airline operations, the ability of general aviation aircraft owners to pay for new safety equipment is for the most part extremely limited. In addition, despite the higher accident rates for general aviation aircraft (see Chapter 1), the probability that any individual pilot will have an accident over a lifetime of flying is nevertheless quite low.

When operators believe that the cost of a safety system exceeds the expected benefit, widespread use is unlikely to occur unless and until regulatory mandates are issued. Consider the Aviation Collision Avoidance System, which was the predecessor to TCAS. These collision avoidance systems reduce the risk of collision for one aircraft only if other aircraft are similarly equipped. As a result, there was minimal benefit to early adopters. The issue of cost versus benefit for airborne collision avoidance systems in the United States was not resolved until the FAA issued a mandate following a midair collision in 1986 with 82 fatalities.

As noted in Chapter 2, however, the NAS is evolving into an increasingly complex system that will need to accommodate new entrants and address emergent risks associated with those new entrants and other factors. If as a result, the safety of the NAS is degraded, even in a minimal way, modernization and innovation in the NAS, including the accommodation of new entrants, could be significantly delayed, thus potentially hindering growth of the U.S. economy and international competitiveness. Developing a consensus to support IASMS research would be facilitated if the output of the recommended research projects, including interim deliverables, is expected to

___________________

⁸ All of the challenges addressed previously in this report are focused on technical issues. This is the only nontechnical challenge.

improve system efficiencies and reduce operational costs as development of an IASMS proceeds, if interim benefits can be achieved with minimal investments by operators in new equipage, and if the cost of adoption is incorporated as a fundamental element in the development and implementation of an IASMS.

RESEARCH PROJECTS

In-time Mitigation Techniques

Research Project Summary Statement: For the high-priority risks that fall within the scope of the IASMS CONOPS, this research project would identify those for which adequate mitigation techniques do not exist and develop approaches and technologies necessary to implement timely mitigation.

Most risk mitigation techniques relevant to an IASMS that have been developed to date involve operators (primarily pilots and air traffic controllers) with some instrumentation support (e.g., collision avoidance systems). Much more is needed, however, to provide the sophisticated decision support systems needed by an IASMS. The research projects “IASMS Concept of Operations and National Airspace System Evolution” and “Identifying and Prioritizing Risks” in Chapter 2 identify those risks that will be included within the scope of an IASMS. For these risks, this research project will focus on enabling an IASMS to be aware of relevant airspace and ground operations, threat detection and assessment, and decision support systems. This research project will be difficult to complete because of the need for new instrumentation, advanced analytic methods, and sophisticated prediction capabilities that take into account the increasing complexities and uncertainties in the evolving NAS, particularly with respect to new entrants. This research project is urgent because the success of an IASMS is dependent on near- and long-term mitigation schemes to maintain the safety and efficiency of the NAS and because of the long time it will take to achieve project goals. Additional background information related to this research project appears in the discussion of the corresponding challenge earlier in this chapter.

The scope of this research project would encompass three areas of interest, as follows:

• Research on expanded awareness of airspace and ground operations would build on new system analytics methods for detecting and identifying known and emergent risks. This will require integrated studies of gate, ground, and air operations and how they interact to affect the overall state of the NAS. The study of environmental effects, such as air turbulence and weather, on air operations will also be critical factors for study. Data-driven methods, such as Bayesian analysis, may form the basis for reducing uncertainty, establishing root causes for observed risks, and then developing mitigation techniques to address the root causes.
• Research on integrated threat detection and assessment will develop prediction tools, simulation methods, and planning and scheduling tools that will form the basis for complex, distributed decision making to facilitate in-time risk mitigation.
• Research on decision support systems will address strategic decision making that an IASMS will need either to make recommendations to operators for corrective action or to initiate corrective action within the limits of its authority. The research will also consider the need for advances in operators’ interactions with decision support systems. There are certain tasks that are clearly best handled by a human. Others are clearly best handled by an IASMS. Between these two extremes are tasks for which neither the human nor an IASMS is best suited. This research will address how a decision support system will determine for any particular situation whether a combination of the two or a selection of one or the other would be most beneficial.

This research will be of increasing importance as more and more tactical operations are automated with the evolution of the NAS. The goal of research in this area would be to provide tools that (1) provide performance precursors, (2) describe a causal chain of events to reduce temporal confusion, (3) improve situation awareness, and (4) monitor and reduce operators’ stress.

Trust in IASMS Safety Assurance Actions

Research Project Summary Statement: Identify factors specific to human trust in IASMS safety assurance actions.

This research project would help achieve the vision for an IASMS because if operators do not trust the ability of an IASMS to recommend or take appropriate action, they will minimize the use of the system or ignore it entirely. Many factors, such as the frequency and complexity of operator interactions with an IASMS in various situations, will need to be understood and addressed in order to foster operator trust in the system. For example, operators may be particularly reluctant to trust an IASMS when the safety assurance actions recommended by the system are unfamiliar, unexpected, or run counter to operators’ training and experience. One way to approach this challenge would be to implement the capabilities of an IASMS incrementally. This would make it easier for developers to ensure that each element of the operational system is trustworthy, and it would make it easier for operators to become familiar with and build trust in the system.

A better understanding of the factors that impact operator trust will also enable the proper balance of tasking between the operator and the IASMS so that the operator does not become overloaded. Change management processes will be critical when the system is deployed. The examination of the above factors, however, will need to begin at a much earlier stage than typical change management processes to assist in shaping the CONOPS, design, and implementation of an IASMS. Additional background information related to this research project appears in the discussion of the corresponding challenge “Trust in IASMS Safety Assurance Actions,” earlier in this chapter.

An IASMS will rely on systems that are growing in functionality and decision-making capabilities. Change management processes will be critical at the point of deployment of the system. The examination of these factors will need to occur at a much earlier stage than typical change management processes to assist in shaping the CONOPS, design, and implementation of an IASMS.

Research on human trust in increasingly autonomous systems is under way to support other applications, such as advanced UAS. Even so, this research project will be difficult to complete because an IASMS will be so much more complex, for example, with regard to the very large number of variables that an IASMS will consider for each functional element of the IASMS CONOPS (i.e., monitoring, assessing, and mitigating). In addition, creating an IASMS that operators will trust and therefore use will require a thorough understanding of the potential capabilities, nuances, and emergent properties of an IASMS. The research project is urgent because operator trust is a relatively new field, and this research project therefore does not have a large body of work to use as a resource. In addition, this research project will be most effective if its results are available early in the IASMS development process.

This research project will identify stated, unstated, met, and unmet operator needs relevant to establishing trust in a highly complex system such as an IASMS. It will also identify metrics and methods to determine the level of operators’ trust in and use of a complex system such as an IASMS, develop general principles that can be used in the design and development of IASMS, and identify evaluation metrics and methods to determine whether the IASMS adheres to those principles. The research project will involve scientists, operational personnel, unions, and the leadership of airlines, other operators, the FAA, and original equipment manufacturers.

System Verification, Validation, and Certification

Research Project Summary Statement: Develop practical methods for verifying, validating, and certifying an IASMS.

This research project would help achieve the vision for an IASMS because systems must be certified before they become operational. Although research in this area is already under way to support related applications such as certification of UTM and autonomous cars, existing research will not meet the unique needs of an IASMS because an IASMS will be much more complex than a highly automated/autonomous aircraft, and it will need to monitor and assess the operational safety of all existing and new entrants that will be operating in the NAS, encompassing the existing ATM systems as well as UTM systems.

This research project will be difficult to complete because development of certification standards for an IASMS will require a new approach to certification that promotes rapid and yet safe changes to the system, especially for an IASMS with adaptive/nondeterministic systems. Existing V&V procedures require months to years to complete, recertification is required if significant changes are made to configuration or core algorithms, and they do not apply to adaptive/nondeterministic systems, which will be incorporated in an advanced IASMS. This research is urgent because of the long time required to (1) create new VV&C processes that can be standardized and applied to other ATM systems (for example, via procedures established by RTCA and/or the European Organisation for Civil Aviation Equipment [EUROCAE]⁹), or (2) develop an alternative approach to VV&C. Additional background information related to this research project appears in the discussion of the corresponding challenge earlier in this chapter.

An initial step in executing this research project could use emerging systems such as UTM as a prototype, since it is essentially a microcosm of the future ATM system. The benefit to focusing this research on emerging NAS systems couples with the urgency in that there are near-term needs in the marketplace for certifying these systems in an efficient manner to facilitate safe and low-cost operations. Without robust certification techniques, standards, and tool frameworks, it is unlikely that an IASMS will be readily incorporated in the NAS. Targeted tasks within this research project will include the following:

• Review agile test methods that can be applied to ATM test beds.
• Identify achievable and desirable target levels of safety for automated and agile test frameworks to validate.
• Identify alternatives to “code coverage” and “parametric analysis” (robustness testing) as described in DO-178C/DO-278C.
• Define the key criteria for which an IASMS should be tested. Development of a clear IASMS CONOPS (see Chapter 2) is a prerequisite for accomplishing this task.
• Develop advanced simulation capabilities that can accept comprehensive input test vectors and develop complex test scenarios.¹⁰
• Determine the most efficient VV&C test environment and whether the test system should use live data as well as simulated data as inputs.
• Develop updated methods and means for initial and ongoing quality assurance and configuration management relevant to a system with the complexity of an IASMS.
• Develop publications for standards development that can be adopted by operators and regulators.

___________________

⁹ RTCA and EUROCAE are organizations in the United States and Europe, respectively, that support the development of aviation standards and regulations.

¹⁰ A test vector typically involves a time-sequenced data set that is used to investigate the functionality of a system relative to a set of requirements and/or a CONOPS. Test vectors are intended to emulate an expected real-world scenario (either nominal or off-nominal) in order to explore system behavior.